{"id":4891,"date":"2013-07-02T15:58:07","date_gmt":"2013-07-02T23:58:07","guid":{"rendered":"http:\/\/www.chesnok.com\/daily\/?p=4891"},"modified":"2014-02-10T13:17:43","modified_gmt":"2014-02-10T21:17:43","slug":"a-practical-guide-to-using-alembic","status":"publish","type":"post","link":"https:\/\/www.chesnok.com\/daily\/2013\/07\/02\/a-practical-guide-to-using-alembic\/","title":{"rendered":"A practical guide to using Alembic"},"content":{"rendered":"

I spent some time guiding a coworker through using Alembic<\/a> for the first time with Socorro<\/a> this morning and what follows are my notes from that meeting.<\/em><\/p>\n

I’ve been using Alembic, a database schema migration<\/a> tool, for about three months now, and really liking it a lot. I created a blog post that served as a slide deck for an internal team called A lightspeed tour of Alembic<\/a> as my first stab at user education.<\/p>\n

Setting things up initially was pretty simple, but explaining it to a coworker after I’d set everything up for myself proved slightly more difficult. Below are my notes on the differences between Alembic<\/a> and some other migration tool.<\/p>\n

Terminology<\/h2>\n

Alembic calls each migration a revision<\/em>. Revisions know what order to be run in because each revision is given a down_revision<\/code> to identify its parent. If down_revision<\/code> is None<\/code>, that revision is the very first revision according to Alembic. You can put your whole schema in that revision, or you can just start adding changes to this initial revision. Alembic doesn’t complain either way.<\/p>\n

A best practice would likely be putting your entire model into the first revision. I may go back and “fix” this for us later. I opted to just have the default use case be to create a database fresh with a tool we call setupdb_app.py<\/code>.<\/p>\n

If you’re looking to migrate to using alembic, you’ll also need to use SQLAlchemy. I used sqlautocode for my initial schema reflection<\/a>, and there’s a new tool sqlacodegen<\/a> you may want to check out for generating your SQLAlchemy models for the first time.<\/p>\n

Preparation: edit config and activate a virtualenv<\/h2>\n

Our environment was set up per the alembic tutorial for creating an environment<\/a>. I ran:<\/p>\n

alembic init alembic<\/code><\/p>\n

I also put an alembic.ini-dist<\/code> file into our project’s config\/<\/code> directory, and modified alembic\/env.py<\/code> to include our model.<\/p>\n

To get started working with an existing install, you’ll need to modify alembic.ini-dist<\/code>, and copy it to config\/alembic.ini<\/code> to fit your environment – setting the connection string and the path to the alembic directory are the two most important settings. We have a script which creates databases from our models.py<\/code> called setupdb_app.py<\/code>. This script takes --database_name<\/code> as a command-line argument. My default for our project is to use breakpad<\/code>.<\/p>\n

We use a virtualenv<\/code> called socorro-virtualenv<\/code>. The virtualenv is created automatically if you run make test<\/code>. If you’re creating a standalone virtualenv, you can do that with virtualenv socorro-virtualenv<\/code>. Activate this with . socorro-virtualenv\/bin\/activate<\/code>.<\/p>\n

Creating a revision<\/h1>\n
    \n
  1. Create a fresh database to work from. For Socorro, the command is: PYTHONPATH=. socorro\/external\/postgresql\/setupdb_app.py --database_name=breakpad<\/code><\/li>\n
  2. Edit models.py<\/code> with the change to the schema<\/li>\n
  3. Run: PYTHONPATH=. alembic -c config\/alembic.ini revision -m 'your message about the migration'<\/code>. The output will include the name of the new file.<\/li>\n
  4. Edit the new file as needed alembic\/versions\/*.py<\/code><\/li>\n
  5. Run: PYTHONPATH=. alembic -c config\/alembic.ini upgrade +1<\/code><\/li>\n
  6. Test your downgrade with PYTHONPATH=. alembic -c config\/alembic.ini downgrade -1<\/code><\/li>\n<\/ol>\n

    If all goes well, your revision is ready! If something goes wrong, edit and try again. The revision will automatically rollback if there are any errors.<\/p>\n

    Downgrades are a little tricky to properly execute. In an ideal world, you’d be able to revert the underlying code, but preserve only<\/em> the commit containing the migration. More on this in a future blog post!<\/p>\n

    Creating a revision using --autogenerate<\/code><\/h1>\n

    This is very similar to the above, with the addition of --autogenerate<\/code> to your revision<\/code> command. This should do the right thing, but definitely check your generated file for accuracy.<\/p>\n

      \n
    1. Create a fresh database to work from. For Socorro, the command is: PYTHONPATH=. socorro\/external\/postgresql\/setupdb_app.py --database_name=breakpad<\/code><\/li>\n
    2. Edit models.py<\/code> with the change to the schema<\/li>\n
    3. Run: PYTHONPATH=. alembic -c config\/alembic.ini revision --autogenerate -m 'your message about the migration'<\/code>. The output will include the name of the new file.<\/li>\n
    4. Edit the new file as needed alembic\/versions\/*.py<\/code><\/li>\n
    5. Run: PYTHONPATH=. alembic -c config\/alembic.ini upgrade +1<\/code><\/li>\n
    6. Test your downgrade with PYTHONPATH=. alembic -c config\/alembic.ini downgrade -1<\/code><\/li>\n<\/ol>\n

      If all goes well, your revision is ready! If something goes wrong, edit and try again. The revision will automatically rollback if there are any errors.<\/p>\n

      Production deployment<\/h1>\n

      You’ll need to deploy an alembic.ini<\/code> on your production database system and probably a virtualenv<\/code> to support your python modules.<\/p>\n

      We deploy our virtualenvs with our application, so this step was pretty simple for everything except for alembic itself. The virtualenv put in full, static paths for the python binaries and had some dependencies that I haven’t figured out yet for actually running alembic. To get around this, I created a virualenv locally on the system for the postgres<\/em> user. Having your postgres user run the migrations locally is a must for me because I need to access the filesystem to pull in new versions of user defined functions stashed in the directory my model lives in.<\/p>\n

      I just deploy a new release of our application code on the database server locally, and then I run alembic against the versions<\/code> directory that’s deployed.<\/p>\n

      FAQ<\/h1>\n

      And here’s an FAQ for the common problems folks ran into:<\/p>\n

      OOPS I forgot to create a database before I created a revision!<\/h2>\n

      To “fix” this, try:<\/p>\n

        \n
      1. Create the database from scratch using your current models.py<\/code>.<\/li>\n
      2. Run: PYTHONPATH=. alembic -c config\/alembic.ini downgrade -1<\/code><\/li>\n
      3. Run: PYTHONPATH=. alembic -c config\/alembic.ini upgrade +1<\/code><\/li>\n<\/ol>\n

        Assuming your downgrade function works, this should allow you reverse the latest revision and then test your migration.<\/p>\n

        Error message: “Only a single head supported so far.”<\/h2>\n

        See Working with Branches<\/a>.<\/p>\n

        I’m using schemas, and alembic doesn’t recognize them when I try to use --autogenerate<\/code>.<\/h2>\n

        See include_symbol<\/a>. And be sure to add this to both the “offline” and “online” versions of the revision code in env.py<\/code>.<\/p>\n

        Error message: Target database is not up to date.<\/h2>\n

        This means you’ve got a file in your versions<\/code> directory that contains one or more migrations that haven’t been applied to the current database. You can either apply them with alembic upgrade head<\/code> or have a look in that directory and remove the migration(s) that you don’t want.<\/p>\n","protected":false},"excerpt":{"rendered":"

        I spent some time guiding a coworker through using Alembic for the first time with Socorro this morning and what follows are my notes from that meeting. I’ve been using Alembic, a database schema migration tool, for about three months … Continue reading →<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,407],"tags":[598,647,624,599],"class_list":["post-4891","post","type-post","status-publish","format-standard","hentry","category-postgresql","category-python","tag-alembic","tag-postgres","tag-postgresql","tag-sqlalchemy"],"_links":{"self":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/4891","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/comments?post=4891"}],"version-history":[{"count":10,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/4891\/revisions"}],"predecessor-version":[{"id":5160,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/4891\/revisions\/5160"}],"wp:attachment":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/media?parent=4891"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/categories?post=4891"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/tags?post=4891"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}