Personal email sabbatical, July 10 – October 15 2014

Starting July 10, 2014, I’m planning to take a personal email sabbatical.

I’m going to unsubscribe from all mailing lists and redirect all email to my personal email addresses ( and to /dev/null.

I have a couple open source project addresses that will probably be redirected to people responsible for those areas during my sabbatical. The traffic on those addresses is so low, however, that this is likely not going to impact anyone significantly.

I will not be reachable via personal email until sometime around October 15, 2014.

Why I’m doing this:

I spend too much time responding to things that other people want me to do.

I have a full-time job where I do that already, and as a result, much of my personal time where I could be creative is spent in the service of others. I’m also about to have a baby! So, it seems like as good a time as any to make an explicit departure from the communication medium that’s been the focus of my life since about 1995.

I’m not sure that I will return to having a public email address outside of work.

I plan to continue blogging, although using a much simplified blogging platform that I’ve written (but haven’t deployed yet :D). I plan to continue to use Twitter, although I’ve separated public and private accounts. I do not plan to increase my use of other social media. I plan to continue to use IRC, although will likely take myself out of most project channels. I’ve removed Twitter from my phone, and plan to remove email from it in July.

I’m taking a cue from danah boyd, although not following her prescription for a lengthy advance notice to collaborators.

I spent the last six months substantially downsizing my volunteer commitments, spending more time on in-person events that are local. I am pretty sure most people won’t even notice my extended volunteer absence on the “greater internet”. I’ll trickle out some individual notices as needed, probably to anyone I’ve had more than three exchanges with in the last two or three years.

Things I’ve learned from my personal trainer

I’ve been seeing a personal trainer for the last couple of years. I injured my back pretty seriously 5-6 years ago, and finally admitted that I had a “real problem” that required doctor and other professional intervention. After x-rays and a couple months more of denial, I started an exercise program focused on weight training.

Previously, my main (and sometimes only) exercise had been running. The back problem I have caused my doctor to recommend that I never run again. At first this was an easy thing to do because running was incredibly painful. Later, I realized that running was not only a fitness thing for me, but also an important part of my mental health. I really needed another way to get in a lot of exercise.

I go to the gym 3 times a week, and I try to walk about 5 miles a day at least 3 additional days a week. Playing Ingress, and bird watching tend to make this pretty easy.

I’m now 7-months pregnant, and I still do both things. I typically don’t get any 10k days in like I did before I was pregnant, but apart from some grueling travel I did in April, I’ve been able to keep it up the entire time.

I’ve threatened to give a talk about this, but rather than wait to assemble all of this until then, I thought it would be nice to just write it down in the event it helps someone else!

My physical activity background:

I did first ballet, then karate and judo as a child. I never played any ball or team sports, and did not learn to swim other than barely keep myself afloat until I was an adult. In high school, I ran cross country for three seasons. Mostly I played the violin, at least 2 hours a day sometimes up to 5 hours a day, which is a physically tiring activity, but not aerobic (usually). I also hiked and backpacked with my family. I was able to run a 7-7:30 minute mile until a couple years after I graduated from college. In general, I thought of myself as a pretty fit person, and ran slower because no coach was yelling at me and no competitors were chasing me as I got older. The last time I seriously “trained” was for a marathon I ended up not running in about 2005. I got myself up to 18 miles and then just cycled about 8 miles a day to and from work for many years.

So, by the time I decided to get a trainer I was not in very good shape.

Here’s the stuff that I learned right away:

  • I did not know how to lift things. “Lift with your legs” is what people say. I did not know what this really meant, and was doing some strange combination of using my quads and my lower back to lift heavy things. That’s pretty much exactly what you should not do.
  • My shoulders were incredibly weak. Part of that was from typing all day for my work and being tense. Another part was just lack of use.
  • My lack of grip strength was causing me a lot pain in my hands and forearms. I’d overuse my wrists and hands every day, and then go back and do it again. I didn’t quite have carpal tunnel, but it was probably coming pretty soon.
  • I was very embarrassed by my lack of basic knowledge in the gym. I’m pretty good with jargon, but holy shit, gym language was complicated. And didn’t map to anything I already knew. The names of the exercises were not helpful to me, and I found myself drawing lots of stick figure pictures to try to remember how to do things properly. I didn’t know the names of equipment, I couldn’t remember the difference between a barbell and a dumbbell, I could not for the life of me recall an exercise if I took more than about a week-long break from it.
  • Strengthening my back causes a lot of soreness that can easily be confused with bad pain. I was terrified at first that I was going to reinjure myself and was extremely cautious about any kind of back “pain” I experienced. Over time I figured out the difference between muscle soreness and real, “don’t do that” kind of pain. But it took me nearly a year to feel confident that I can (mostly) tell the difference.
  • I will never be able to perform a new exercise perfectly the first time. And I really need to be ok with that. As a person who regularly masters new technical skills with little or no help from colleagues or friends, it’s regularly an embarrassing and humbling experience to go to the gym.
  • “Testing out” injuries to joints doesn’t help anything. I am able to hyperextend most of my joints, including my wrists. I’d injured my wrists and had to do pushups and things like that from my fists for many months. When I first discovered I was injured, I kept “testing” which likely was just reinjuring myself over and over. It took probably 8 weeks the first time, and maybe 4 weeks the second time to recover from wrist injuries enough that I could do open-handed pushups. Learning to avoid the avoidable reinjuries was extremely helpful.

So what are the kinds of facts that I wish I would have known earlier in my life?

  • “Lift with your legs” basically means “lift by squeezing your glutes.” These are usually the largest muscles in your body, and basically they are your butt muscles. As my glutes have gotten stronger, my back pain has entirely gone away. In the last two years, I have had two incidents requiring more than a couple hours worth of recovery time. Previous to weight lifting, I’d needed sometimes 5 days to be able to stand up properly from a couch. While injured, I’d had to roll myself down to the floor and slowly push my entire body up, barely able to stand. Of course, this is specific to the injury I have. However, a lot of people sit in terrible chairs, hunched over computers all day and they could really benefit from learning how to strengthen their glutes.
  • Increasing glute strength through barbell lifting from a standing position (squats, deadlifts mostly) improved my posture dramatically. I now get people saying things in passing like “you have great posture”, something that I NEVER heard for the first 35 or so years of my life.
  • I need a lot of protein first thing in the morning to feel well through out the day. I eat two eggs every morning, usually also some yogurt and fruit. Cereal does not cut it. Oatmeal can sometimes be ok if combined with a bunch of other things.
  • Pain does not necessarily mean you are injured. You might just need to stretch something out.
  • Lifting weights when you feel like crap often will make you feel much much better.
  • It doesn’t take very long to get really strong. Within three months, I had dramatic improvements in basic stuff like being able to lift boxes of heavy stuff.
  • Grip strength improves amazingly as you increase the weight of barbell and dumbell lifts. I rarely feel pain in my hands and forearms anymore, and I have a pretty crushing grip.
  • I work best with minimal verbal cues, a lot of physical queues (poking muscles, mostly) and being able to see someone try to do something before I try it. As an obsessive reader, it never occurred to me that in the physical world, I would mostly require this kind of instruction. The bigger realization was probably that demonstration was extremely effective for me, more effective than: telling me to do something, being observed doing it, and getting feedback. Part of that is my a lack of gym vocabulary, but I also really need to see each exercise before I am able to mimic proper form (and ask a ton of questions).
  • Foam rollers are amazing. I foam roll a bunch of things before I work out and sometimes at home. Currently I’m focused on my IT-bands and quads, and sometimes I roll out parts of my arms. My elbows started pinging when I started increasing the weights I used for curls. After I was able to do curls with something like a 55-lb barbell, the tightness and pain went away.

Anyway, so that’s the stuff that I know now. I have said to myself that I wished I would have gotten into weight lifting earlier in life. But, to be honest, I don’t know that I would have been that into it without an enthusiastic teacher and a group of people encouraging me to do it. Now, I feel pretty good about independent physical activity, and I can afford to pay someone to help me stay on track.

I’m also extremely fortunate to be able to continue to lift while I’m quite pregnant. I’ve dramatically reduced the weight I lift for squats and deadlifts, after my doctor freaked out and told me I really shouldn’t be lifting so much.

If I ever get pregnant again, I might try to do some research and figure out what my real limits should be on this. Some current wisdom on this says that you should only do lifts that work your abdominals and lower body at about 60% of your max while pregnant. I have no idea where that comes from, and sincerely wish there was more than just hearsay and fear related to lifting while pregnant. There’s of course a lot of documentation of how bad injuries to a woman’s pelvic floor can be while pregnant, so there’s real science backing up the urge for caution. But if you’re otherwise very strong, I’m not sure that the freakouts about specific weight are warranted (if you’re healthy and not trying to achieve personal bests :D) any more than freaking out about women running until they give birth is called for.

I’ve been also thinking about the ways in which gym discipline has affected how I think about teaching and programming, but that’s a topic for another post!

Monitorama 2014 wrapup

I’m just settling back into the daily routine after RelEng/RelOps’ workweek and then Monitorama back-to-back.

Videos will eventually be posted here.

I thought it was awesome the conference started with some #hugops.

Here are my highlights:

  • I gave a talk about crontabber! I have my speakers notes if you’re interested!

  • Dan Slimmons gave a nice talk about basic probability and how understanding the difference between sensitivity and specificity can help you choose more useful alerts. It was super basic stats stuff, but a good foundation for building up stats competency in teams.

  • James Mickens gave a hilarious talk about the cloud that is well-worth finding when it goes up.

  • Ashe Dryden gave a talk about gender issues and “our most wicked problem”. It was very well-received by the audience, which was gratifying for me personally. I think the audience walked away with some very practical things to do: speak up among peers when someone says things that make you uncomfortable and ask questions about equal treatment in your company for things like salary, perks and benefits.

  • Several talks were given about monitoring and managing ops inside companies. My favorite was from Daniel Schauenberg (contributor to statsd) of Etsy. and Scott Sanders spoke about similar topics in this presentaton on Github’s outage lifecycle. And related, but not at the conference, Heroku just published an incident response runbook.

  • There was a hilarious lightning talk about the failure of the Swedish ship Vasa as an object lesson for massive project failure. Here’s a link to the case study the lightning talk was based on.

  • Larry Price (@laprice) gave a 5-minute talk about Postgres autovacuum tuning, which was awesome, and I hope he posts the slides. It reminded me that I should do a couple brownbags about Postgres config this summer!

  • I was struck by how many people said they used Postgres in production. Someone else asked the question during a talk, and nearly half the audience raised their hands.

  • InfluxDB, a new timeseries database emphasizing an HTTP API (remind anyone of CouchDB? :D), seemed interesting, although maybe rough around the edges when it came to documenting useful features/best practices. When I mentioned it on Twitter, I found a few folks already trying to use it in production and got at least one bug filed. :)

  • I also saw an amazing demo of Kibana, which seems like a very interesting dashboard/investigation/querying interface to Elastic Search. I watched a friend deploy it in about an hour to look at their ES systems last Wednesday.

  • Dashing from Shopify was also very interesting, although a rubyist project, so not easy to integrate with our Pythonic world. However, putting on a contributor relations hat — it could be a wonderful and beautiful way for contributors to interact with our many APIs.

I’m looking forward to the videos coming out and a list of slide decks, as I missed a few talks during hallway track conversations. I met several people who are managing similar or larger event loads than we do with Socorro, so it was fun swapping stories and seeing how their software stacks are evolving. RabbitMQ was a weapon of choice for reporting environments, along with Storm. Lots of love for Kafka was out there for the people dealing with real-time customer response.

Overall, highly recommend attending Monitorama to dip a toe into the state of the art with regard to system operations, monitoring and ops management.

The Final Crontab: an introduction to crontabber

I gave a talk at Monitorama today about crontabber. (slides)

My coworker tells me that I left out the part of “why you should care” about crontabber from my first few slides. So here’s a list:

  • Retries jobs on failure automatically
  • Dependency-aware, and won’t execute child jobs that depend on parents that have failed
  • Nagios integration including support for WARNINGs and CRITICALs, and configurable escalation from WARNING to CRITICAL (e.g. 3 WARNINGS == CRITICAL).

Those three are probably the top features sysadmins who are not happy with how cron is managing jobs wish they had.

Crontabber needs at least Python 2.6, Postgres 9.2, is FOSS and being used in production. We’ve used a version of the code since February 2013, and currently have the python module version you can install with pip install crontabber is currently running in our stage environment.

Let us know what you think!

Release Engineering: A draft of an architecture diagram

One of the things that I like to do is create architecture diagrams of complicated systems.

We had Release Engineering and Release Operations in the Portland Mozilla office this week, providing a perfect opportunity to pick everyone’s brains about what the current state of our release infrastructure is like.

Behold: releng flow onepage

And here’s a version that includes some “tree closure reasons” in magenta:

Releng infra with tree closure reason codes

A tree closure is defined as an hg hook that prevents people from committing to a tree (like mozilla-central). It looks up status at to figure out whether or not the tree is closed, and this value is updated manually by “sheriffs” who track tree status.

And the an initial key to the tree closure reasons (the numbers on the magenta blobs), is documented on the Mozilla wiki.

The goal of this document was to take brain dump information from everyone in the meeting, and create a relationship diagram of all the systems that everyone here supports. As you can see, it is pretty complex.

What I took away from creating this was:

  • The cognitive load is very high for trying to diagnose the root cause for several kinds of tree closures.
  • People loved being able to look at how each systems related to the others.
  • No single person really had a model in their head of how everything represented in this diagram was related.

There’s a lot more work to do to link in documentation and create some related diagrams, which I’ll tackle next week. The kinds of questions I’d like to try to answer based on the information that I’ve gathered include:

  • How does my patch get a build created for it?
  • What single points of failure can we mitigate?
  • What kinds of resilience do we need for our typical transient failures?

I really enjoyed identifying sources of tree closure and the kinds of failures that cause it. These are the kinds of problems I love working on solving — complicated, often unpredictable and largely driven by the normal work that people need to do to get their jobs done. There’s rarely a simple solution to things like experimental patches taking down large portions of a build infrastructure, and how we solve, or at least mitigate, these problems is fascinating.