[berlin] TaskCluster Platform: A Year of Development

Back in September, the TaskCluster Platform team held a workweek in Berlin to discuss upcoming feature development, focus on platform stability and monitoring and plan for the coming quarter’s work related to Release Engineering and supporting Firefox Release. These posts are documenting the many discussions we had there.

Jonas kicked off our workweek with a brief look back on the previous year of development.

Prototype to Production

In the last year, TaskCluster went from an idea with a few tasks running to running all of FirefoxOS aka B2G continuous integration, which is about 40 tasks per minute in the current environment.

Architecture-wise, not a lot of major changes were made. We went from CloudAMQP to Pulse (in-house RabbitMQ). And shortly, Pulse itself will be moving it’s backend to CloudAMQP! We introduced task statuses, and then simplified them.

On the implementation side, however, a lot changed. We added many features and addressed a ton of docker worker bugs. We killed Postgres and added Azure Table Storage. We rewrote the provisioner almost entirely, and moved to ES6. We learned a lot about babel-node.

We introduced the first alternative to the Docker worker, the Generic worker. We for the first time had Release Engineering create a worker, the Buildbot Bridge.

We have several new users of TaskCluster! Brian Anderson from Rust created a system for testing all Cargo packages for breakage against release versions. We’ve had a number of external contributors create builds for FirefoxOS devices. We’ve had a few Github-based projects jump on taskcluster-github.

Features that go beyond BuildBot

One of the goals of creating TaskCluster was to not just get feature parity, but go beyond and support exciting, transformative features to make developer use of the CI system easier and fun.

Some of the features include:

Features coming in the near future to support Release

Release is a special use case that we need to support in order to take on Firefox production worload. The focus of development work in Q4 and beyond includes:

  • Secrets handling to support Release and ops workflows. In Q4, we should see secrets.taskcluster.net go into production and UI for roles-based management.
  • Scheduling support for coalescing, SETA and cache locality. In Q4, we’re focusing on an external data solution to support coalescing and SETA.
  • Private data hosting. In Q4, we’ll be using a roles-based solution to support these.

TaskCluster Platform: 2015Q3 Retrospective

Welcome to TaskCluster Platform’s 2015Q3 Retrospective! I’ve been managing this team this quarter and thought it would be nice to look back on what we’ve done. This report covers what we did for our quarterly goals. I’ve linked to “Publications” at the bottom of this page, and we have a TaskCluster Mozilla Wiki page that’s worth checking out.

High level accomplishments

  • Dramatically improved stability of TaskCluster Platform for Sheriffs by fixing TreeHerder ingestion logic and regexes, adding better logging and fixing bugs in our taskcluster-vcs and mozilla-taskcluster components
  • Created and Deployed CI builds on three major platforms:
    • Added Linux64 (CentOS), Mac OS X cross-compiled builds as Tier2 CI builds
    • Completed and documented a prototype Windows 2012 builds in AWS and task configuration
  • Deployed auth.taskcluster.net, enabling better security, better support for self-service authorization and easier contributions from outside our team
  • Added region biasing based on cost and availability of spot instances to our AWS provisioner
  • Managed the workload of two interns, and significantly mentored a third
  • Onboarded Selena as a new manager
  • Held a workweek to focus attention on bringing our environment into production support of Release Engineering

Goals, Bugs and Collaborators

We laid out our Q3 goals in this etherpad. Our chosen themes this quarter were:

  • Improve operational excellence — focus on sheriff concerns, data collection,
  • Facilitate self-serve consumption — refactoring auth and supporting roles for scopes, and
  • Exploit opportunities to differentiate from other platforms — support for interactive sessions, docker images as artifacts, github integration and more blogging/docs.

We had 139 Resolved FIXED bugs in TaskCluster product.

Link to graph of resolved bugs

We also resolved 7 bugs in FirefoxOS, TreeHerder and RelEng products/components.

We received significant contributions from other teams: Morgan (mrrrgn) designed, created and deployed taskcluster-github; Ted deployed Mac OS X cross compiled builds; Dustin reworked the Linux TC builds to use CentOS, and resolved 11 bugs related to TaskCluster and Linux builds.

An additional 9 people contributed code to core TaskCluster, intree build scripts and and task definitions: aus, rwood, rail, mshal, gerard-majax, mihneadb@gmail.com, htsai, cmanchester, and echen.

The Big Picture: TaskCluster integration into Platform Operations

Moving from B2G to Platform was a big shift. The team had already made a goal of enabling Firefox Release builds, but it wasn’t entirely clear how to accomplish that. We spent a lot of this quarter learning things from RelEng and prioritizing. The whole team spent the majority of our time supporting others use of TaskCluster through training and support, developing task configurations and resolving infrastructure problems. At the same time, we shipped docker-worker features, provisioner biasing and a new authorization system. One tricky infra issue that John and Jonas worked on early in the quarter was a strange AWS Provisioner failure that came down to an obscure missing dependency. We had a few git-related tree closures that Greg worked closely on and ultimately committed fixes to taskcluster-vcs to help resolve. Everyone spent a lot of time responding to bugs filed by the sheriffs and requests for help on IRC.

It’s hard to overstate how important the Sheriff relationship and TreeHerder work was. A couple teams had the impression that TaskCluster itself was unstable. Fixing this was a joint effort across TreeHerder, Sheriffs and TaskCluster teams.

When we finished, useful errors were finally being reported by tasks and starring became much more specific and actionable. We may have received a partial compliment on this from philor. The extent of artifact upload retries, for example, was made much clearer and we’ve prioritized fixing this in early Q4.

Both Greg and Jonas spent many weeks meeting with Ed and Cam, designing systems, fixing issues in TaskCluster components and contributing code back to TreeHerder. These meetings also led to Jonas and Cam collaborating more on API and data design, and this work is ongoing.

We had our own “intern” who was hired on as a contractor for the summer, Edgar Chen. He did some work with the docker-worker, implementing Interactive Sessions, and did analysis on our provisioner/worker efficiency. We made him give a short, sweet presentation on the interactive sessions. Edgar is now at CMU for his sophomore year and has referred at least one friend back to Mozilla to apply for an internship next summer.

Pete completed a Windows 2012 prototype build of Firefox that’s available from Try, with documentation and a completely automated process for creating AMIs. He hasn’t created a narrated video with dueling, British-English accented robot voices for this build yet.

We also invested a great deal of time in the RelEng interns. Jonas and Greg worked with Anhad on getting him productive with TaskCluster. When Anthony arrived, we also onboarded him. Jonas worked closely to get him working on a new project, hooks.taskcluster.net. To take these two bits of work from RelEng on, I pushed TaskCluster’s roadmap for generic-worker features back a quarter and Jonas pushed his stretch goal of getting the big graph scheduler into production to Q4.

We worked a great deal with other teams this quarter on taskcluster-github, supporting new Firefox and B2G builds, RRAs for the workers and generally telling Mozilla about TaskCluster.

Finally, we spent a significant amount of time interviewing, and then creating a more formal interview process that includes a coding challenge and structured-interview type questions. This is still in flux, but the first two portions are being used and refined currently. Jonas, Greg and Pete spent many hours interviewing candidates.

Berlin Work Week

TaskCluster Platform Team in Berlin

Toward the end of the quarter, we held a workweek in Berlin to focus our next round of work on critical RelEng and Release-specific features as well as production monitoring planning. Dustin surprised us with delightful laser cut acrylic versions of the TaskCluster logo for the team! All team members reported that they benefited from being in one room to discuss key designs, get immediate code review, and demonstrate work in progress.

We came out of this with 20+ detailed documents from our conversations, greater alignment on the priorities for Platform Operations and a plan for trainings and tutorials to give at Orlando. Dustin followed this up with a series of ‘TC Topics’ Vidyo sessions targeted mostly at RelEng.

Our Q4 roadmap is focused on key RelEng features to support Release.


Our team published a few blog posts and videos this quarter:

TaskCluster migration: about the Buildbot Bridge

Back on May 7, Ben Hearsum gave a short talk about an important piece of technology supporting our transition to TaskCluster, the Buildbot Bridge. A recording is available.

I took some detailed notes to spread the word about how this work is enabling a great deal of important Q3 work like the Release Promotion project. Basically, the bridge allows us to separate out work that Buildbot currently runs in a somewhat monolithic way into TaskGraphs and Tasks that can be scheduled separately and independently. This decoupling is a powerful enabler for future work.

Of course, you might argue that we could perform this decoupling in Buildbot.

However, moving to TaskCluster means adopting a modern, distributed queue-based approach to managing incoming jobs. We will be freed of the performance tradeoffs and careful attention required when using relational databases for queue management (Buildbot uses MySQL for it’s queues, TaskCluster uses RabbitMQ and Azure). We also will be moving “decision tasks” in-tree, meaning that they will be closer to developer environments and likely easier to manage keeping developer and build system environments in sync.

Here are my notes:

Why have the bridge?

  • Allows a graceful transition
  • We’re in an annoying state where we can’t have dependencies between buildbot builds and taskcluster tasks. For example: we can’t move firefox linux builds into taskcluster without moving everything downstream of those also into taskcluster
  • It’s not practical and sometimes just not possible to move everything at the same time. This let’s us reimplement buildbot schedulers as task graphs. Buildbot builds are tasks on the task graphs enabling us to change each task to be implemented by a Docker worker, a generic worker or anything we want or need at that point.
  • One of the driving forces is the build promotion project – the funsize and anti-virus scanning and binary moving – this is going to be implemented in taskcluster tasks but the rest will be in Buildbot. We need to be able to bounce between the two.

What is the Buildbot Bridge (BBB)

BBB acts as a TC worker and provisioner and delegates all those things to BuildBot. As far as TC is concerned, BBB is doing all this work, not Buildbot itself. TC knows nothing about Buildbot.

There are three services:

  • TC Listener: responds to things happening in TC
  • BuildBot Listener: responds to BB events
  • Reflector: takes care of things that can’t be done in response to events — it reclaims tasks periodically, for example. TC expects Tasks to reclaim tasks. If a Task stops reclaiming, TC considers that Task dead.

BBB has a small database that associates build requests with TC taskids and runids.

BBB is designed to be multihomed. It is currently deployed but not running on three Buildbot masters. We can lose an AWS region and the bridge will still function. It consumes from Pulse.

The system is dependent on Pulse, SchedulerDB and Self-serve (in addition to a Buildbot master and Taskcluster).

Taskcluster Listener

Reacts to events coming from TC Pulse exchanges.

Creates build requests in response to tasks becoming “pending”. When someone pushes to mozilla-central, BBB inserts BuildRequests into BB SchedulerDB. Pending jobs appear in BB. BBB cancels BuildRequests as well — can happen from timeouts, someone explicitly cancelling in TC.

Buildbot Listener

Responds to events coming from the BB Pulse exchanges.

Claims a Task when builds start. Attaches BuildBot Properties to Tasks as artifacts. Has a buildslave name, information/metadata. It resolves those Tasks.

Buildbot and TC don’t have a 1:1 mapping of BB statuses and TC resolution. Also needs to coordinate with Treeherder color. A short discussion happened about implementing these colors in an artifact rather than inferring them from return codes or statuses inherent to BB or TC.


  • Runs on a timer – every 60 seconds
  • Reclaims tasks: need to do this every 30-60 minutes
  • Cancels Tasks when a BuildRequest is cancelled on the BB side (have to troll through BB DB to detect this state if it is cancelled on the buildbot side)


  • A successful build!

Task is created. Task in TC is pending, nothnig in BB. TCListener picks up the event and creates a BuildRequest (pending).

BB creates a Build. BBListener receives buildstarted event, claims the Task.

Reflector reclaims the Task while the Build is running.

Build completes successfully. BBListener receives log uploaded event (build finished), reports success in TaskCluster.

  • Build fails initially, succeeds upon retry

(500 from hg – common reason to retry)

Same through Reflector.

BB fails, marked as RETRY BBListener receives log uploaded event, reports exception to Taskcluster and calls rerun Task.

BB has already started a new Build TCListener receives task-pending event, updates runid, does not create a new BuildRequest.

Build completes successfully Buildbot Listener receives log uploaded event, reports success to TaskCluster.

  • Task exceeds deadline before Build starts

Task created TCListener receives task-pending event, creates BuildRequest Nothing happens. Task goes past deadline, TaskCluster cancels it. TCListener receives task-exception event, cancels BuildRequest through Self-serve


  • TC deadline, what is it? Queue: a task past a deadline is marked as timeout/deadline exceeded

On TH, if someone requests a rebuild twice what happens? * There is no retry/rerun, we duplicate the subgraph — where ever we retrigger, you get everything below it. You’d end up with duplicates Retries and rebuilds are separate. Rebuilds are triggered by humans, retries are internal to BB. TC doesn’t have a concept of retries.

  • How do we avoid duplicate reporting? TC will be considered source of truth in the future. Unsure about interim. Maybe TH can ignore duplicates since the builder names will be the same.

  • Replacing the scheduler what does that mean exactly?

    • Mostly moving decision tasks in-tree — practical impact: YAML files get moved into the tree
    • Remove all scheduling from BuildBot and Hg polling

Roll-out plan

  • Connected to the Alder branch currently
  • Replacing some of the Alder schedulers with TaskGraphs
  • All the BB Alder schedulers are disabled, and was able to get a push to generate a TaskGraph!

Next steps might be release scheduling tasks, rather than merging into central. Someone else might be able to work on other CI tasks in parallel.

TaskCluster migration: a “hello, world” for worker task creator

On June 1, 2015, Morgan and Dustin presented an introduction to configuring and testing TaskCluster worker tasks. The session was recorded. Their notes are also available in an etherpad.

The key tutorial information centered on how to set up jobs, test/run them locally and selecting appropriate worker types for jobs.

This past quarter Morgan has been working on Linux Docker images and TaskCluster workers for Firefox builds. Using that work as an example, Morgan showed how to set up new jobs with Docker images. She also touched on a couple issues that remain, like sharing sensitive or encrypted information on publicly available infrastructure.

A couple really nice things:

  • You can run the whole configuration locally by copy and pasting a shell script that’s output by the TaskCluster tools
  • There are a number of predefined workers you can use, so that you’re not creating everything from scratch

Dustin gave an overview of task graphs using a specific example. Looking through the docs, I think the best source of documentation other than this video is probably the API documentation. The docs could use a little more narrative for context, as Dustin’s short talk about it demonstrated.

The talk closed with an invitation to help write new tasks, with pointers to the Android work Dustin’s been doing.

Migrating to Taskcluster: work underway!

Mozilla’s build and test infrastructure has relied on Buildbot as the backbone of our systems for many years. Asking around, I heard that we started using Buildbot around 2008. The time has come for a change!

Many of the people working on migrating from Buildbot to Taskcluster gathered all together for the first time to talk about migration this morning. (A recording of the meeting is available)

The goal of this work is to shut down Buildbot and identify a timeline. Our first goal post is to eliminate the Buildbot Scheduler by moving build production entirely into TaskCluster, and scheduling tests in TaskCluster.

Today, most FirefoxOS builds and tests are in Taskcluster. Nearly everything else for Firefox is driven by Buildbot.

Our current tracker bug is ‘Buildbot -> TaskCluster transition‘. At a high level, the big projects underway are:

We have quite a few things to figure out in the Windows and Mac OS X realm where we’re interacting with hardware, and some work is left to be done to support Windows in AWS. We’re planning to get more clarity on the work that needs to be done there next week.

The bugs identified seem tantalizingly close to describing most of the issues that remain in porting our builds. The plan is to have a timeline documented for builds to be fully migrated over by Whistler! We are also working on migrating tests, but for now believe the Buildbot Bridge will help us get tests out of the Buildbot scheduler, even if we continue to need Buildbot masters for a while. An interesting idea about using runner to manage hardware instead of the masters was raised during the meeting that we’ll be exploring further.

If you’re interested in learning more about TaskCluster and how to use it, Chris Cooper is running a training on Monday June 1 at 1:30pm PT.

Ping me on IRC, Twitter or email if you have questions!

pushlog from last night, a brief look at Try

One of the mysterious and amazing parts of Mozilla’s Release Engineering infrastructure is the Try server, or just “Try”. This is how Firefox and FirefoxOS developers can push changes to Mozilla’s large build and test system, made up of about 3000 servers at any time. There are a couple amazing things about this — one is that anyone can request access to push to try, not just core developers or Mozilla employees. It is a publicly-available system and tool. Second is that a remarkable amount of control is offered over which builds to produce and which tests are run. Each Try run could consume 300+ hours of machine time if every posible build and test option is selected.

This blog post is a brain dump from a few days of noodling and a quick review of of the pushlog for Try, which shows exactly the options developers are choosing for their Try runs.

To use Try, you need to include a string of configuration that looks something like this in your topmost hg commit:

try: -b do -p emulator,emulator-jb,emulator-kk,linux32_gecko,linux64_gecko,macosx64_gecko,win32_gecko -u all -t none

That’s a recommended string for B2G developers from the Sheriff best practices wiki page. If you’re interested in how this works, the code for the try syntax parser itself is here.

You can include a Try configuration string in an empty commit, or as the last part of an existing commit message. What most developers tell me they do is have a an empty commit with the Try string in it, and they remove the extra commit before merging a patch. From all the feedback I’ve read and heard, I think that’s probably what we should document on the wiki page for Try, and maybe have a secondary page with “variants” for those that want to use more advanced tooling. KISS rule seems to apply here.

If you’re a regular user of Try, you might have heard of the high scores tracker. What you might not know is that there is a JSON file behind this page and it contains quite a bit of history that’s used to generate that page. You can find it if you just replace ‘.html’ with ‘.json’.

Something about the 8-bit ambiance of this page that made me think of “texts from last night”. But in reality, Try is most busy during typical Pacific Time working hours.

The high scores page also made me curious about the actual Try strings that people were using. I pulled them all out and had a look at what config options were most common.

Of the 1262 pushes documented today in that file:

  • 760 used ‘-b do’ meaning both Debug and Opt builds are made. I wonder whether this should just be the default, or we should have some clear recomendations about what developers should do here.
  • 366 used ‘-p all’ meaning build on 28 platforms, and produce 28 binaries. Some people might intend this, but I wonder if some other default might be more helpful.
  • 456 used ‘-u all’ meaning that all the unit tests were run.
  • 1024 used ‘-t none’ reflecting the waning use of Talos tests.

I’m still thinking about how to use this information. I have a few ideas:

  • Change the defaults for a minimal try run
  • Make some commonly-used aliases for things like “build on B2G platforms” or “tests that catch a lot of problems”
  • Create a dashboard that shows TopN try syntax strings
  • update the parser to include more of the options documented on various wiki pages as environment variables

If you’re a regular user of Try, what would you like to see changed? Ping me in #releng, email or tweet your thoughts!

And some background on me: I’ve been working with the Release Engineering team since April 1, 2015, and most of that time so far was spent on #buildduty, a topic I’m planning to write a few blog posts about. I’m also having a look at planning for the Task Cluster migration (away from BuildBot), monitoring and developer environments for releng tooling. I’m also working on a zine to share at Whistler of what is going on when you push to Try. Finally, I stood up a bot for reporting alerts through AWS SNS and to be able to file #buildduty related bugzilla bugs.

My goal right now is to find ways of communicating about and sharing the important work that Release Engineering is doing. Some of that is creating tracker bugs and having meetings to spread knowlege. Some of that is documenting our infrastructure and drawing pictures of how our systems interact. A lot of it is listening and learning from the engineers who build and ship Firefox to the world.

My recent op-ed published about Portland and startups

I was featured in the Portland Business Journal last Friday! I wrote an essay on startups and the experiences of women in the Portland tech community that have caused me to not refer women into startups for jobs unless the startups are run by fellow PyLadies.

Some excerpts:

It takes more than one CEO’s alleged behavior to cause 56 percent of women to leave technology related fields by mid-career, according to a Harvard Business Review study. That’s twice the rate that men leave the tech industry.

After all, 63 percent of women in STEM industries (science, technology engineering and math) have experienced sexual harassment, according to a 2008 study.

I can’t recommend that women work for startups in Portland.

Startup funders should keep holding executives accountable. Company cultures grow from the seeds planted by their leaders.

These companies need [qualified HR, skilled with workforce diversity issues], and our tech leaders should demand it.

Read the whole thing at the Portland Business Journal’s site!

Some thoughts on handling harassment and toxic behavior privately

I’ve seen some calls for people like @shanley to handle their complaints about abuse and harassment privately, or maybe “more privately” than they have. So, today I just mused aloud for a few minutes on Twitter the thoughts I have about private handling of public acts of cruelty, harassment and abuse. Here’s those tweets, slightly edited, in a blog form. This is meant as a discussion of why public responses to public harassment are not only justified, but helpful. I don’t believe that those who are harassed are obligated in any way to respond publicly to harassment.

Why is it important to “handle in private” the responses to public odious, toxic, anti-social and harassing behavior? How does that help?

I believe in proportionate response. However, when the interactions are online and there is no physical public space, just “public media”, there’s a serious problem with the idea that a private response, particularly from the harassed, works at all.

My experience has been that private responses to people who harass me don’t help. Harassment continues, the harasser does not change. The only thing that has ever “changed behavior” is directing a complaint through the public or someone with authority over the harasser. And, that only worked for me a small percentage of the time that I asked for help. Asking for help has also increased harassment.

I’m not sold on the idea of private responses to public acts shared on the internet as “effective for behavior change”.

There’s also a difference between individual behavior change (personal reform), and cultural change (what we find acceptable as a group). Not everyone is aboard the cultural change train. And, I see twitter in particular as performance art, not a conversation happening “inside” a community.

The more time I spend thinking about this, the more I realize I haven’t spent enough time defining what my community is. Because for a long time I bought into the idea that there was some kind of global FOSS community protecting me, caring for me, backing me up.

But what I believe now is that I have a few close friends, some who know each other, some who don’t — but not a coherent community. And the reason why I don’t think there’s coherency, is because they don’t respond the way that a community does when there’s danger.

There is real, lasting damage done to people I care about by harassers and abusers. Things I hear about afterward, things that make me ill to think about and repeat. Things I wish I could prevent. That we could all prevent. But the worst damage done, I think, happens when those who come forward aren’t believed.

I didn’t go into detail about this in the twitter thread, but I have found that many people, who I come in contact with through FOSS, who are abusive on social media and on public mailing lists, are also abusive and harassing in private. Assuming that the public behavior is the worst behavior that a harasser exhibits is not a safe or reasonable assumption.

So. Handling things in public when we’re dealing with public acts, cruelty, harassment: Yes, I think we should.

Personal email sabbatical, July 10 – October 15 2014

Starting July 10, 2014, I’m planning to take a personal email sabbatical.

I’m going to unsubscribe from all mailing lists and redirect all email to my personal email addresses (chesnok.com and gmail.com) to /dev/null.

I have a couple open source project addresses that will probably be redirected to people responsible for those areas during my sabbatical. The traffic on those addresses is so low, however, that this is likely not going to impact anyone significantly.

I will not be reachable via personal email until sometime around October 15, 2014.

Why I’m doing this:

I spend too much time responding to things that other people want me to do.

I have a full-time job where I do that already, and as a result, much of my personal time where I could be creative is spent in the service of others. I’m also about to have a baby! So, it seems like as good a time as any to make an explicit departure from the communication medium that’s been the focus of my life since about 1995.

I’m not sure that I will return to having a public email address outside of work.

I plan to continue blogging, although using a much simplified blogging platform that I’ve written (but haven’t deployed yet :D). I plan to continue to use Twitter, although I’ve separated public and private accounts. I do not plan to increase my use of other social media. I plan to continue to use IRC, although will likely take myself out of most project channels. I’ve removed Twitter from my phone, and plan to remove email from it in July.

I’m taking a cue from danah boyd, although not following her prescription for a lengthy advance notice to collaborators.

I spent the last six months substantially downsizing my volunteer commitments, spending more time on in-person events that are local. I am pretty sure most people won’t even notice my extended volunteer absence on the “greater internet”. I’ll trickle out some individual notices as needed, probably to anyone I’ve had more than three exchanges with in the last two or three years.

Things I’ve learned from my personal trainer

I’ve been seeing a personal trainer for the last couple of years. I injured my back pretty seriously 5-6 years ago, and finally admitted that I had a “real problem” that required doctor and other professional intervention. After x-rays and a couple months more of denial, I started an exercise program focused on weight training.

Previously, my main (and sometimes only) exercise had been running. The back problem I have caused my doctor to recommend that I never run again. At first this was an easy thing to do because running was incredibly painful. Later, I realized that running was not only a fitness thing for me, but also an important part of my mental health. I really needed another way to get in a lot of exercise.

I go to the gym 3 times a week, and I try to walk about 5 miles a day at least 3 additional days a week. Playing Ingress, and bird watching tend to make this pretty easy.

I’m now 7-months pregnant, and I still do both things. I typically don’t get any 10k days in like I did before I was pregnant, but apart from some grueling travel I did in April, I’ve been able to keep it up the entire time.

I’ve threatened to give a talk about this, but rather than wait to assemble all of this until then, I thought it would be nice to just write it down in the event it helps someone else!

My physical activity background:

I did first ballet, then karate and judo as a child. I never played any ball or team sports, and did not learn to swim other than barely keep myself afloat until I was an adult. In high school, I ran cross country for three seasons. Mostly I played the violin, at least 2 hours a day sometimes up to 5 hours a day, which is a physically tiring activity, but not aerobic (usually). I also hiked and backpacked with my family. I was able to run a 7-7:30 minute mile until a couple years after I graduated from college. In general, I thought of myself as a pretty fit person, and ran slower because no coach was yelling at me and no competitors were chasing me as I got older. The last time I seriously “trained” was for a marathon I ended up not running in about 2005. I got myself up to 18 miles and then just cycled about 8 miles a day to and from work for many years.

So, by the time I decided to get a trainer I was not in very good shape.

Here’s the stuff that I learned right away:

  • I did not know how to lift things. “Lift with your legs” is what people say. I did not know what this really meant, and was doing some strange combination of using my quads and my lower back to lift heavy things. That’s pretty much exactly what you should not do.
  • My shoulders were incredibly weak. Part of that was from typing all day for my work and being tense. Another part was just lack of use.
  • My lack of grip strength was causing me a lot pain in my hands and forearms. I’d overuse my wrists and hands every day, and then go back and do it again. I didn’t quite have carpal tunnel, but it was probably coming pretty soon.
  • I was very embarrassed by my lack of basic knowledge in the gym. I’m pretty good with jargon, but holy shit, gym language was complicated. And didn’t map to anything I already knew. The names of the exercises were not helpful to me, and I found myself drawing lots of stick figure pictures to try to remember how to do things properly. I didn’t know the names of equipment, I couldn’t remember the difference between a barbell and a dumbbell, I could not for the life of me recall an exercise if I took more than about a week-long break from it.
  • Strengthening my back causes a lot of soreness that can easily be confused with bad pain. I was terrified at first that I was going to reinjure myself and was extremely cautious about any kind of back “pain” I experienced. Over time I figured out the difference between muscle soreness and real, “don’t do that” kind of pain. But it took me nearly a year to feel confident that I can (mostly) tell the difference.
  • I will never be able to perform a new exercise perfectly the first time. And I really need to be ok with that. As a person who regularly masters new technical skills with little or no help from colleagues or friends, it’s regularly an embarrassing and humbling experience to go to the gym.
  • “Testing out” injuries to joints doesn’t help anything. I am able to hyperextend most of my joints, including my wrists. I’d injured my wrists and had to do pushups and things like that from my fists for many months. When I first discovered I was injured, I kept “testing” which likely was just reinjuring myself over and over. It took probably 8 weeks the first time, and maybe 4 weeks the second time to recover from wrist injuries enough that I could do open-handed pushups. Learning to avoid the avoidable reinjuries was extremely helpful.

So what are the kinds of facts that I wish I would have known earlier in my life?

  • “Lift with your legs” basically means “lift by squeezing your glutes.” These are usually the largest muscles in your body, and basically they are your butt muscles. As my glutes have gotten stronger, my back pain has entirely gone away. In the last two years, I have had two incidents requiring more than a couple hours worth of recovery time. Previous to weight lifting, I’d needed sometimes 5 days to be able to stand up properly from a couch. While injured, I’d had to roll myself down to the floor and slowly push my entire body up, barely able to stand. Of course, this is specific to the injury I have. However, a lot of people sit in terrible chairs, hunched over computers all day and they could really benefit from learning how to strengthen their glutes.
  • Increasing glute strength through barbell lifting from a standing position (squats, deadlifts mostly) improved my posture dramatically. I now get people saying things in passing like “you have great posture”, something that I NEVER heard for the first 35 or so years of my life.
  • I need a lot of protein first thing in the morning to feel well through out the day. I eat two eggs every morning, usually also some yogurt and fruit. Cereal does not cut it. Oatmeal can sometimes be ok if combined with a bunch of other things.
  • Pain does not necessarily mean you are injured. You might just need to stretch something out.
  • Lifting weights when you feel like crap often will make you feel much much better.
  • It doesn’t take very long to get really strong. Within three months, I had dramatic improvements in basic stuff like being able to lift boxes of heavy stuff.
  • Grip strength improves amazingly as you increase the weight of barbell and dumbell lifts. I rarely feel pain in my hands and forearms anymore, and I have a pretty crushing grip.
  • I work best with minimal verbal cues, a lot of physical queues (poking muscles, mostly) and being able to see someone try to do something before I try it. As an obsessive reader, it never occurred to me that in the physical world, I would mostly require this kind of instruction. The bigger realization was probably that demonstration was extremely effective for me, more effective than: telling me to do something, being observed doing it, and getting feedback. Part of that is my a lack of gym vocabulary, but I also really need to see each exercise before I am able to mimic proper form (and ask a ton of questions).
  • Foam rollers are amazing. I foam roll a bunch of things before I work out and sometimes at home. Currently I’m focused on my IT-bands and quads, and sometimes I roll out parts of my arms. My elbows started pinging when I started increasing the weights I used for curls. After I was able to do curls with something like a 55-lb barbell, the tightness and pain went away.

Anyway, so that’s the stuff that I know now. I have said to myself that I wished I would have gotten into weight lifting earlier in life. But, to be honest, I don’t know that I would have been that into it without an enthusiastic teacher and a group of people encouraging me to do it. Now, I feel pretty good about independent physical activity, and I can afford to pay someone to help me stay on track.

I’m also extremely fortunate to be able to continue to lift while I’m quite pregnant. I’ve dramatically reduced the weight I lift for squats and deadlifts, after my doctor freaked out and told me I really shouldn’t be lifting so much.

If I ever get pregnant again, I might try to do some research and figure out what my real limits should be on this. Some current wisdom on this says that you should only do lifts that work your abdominals and lower body at about 60% of your max while pregnant. I have no idea where that comes from, and sincerely wish there was more than just hearsay and fear related to lifting while pregnant. There’s of course a lot of documentation of how bad injuries to a woman’s pelvic floor can be while pregnant, so there’s real science backing up the urge for caution. But if you’re otherwise very strong, I’m not sure that the freakouts about specific weight are warranted (if you’re healthy and not trying to achieve personal bests :D) any more than freaking out about women running until they give birth is called for.

I’ve been also thinking about the ways in which gym discipline has affected how I think about teaching and programming, but that’s a topic for another post!