Wrap-up meetings provide feedback that teams need to improve

(originally posted on tech emma)

Nobody likes admit­ting mis­takes. Except this guy.

Getting peo­ple to go to a post-mortem meet­ing is easy. Getting peo­ple to par­tic­i­pate with­out a sense of impend­ing doom is hard. Most peo­ple don’t want to be there. They show up ready to fight or make excuses, with a pit in their stom­ach as they wait to talk about what went wrong.

So, how do you fix that pit-in-the-stomach feeling? We’ve worked on this a bit at Emma, and here’s my formula:

  • Set high-level, achiev­able goals and have meet­ings even when things go right.
  • Focus on how every­one will work together to make things bet­ter in the future, not what went wrong.
  • Get every­one to participate.
  • Share with the whole com­pany what the group learned.

Now might be a good time to tell you that I wrote about some changes to our inter­nal down­time process last week (read that post here); today I’d like to fol­low up with details about our ver­sion of a post-mortem meeting.

Set high-level, achiev­able goals and meet about success

A main­te­nance win­dow here is con­sid­ered a suc­cess when we make our changes, recover from any fail­ures with­out impact­ing pro­duc­tion and end on time.

As a group, we decided what’s okay to include in the win­dow, and stripped out some riskier changes. Those included tasks that were hard to esti­mate time for, or ones that would push against the amount of time we allo­cated for test­ing. At this point, going into each win­dow, we have a clear list of tasks, and we can assess suc­cess or fail­ure of each task after the change.

In that first win­dow in January, we com­pleted the following:

  • Upgraded our PostgreSQL databases
  • Recovered 5% of the disk space on our largest data­base cluster
  • Fixed a long-standing main­te­nance issue with par­ent tables on our largest database

We decided to have a meet­ing after the win­dow — regard­less of whether the change suc­ceeded or failed.

Talk about what went well (aka Why I decided to call these meet­ings “wrap-ups”)

I always hated call­ing these dis­cus­sions “post-mortems.” I get why tech peo­ple want to com­pare the process to a med­ical pro­ce­dure, and I love a good zom­bie movie, but it sets the wrong tone. I decided to call them “wrap-ups,” to help make it clear that we’re there to reflect on the project, not find blame.

And here’s what we try to do in each wrap-up:

  • Spend time talk­ing about how things went well, and why
  • Focus on how to improve future projects
  • Distill what we learned

Documenting how the team man­ages main­te­nance win­dows makes the great work peo­ple were already doing vis­i­ble. We also open up the meet­ings so non-IT folks at Emma can con­tribute and make them better.

Conduct the dis­cus­sion for 100% participation

After a main­te­nance win­dow, we com­mu­ni­cate the out­come to the rest of our col­leagues. Then, I sched­ule a 30-minute meet­ing with a sim­ple agenda. We go over what hap­pened dur­ing the main­te­nance win­dow to:

  • Discuss what went right
  • Discuss what went wrong
  • And deter­mine what we could do to make things bet­ter next time

In our most recent wrap-up, seven peo­ple attended, and I requested at least one com­ment from each per­son on the agenda bul­let points.

What we learned

In just 30 min­utes, we came up with plenty of things that the group felt good about doing well and a set of clear changes to make in the future.

Here are some of the things peo­ple liked:

  • Creating a cus­tom error mes­sage for the main­te­nance window
  • Having a phone bridge and using Campfire through­out the win­dow to communicate
  • Using a wiki page to orga­nize tasks and each task’s owner dur­ing the main­te­nance window
  • Using the change win­dow to test out new Linux ser­vice scripts for the sys­tem admin­is­tra­tion team

This was our first main­te­nance win­dow where we used both Campfire and a phone bridge at the same time for the whole team. We chose Campfire because any­one new who joined could eas­ily see what con­ver­sa­tion had already taken place. We used the phone bridge to make it sim­ple to type com­mands and stay in touch at the same time.

In the past, we’d used email and RT tick­ets to doc­u­ment what was hap­pen­ing in the main­te­nance win­dow. Everyone loved hav­ing a wiki page to ref­er­ence and update instead. The wiki just had a bet­ter UI than email or a ticket, and pro­vided a bet­ter experience.

Finally, the sys­tems admin­is­tra­tion team used the win­dow to test out new ser­vice start/stop scripts for a series of cus­tom appli­ca­tions. This is the type of thing that can go un-exercised when you rarely have down­times or main­te­nance win­dows. The team was smart to seize the opportunity!

We also thought a few things didn’t go so well:

  • We didn’t give our cus­tomers enough of a heads-up.
  • Steps for the changes should have num­bers, not just times asso­ci­ated with them.
  • Our test­ing took quite a while because the change affected all the data­bases at the same time, and tests only looked at one data­base at a time.

There may have been other things that peo­ple thought we could have done bet­ter, but we kept the list short and action­able. We’ll change the process slightly in the future to inform cus­tomers bet­ter, add num­bers to all the steps and test data­bases concurrently.

Beyond this cur­rent win­dow, I also asked every­one to imag­ine how we might do things dif­fer­ently or bet­ter dur­ing other downtimes.

A few ideas included:

  • Trying out video con­fer­enc­ing dur­ing the main­te­nance, like Tokbox, to help make com­mu­ni­ca­tion even better
  • Pulling in more helpers for test­ing — for train­ing, and mak­ing the work­load lighter for the QA team
  • Using Salesforce to com­mu­ni­cate upcom­ing changes internally

My favorite sug­ges­tion, though, was:

Feel free to com­ment below — I’d love to hear how you man­age your meet­ings, and what you’ve learned.

twittering on 2008-03-11

  • i can’t wait to see @notbenh ride the moose. #
  • @radiogretchen YES. always works. esp with grumpy-pants. #
  • hah! tech group convergence: perlmongers will be at the lucky lab at the same time as werewolf #
  • maybe i can finally get that book from @stevelle #
  • behold: MacOSX support for ptop http://tinyurl.com/2wbgo2 #
  • and per @petdance, next feature for ptop will be full support for remotely connecting to DBs. possible w/ current app, but its not pretty :/ #
  • @linuxpoet mark & i were talking about writing some stored procs #
  • congrats, @peat!! #
  • @jabancroft its good to hear that you’re enjoying sxsw. the blog posts about mean crowd behavior make me sad. #
  • i don’t want a nxnwi (better name TBD) to either bring to portland or be associated with mobs of assholes. #
  • getting coffee, swapping out ppc for intel mac pro this morning. #
  • @aurynn and 4GB ram too. i am a lucky sysadmin. #
  • @jabancroft you should be sorry! 😉 i’ve been enjoying @geekygirldawns posts about BarCampAustin. #
  • @grigs check yr email 🙂 #
  • wishing i had something like the sxsw scheduler for my conference. #
  • @aurynn it’s worth it in the end! hang in there. #
  • @aurynn do they have co-working spaces in Calgary? i’m guessing not — i’m from kalispell, MT 🙂 #
  • @aurynn total bummer. co-working in portland is ~$250/mo at cubespace. lots of sharable loft space too. #
  • @aurynn my husband said that if i ever worked from home, that i *must* have an office outside the house 3 (or more) days/week. #
  • @portvangelistas – did you hear that? i just recruited a canadian! #
  • @bryanstearns – so i hear that you have some cool conference scheduling software? 🙂 #
  • @xolotl yes. others are what are commonly referred to as "groupies". #
  • @mobilehippie *sigh* yeah, yeah. #
  • @robotadam thx for the pointer to pycon’s scheduler! looks nice. #
  • hi @zbskii! what was the database issue? #
  • use postgresql? in the south florida area? SPUG is recruiting! check out the poll for a meeting location: http://snurl.com/21j6f #

twittering on 2008-03-10

  • @verso use ‘TechCrunch’ as your invite code 🙂 #
  • funny, the response to sarah lacy (cruel/emotional heckling) happens on IRC during conferences all the time. not just to women. #
  • just read @grigs response on crunchgear. thanks for writing that, jason. #
  • @grigs your comment on the article. i think the tone was spot on. ppl need to be called out on the sexism. sure, the interview wasn’t good – #
  • @grigs good for SXSW, but the blogging about it has largely been ridiculous. i thought this was good: http://snurl.com/21fcq #
  • wondering if @linuxpoet has been spared the plague. hoping his little ones don’t have scarlet fever! #
  • feeling victorious. #
  • @mjollnir THANK YOU. #
  • @andeezy yup. but the silver lining is that it WAS out in the public, so there was a public response that the heckling was bad. #
  • @andeezy yeah, it has already started. a couple people pointed out that SXSW should have done a better job curating. #
  • @andeezy lol. right. its pretty childish. i guess ppl think its fine to throw stones when they’re hiding behind a mob or on the interweb. #
  • drupalverse: is it possible i just display a menu as a page? without using copy and paste? thx. #
  • @xolotl thanks! checking it out…. #
  • @xolotl fixing the postgres table definition for panels…. 🙂 #

twittering on 2008-03-07

  • @geekygirldawn just portvangelista’d! #
  • @gwalter, @xolotl drupal also supports postgresql! mostly! #
  • jealous of @br3nda’s camping experience. will summer never arrive? #
  • hurting myself laughing at the latest stuffwhitepeoplelike #
  • a postgresql song: http://tinyurl.com/2vfwkj #
  • whoa! good morning, @theory! #
  • @linuxpoet GOOD MORNING! #
  • i really think that thursday night upgrades are the best. #
  • hi @aurynn! found any good pants yet? #
  • oh geez, i saw that this morning, wondered what it was http://tinyurl.com/277otz #
  • @ahockley a souffle? #
  • hi @ebeckman! #
  • @aurynn why are you learning calculus? #
  • @aurynn there’s an illustrated calculus book that i can’t remember the name of. i’ll go browsing at powells and see if i can find it again. #
  • w00t! used ptop to troubleshoot a stalled query today! #
  • hey @betsywhim, @thesethings and i are giving a wordpress v. drupal talk at LUGRadio Live! #
  • sounds like i need to listen in on @grigs too. #
  • oops my drupal/wordpress comment should have gone to @mobilehippie #
  • @metafluence i remember helping troubleshoot some mac printing problems for the NXNW people a few years ago #
  • @metafluence that’s sad that they just absorbed nxnw. all ur XWs are belong to us. #
  • @metafluence – you had me at bacon. i’m in for reviving NXNW too. #
  • i wish there was an ubuntu superhero that could fix @linuxpoet’s problem. #
  • @ljbanks obvious bene: it is pretty sweet not to have to create new passwords and accounts. also nice to tie identity to a central source. #
  • @linuxpoet don’t make me start a twitterfight about which city has more tech cred! #realtalk #
  • @mtrichardson, @turoczy, @metafluence maybe we should take a cue from portvangelistas and rebrand nxnw as NXNWi. domain is available! #
  • nxnwi.com snagged! should i install drupal? oh wait, already did. just waiting for DNS to propagate… #
  • @theory revival of NXNW – but with an "interactive" focus. last NXNW (circa 2000) was a bust. @turoczy and @metafluence started the convo #
  • sounds like #SXSW should get the 411 from SCaLE on registration logistics #
  • @xolotl NXNWi appreciates portvangelistas overture. oh, i’m sure we’ll find a use for you. #
  • hi @jkwatson! #
  • @notbenh i’ll get back to you ~3:20. need to get word from the old man re: dinner plans. otherwise, i am down. #
  • @kenefsky cold shower! gah. #
  • @kenefsky maybe we can just ask them for permission? #
  • @spinnerin that’s the spirit! use it or lose it! #
  • @sarahgilbert how do we do it??! #
  • alright, maybe we should just pick a different, Better name for NXNWi. #
  • i mean, i know that *we* can live this: http://snurl.com/218oydown, but do we want to pay SXSW to do it? #
  • (in previous tweet, meant "we can live this down") #
  • @adamd: it totally would be on the east side. we’d let the westside come over for a visit. #
  • @mattking damn that dreamhost one-clicky siren! #
  • @ahockley you are speaking my language. #

twittering on 2008-03-06

  • looking forward to @notbenh’s Moose talk at PDXPM http://snurl.com/212ih #
  • @marshallk 🙂 what’s the original list look like? #
  • @ryansnyder yeah, that sunrise was amazing this morning. i wish i still lived in SE and could do a Mt Tabor run on mornings like these. #
  • @radiogretchen i prefer sysadmin, and the answer is yes 🙂 #
  • i <3 Pages. and coffee. #
  • @oleoptene i will take a solo photo next time we meet 🙂 #
  • @spinnerin YES! I want turkeys as well. Guard Turkeys. #
  • can’t wait for chocolate and beer with @gorthx this evening. #
  • @karolcooks well, a girl has to live! the trick is, a run just before. #
  • @xolotl is it possible to be both a portvangelist and a portvangelista? could i switch between w/ costume changes or a trip over a bridge? #
  • @lelonopo: love is a battlefield is the best karaoke song ever. #
  • just submitted a post to #portvangelista. xo drupal. http://portvangelista.com/ #
  • @spinnerin check out @xolotl’s commentary. also, the magic of drupal. #
  • @linuxpoet we will! #
  • @andeezy, your wish granted: http://snurl.com/214w5 #
  • @linuxpoet i use kerberos and PAM. #
  • @xolotl you’re welcome! i could not resist "faster, cooler and better looking." #
  • @spinnerin i got a replacement cord at the apple store for my mac. #
  • @spinnerin rub some cayenne pepper on it! muy caliente! #
  • @xolotl dammit. tricked by marketing YET AGAIN. #
  • i dub @xolotl the official Portvangelista Propagandist. #
  • @kenefsky dunno. i dump out my todo list via IM periodically. but once i finish a task, i’m happy not to see it again 🙂 #
  • eating carrots. #
  • @br3nda bash it with a rock!! #

twittering on 2008-03-05

  • woot! first sponsorship for #barcampportland 2 pledged! #
  • its a crazy coincidence that huckabee and bill clinton grew up in the same town (hope, AK) #
  • oops.. hope, r #
  • @linuxpoet barcamp is an un-conference. the attendees set the agenda. @aurynn sent you to the right place 🙂 #
  • working on a fun idea for warranty registration. xo US postal service. #
  • @linuxpoet ? no, i don’t think so. we should still do talks/tutorials. i don’t know that the barcamp format would work for pgday. #
  • @karolcooks holla! i’m in it with you – day 4. #
  • @linuxpoet database topics are so special-interest. not sure there’d be enough interest in a database free-for-all. #
  • @karolcooks oh yeah, dude. i’m gonna be in fighting shape this summer. making it not so hard to hall ass up the climbing wall -pullups! #
  • coworker taunting me with Arial #
  • @muckp its a promo to get people to register product. we’re gonna send people something fun, but its a secret for a little while longer. #
  • @spinnerin that donut opening scene just made me SO HUNGRY. #
  • @muckp that’s probably about right. i have a few things to rig up to make the postage stuff work out right. #
  • favorite use for crockpot: making stuffing. thx for the cookbook ref, @spinnerin & @sarahgilbert #
  • @verso i’ll take one! #
  • looking forward to enjoying my lunch outside in the sunshine. #
  • @jabancroft @verso just hooked me up with jaiku! #
  • @andeezy stopping myself from making jezebel-style commentary on the threat/promise. #
  • @br3nda is feeling the #postgresql love #