Looking for worthy “mistakes were made” stories

I’m giving a couple talks at OSCON this year! One of them is titled “Mistakes were made“, and I need a few good stories.

I’m focused on web operations, but really, any stories where there was a plan, and it went horribly wrong, would be great. And, I’d love to know whether whatever went wrong ultimately got fixed, hacked around or was just left as-is.