- UpdatePDX tonight! At P.I.E. http://www.meetup.com/updatepdx/events/16416694/?a=md1.2p_evn&rv=md1.2p #
- I'm experiencing an eventsplosion. Not sure how I'm going to sleep for the next two weeks. #
- "You can't tune away the horror of 300-line SQL." -@lusciouspear #updatepdx #
- learned that we needed to hire CS people #updatepdx #nosql #
- "Once you stop answering email in 24-hours, they start sending more email." -@sarahnovotny #fail #updatepdx #
- "you need to check that you have backups BEFORE you do your maintenance." -@sarahnovotny #backups #updatepdx #
- "You don't have valid backups. You have valid *restores*." -@sarahnovotny #updatepdx #
- "Twitter is really a series of caches." -@al3x #updatepdx #
- "We had no idea how much we had to learn." -@al3x #updatepdx #
- many <3s for computer science education today #updatepdx #
- shout out for Pivotal Labs /cc @sarahmei 🙂 #updatepdx #
- "both readthrough and writethrough caching was happening at the same time." #fail #updatepdx #
- "But the issue was not technology, but responsibility in the organization." #fail #updatepdx #
- "Only use NOSQL if you reach a certain point of despair." -@timanglade #updatepdx #
- "If you ask for NOSQL by name, you probably don't need it. " -@timanglade #updatepdx #nosql #fail #
- "Usually they should just hire a MySQL consultant." -@timanglade #updatepdx #
- "Trust no one. that goes double if they talk about CAP." -@timanglade #updatepdx #
- I see a niche market for database and devops-specific comedy. #
- "Get a fresh set of eyes on the problem." (story of replacing data stored in Riak w/Postgres) #updatepdx re/collab while scaling @al3x #
- talking about learning culture, passion for tech and how to hire when jobs are no longer silo'd between dev, ops, dba #updatepdx #
Monthly Archives: April 2011
UpdatePDX: NOSQL, operational complexity and hiring
Last night, Tim Anglade, Bradford Stephens, Sarah Novotny and Alex Payne put together a three-part discussion to talk complexity, caching and collaboration, and in some cases, skewer popular notions of problem-solving around NOSQL.
Big thanks to Michael Schurter and Rick Turoczy for organizing and providing us space at PIE.
Tim was the thinker and designer behind the event. Many thanks to him for putting so much energy and time into it, and helping many leave inspired.
“Catchy phrases are red herrings.” -@timanglade
The night began with stories of failure: Bradford touched on the horror of 300-line SQL and the value of applying computer science to solve problems; Sarah recalled not having restorable data in an emergency because someone had tested a new backup strategy – for six weeks; Alex talked about the hard problems in computer science (ok, caching) and the value of culture and mentoring.
What I loved about this part of the evening, is that each person had a story. Technical presentations sometimes lack guideposts, but these stories all had villains, heros and a lesson.
“Usually they should just hire a MySQL consultant.” -@timanglade
And then Tim followed with a wickedly funny sendup of “the NOSQL movement”. The thing that struck a nerve and made the audience laugh uproariously was: “Only use NOSQL if you reach a certain point of despair.”
His other points included: Never forget the operational complexity; Some things will always be better achieved with an RDBMS; Distribution Model vs. Data Model vs. Disk Data Structure (invoking the Moon Methodology); Hardware will always help, but it will never save you; Given enough time, most NOSQL projects gravitate towards a MapReduce-like model for computations (and querying); Trust no one. That goes double if they talk about CAP.
And then we talked.
The audience had plenty of questions, starting with how do we address collaboration in the context of scalability?
The answers from the panel, disappointingly, seemed to came down to separation of databases and people. While it’s true that it is easier to give out more databases than trying to communicate, it shows how far we have to go as an industry. Another point made was that it’s not typically possible to repurpose a DBA to maintain something like a Hadoop cluster.
In my notes, I wrote: “when problems are of a certain size, and affects the DNA of a company.” And when that happens, the typical separation of responsibilities between developer, DBA and operations break down.
Alex made my night by mentioning that sometimes solving a problem just takes getting a fresh set of eyes on it. He talked about replacing a particular bit of technology by applying some features of Postgres, with the help of a new hire.
I asked the final question – how does a company hire for the types of skills needed to solve these types of problems? Most agreed that finding someone with a passion for learning was critical. There was a dissenting voice – I think Bradford’s. My notes trailed off at that point, unfortunately. My guess is that he spoke up for logic and patience, and that some of the problems companies face have been largely solved, if you spend the time to study the science.
Epilogue
We wrapped up the night out at Little Big Burger and the Teardrop. Several people commented on how lovely Portland was – even in the rain. But damn, they wish they could hire the types of people they needed here.
So, people who are interested in big data: consider that a call to action.
Visit us for OSCON Data this summer. Then, move to Portland.
twittering on 2011-04-04
- Three marketing areas where #PDX11 fails: http://chesnok.com/u/1f #
Three marketing areas where PDX11 fails
I had a short conversation yesterday about what I wish PDX11 was doing better with it’s marketing. I was the person who suggested we call this effort PDX11, and I bought the domains and asked the Open Source Lab to host our sites. I created the mailing lists and setup and ran the hackathon.
So, really, if anyone is to blame for this, it’s me. 🙂
But I’d like some help, so I’ll lay out to you what I think is wrong:
- Ensure that the Portland open source community knows what PDX11 is.
Work in progress by the Knowledge Network is starting to address this issue. But much of the open source community has no idea what PDX11 is, or why they should care about it. - Align some of the goals of open source community and the proprietary software community.
Right now, it seems as though the values of one community diverge greatly from the other.The Mentor network seems like the most likely place for the groups to come together, but there’s still quite a significant cultural gap between the Software Association of Oregon and a very large open source community. It makes me wonder if the SAO hasn’t realized that open source developers are a meaningfully large and growing group that they should be serving the needs of.
This reminds me of a post I did back in 2009 about growth in PostgreSQL job postings.
So I looked up the relative growth rate of jobs with “open source” in the description:
Seems like growth worth paying attention to, especially given that “Mobile App” has explosive growth, but still only represents .0007% of all job postings – whereas “open source” is in .004%:
- Marketing to the general public about why it’s important that the PDC focus effort on the software industry cluster.
A friend asked the other day “Why should a non-geek care about PDX11?” And, I didn’t have a snappy answer. One thing that was said today during the unconference that I’m still mulling over was: “Software is the last growth industry we have in the US.” That’s maybe too agressive. 🙂Talking with Audrey today, I learned that it’s an established practice to target growing industries in a municipal area for public investment, and software industry was identified through an objective measure to be clustering and growing at a rate that warranted the city’s encouragement. The PDC has a site that shows some stats, but it’s far from clear to me, even as a person working on this project, exactly what is relevant to the general public. A bunch of this data was put into a PDF. One key number is that there are over 1,400 software companies in Oregon, and employment in this industry grew 19.2% from 2004-2009.
So, we could use your help. How should we address this?
Why should a non-geek care about the City’s and software community’s efforts to bring more software industry to Portland?
Weekly tweet digest for 2011-04-03
- The struggle to right oneself http://flic.kr/p/9u6Dj6 #
- OH: "there's a troupe of mimes…" #planedelay #dc #italianopera #random #
- At donors choose http://flic.kr/p/9uqoeu #
- Current status: http://youtu.be/VqyqHZEDbRI #
- Looking forward to this Friday's PDX11 Civic Unconference: http://bit.ly/fAiVxl #
- Awww. I <3 you portland: http://bit.ly/fyIrKj #
- jealous of @debbryant and others at #NASAoss #
- Have you registered for PDX11 Civic Hackathon yet? Do it now so we have enough beer and food! http://pdx11unconference.eventbrite.com/ #
- More about #pdx11 civic unconference: http://pdx11.org/node/2919 #
- Set up an IRC channel for #pdx11 on freenode #
- 5 Google engineering management mistakes: http://chesnok.com/u/19 #
- Registrations keep rolling in for the #pdx11 unconference: http://pdx11unconference.eventbrite.com/ #
- i totally giggled about noops. http://blogs.forrester.com/mike_gualtieri/11-02-07-i_dont_want_devops_i_want_noops #
- Where meritocracy fails http://chesnok.com/u/1a (me writing out what i meant last week about this) #
- wow! the meritocracy post got linked on HN: http://news.ycombinator.com/item?id=2390757 #
- Not a joke! #PDX11 Civic Unconference kicks off tonight at 5pm: http://pdx11unconference.eventbrite.com/ #
- Setup for #pdx11 unconference http://t.co/nzg5MtZ #
- Tons of free food and beer at psu engineering bldg until 8pm. Stop by 🙂 #
- Pdx11 meetup http://flic.kr/p/9vbvK5 #
- Headed to #pdx11 afterparty at vendetta #
- Pdx tech community goes hard in the paint. #forreal #pdx11 #
- dangerously fun night with @po8, @WardCunningham, @mildwail and @kquetzl. looking forward to today's #pdx11 unconference and hackathon! #
- on my way to PSU #
- First #pdx11 unconf discussion: http://etherpad.opensourcebridge.org/who-wants-fiber-pdx11-unconference #
- Starting up the #pdx11 conversation about data.gov #
- etherpad for convo: http://etherpad.opensourcebridge.org/datagov-shutdown-pdx11-unconference #
- So, I registered a domain. Notes that lead to it here: http://etherpad.opensourcebridge.org/datagov-shutdown-pdx11-unconference #pdx11 #
- #pdx11 lunch is served! #
- Here's video from this morning's #pdx11 civic unconference kickoff: http://blip.tv/file/4967795 #
- Reminder about happy hour "suds and spirits" reception with the PDC April 6: http://pdx11.org/node/4017 #
- Great photos from Sam Churchill from today's #pdx11 event http://www.flickr.com/photos/samchurchill/sets/72157626415775186/ #
Weekly tweet digest for 2011-04-03
- The struggle to right oneself http://flic.kr/p/9u6Dj6 #
- OH: "there's a troupe of mimes…" #planedelay #dc #italianopera #random #
- At donors choose http://flic.kr/p/9uqoeu #
- Current status: http://youtu.be/VqyqHZEDbRI #
- Looking forward to this Friday's PDX11 Civic Unconference: http://bit.ly/fAiVxl #
- Awww. I <3 you portland: http://bit.ly/fyIrKj #
- jealous of @debbryant and others at #NASAoss #
- Have you registered for PDX11 Civic Hackathon yet? Do it now so we have enough beer and food! http://pdx11unconference.eventbrite.com/ #
- More about #pdx11 civic unconference: http://pdx11.org/node/2919 #
- Set up an IRC channel for #pdx11 on freenode #
- 5 Google engineering management mistakes: http://chesnok.com/u/19 #
- Registrations keep rolling in for the #pdx11 unconference: http://pdx11unconference.eventbrite.com/ #
- i totally giggled about noops. http://blogs.forrester.com/mike_gualtieri/11-02-07-i_dont_want_devops_i_want_noops #
- Where meritocracy fails http://chesnok.com/u/1a (me writing out what i meant last week about this) #
- wow! the meritocracy post got linked on HN: http://news.ycombinator.com/item?id=2390757 #
- Not a joke! #PDX11 Civic Unconference kicks off tonight at 5pm: http://pdx11unconference.eventbrite.com/ #
- Setup for #pdx11 unconference http://t.co/nzg5MtZ #
- Tons of free food and beer at psu engineering bldg until 8pm. Stop by 🙂 #
- Pdx11 meetup http://flic.kr/p/9vbvK5 #
- Headed to #pdx11 afterparty at vendetta #
- Pdx tech community goes hard in the paint. #forreal #pdx11 #
- dangerously fun night with @po8, @WardCunningham, @mildwail and @kquetzl. looking forward to today's #pdx11 unconference and hackathon! #
- on my way to PSU #
- First #pdx11 unconf discussion: http://etherpad.opensourcebridge.org/who-wants-fiber-pdx11-unconference #
- Starting up the #pdx11 conversation about data.gov #
- etherpad for convo: http://etherpad.opensourcebridge.org/datagov-shutdown-pdx11-unconference #
- So, I registered a domain. Notes that lead to it here: http://etherpad.opensourcebridge.org/datagov-shutdown-pdx11-unconference #pdx11 #
- #pdx11 lunch is served! #
- Here's video from this morning's #pdx11 civic unconference kickoff: http://blip.tv/file/4967795 #
- Reminder about happy hour "suds and spirits" reception with the PDC April 6: http://pdx11.org/node/4017 #
- Great photos from Sam Churchill from today's #pdx11 event http://www.flickr.com/photos/samchurchill/sets/72157626415775186/ #
twittering on 2011-04-03
- dangerously fun night with @po8, @WardCunningham, @mildwail and @kquetzl. looking forward to today's #pdx11 unconference and hackathon! #
- on my way to PSU #
- First #pdx11 unconf discussion: http://etherpad.opensourcebridge.org/who-wants-fiber-pdx11-unconference #
- Starting up the #pdx11 conversation about data.gov #
- etherpad for convo: http://etherpad.opensourcebridge.org/datagov-shutdown-pdx11-unconference #
- So, I registered a domain. Notes that lead to it here: http://etherpad.opensourcebridge.org/datagov-shutdown-pdx11-unconference #pdx11 #
- #pdx11 lunch is served! #
- Here's video from this morning's #pdx11 civic unconference kickoff: http://blip.tv/file/4967795 #
- Reminder about happy hour "suds and spirits" reception with the PDC April 6: http://pdx11.org/node/4017 #
- Great photos from Sam Churchill from today's #pdx11 event http://www.flickr.com/photos/samchurchill/sets/72157626415775186/ #
twittering on 2011-04-03
- dangerously fun night with @po8, @WardCunningham, @mildwail and @kquetzl. looking forward to today's #pdx11 unconference and hackathon! #
- on my way to PSU #
- First #pdx11 unconf discussion: http://etherpad.opensourcebridge.org/who-wants-fiber-pdx11-unconference #
- Starting up the #pdx11 conversation about data.gov #
- etherpad for convo: http://etherpad.opensourcebridge.org/datagov-shutdown-pdx11-unconference #
- So, I registered a domain. Notes that lead to it here: http://etherpad.opensourcebridge.org/datagov-shutdown-pdx11-unconference #pdx11 #
- #pdx11 lunch is served! #
- Here's video from this morning's #pdx11 civic unconference kickoff: http://blip.tv/file/4967795 #
- Reminder about happy hour "suds and spirits" reception with the PDC April 6: http://pdx11.org/node/4017 #
- Great photos from Sam Churchill from today's #pdx11 event http://www.flickr.com/photos/samchurchill/sets/72157626415775186/ #
twittering on 2011-04-02
- Not a joke! #PDX11 Civic Unconference kicks off tonight at 5pm: http://pdx11unconference.eventbrite.com/ #
- Setup for #pdx11 unconference http://t.co/nzg5MtZ #
- Tons of free food and beer at psu engineering bldg until 8pm. Stop by 🙂 #
- Pdx11 meetup http://flic.kr/p/9vbvK5 #
- Headed to #pdx11 afterparty at vendetta #
- Pdx tech community goes hard in the paint. #forreal #pdx11 #
twittering on 2011-04-01
- wow! the meritocracy post got linked on HN: http://news.ycombinator.com/item?id=2390757 #