twittering on 2011-04-05

  • UpdatePDX tonight! At P.I.E. http://www.meetup.com/updatepdx/events/16416694/?a=md1.2p_evn&rv=md1.2p #
  • I'm experiencing an eventsplosion. Not sure how I'm going to sleep for the next two weeks. #
  • "You can't tune away the horror of 300-line SQL." -@lusciouspear #updatepdx #
  • learned that we needed to hire CS people #updatepdx #nosql #
  • "Once you stop answering email in 24-hours, they start sending more email." -@sarahnovotny #fail #updatepdx #
  • "you need to check that you have backups BEFORE you do your maintenance." -@sarahnovotny #backups #updatepdx #
  • "You don't have valid backups. You have valid *restores*." -@sarahnovotny #updatepdx #
  • "Twitter is really a series of caches." -@al3x #updatepdx #
  • "We had no idea how much we had to learn." -@al3x #updatepdx #
  • many <3s for computer science education today #updatepdx #
  • shout out for Pivotal Labs /cc @sarahmei :) #updatepdx #
  • "both readthrough and writethrough caching was happening at the same time." #fail #updatepdx #
  • "But the issue was not technology, but responsibility in the organization." #fail #updatepdx #
  • "Only use NOSQL if you reach a certain point of despair." -@timanglade #updatepdx #
  • "If you ask for NOSQL by name, you probably don't need it. " -@timanglade #updatepdx #nosql #fail #
  • "Usually they should just hire a MySQL consultant." -@timanglade #updatepdx #
  • "Trust no one. that goes double if they talk about CAP." -@timanglade #updatepdx #
  • I see a niche market for database and devops-specific comedy. #
  • "Get a fresh set of eyes on the problem." (story of replacing data stored in Riak w/Postgres) #updatepdx re/collab while scaling @al3x #
  • talking about learning culture, passion for tech and how to hire when jobs are no longer silo'd between dev, ops, dba #updatepdx #

UpdatePDX: NOSQL, operational complexity and hiring

Last night, Tim Anglade, Bradford Stephens, Sarah Novotny and Alex Payne put together a three-part discussion to talk complexity, caching and collaboration, and in some cases, skewer popular notions of problem-solving around NOSQL.

Big thanks to Michael Schurter and Rick Turoczy for organizing and providing us space at PIE.

Tim was the thinker and designer behind the event. Many thanks to him for putting so much energy and time into it, and helping many leave inspired.

“Catchy phrases are red herrings.” -@timanglade

The night began with stories of failure: Bradford touched on the horror of 300-line SQL and the value of applying computer science to solve problems; Sarah recalled not having restorable data in an emergency because someone had tested a new backup strategy – for six weeks; Alex talked about the hard problems in computer science (ok, caching) and the value of culture and mentoring.

What I loved about this part of the evening, is that each person had a story. Technical presentations sometimes lack guideposts, but these stories all had villains, heros and a lesson.

“Usually they should just hire a MySQL consultant.” -@timanglade

And then Tim followed with a wickedly funny sendup of “the NOSQL movement”. The thing that struck a nerve and made the audience laugh uproariously was: “Only use NOSQL if you reach a certain point of despair.

His other points included: Never forget the operational complexity; Some things will always be better achieved with an RDBMS; Distribution Model vs. Data Model vs. Disk Data Structure (invoking the Moon Methodology); Hardware will always help, but it will never save you; Given enough time, most NOSQL projects gravitate towards a MapReduce-like model for computations (and querying); Trust no one. That goes double if they talk about CAP.

And then we talked.

The audience had plenty of questions, starting with how do we address collaboration in the context of scalability?

The answers from the panel, disappointingly, seemed to came down to separation of databases and people. While it’s true that it is easier to give out more databases than trying to communicate, it shows how far we have to go as an industry. Another point made was that it’s not typically possible to repurpose a DBA to maintain something like a Hadoop cluster.

In my notes, I wrote: “when problems are of a certain size, and affects the DNA of a company.” And when that happens, the typical separation of responsibilities between developer, DBA and operations break down.

Alex made my night by mentioning that sometimes solving a problem just takes getting a fresh set of eyes on it. He talked about replacing a particular bit of technology by applying some features of Postgres, with the help of a new hire.

I asked the final question – how does a company hire for the types of skills needed to solve these types of problems? Most agreed that finding someone with a passion for learning was critical. There was a dissenting voice – I think Bradford’s. My notes trailed off at that point, unfortunately. My guess is that he spoke up for logic and patience, and that some of the problems companies face have been largely solved, if you spend the time to study the science.

Epilogue

We wrapped up the night out at Little Big Burger and the Teardrop. Several people commented on how lovely Portland was – even in the rain. But damn, they wish they could hire the types of people they needed here.

So, people who are interested in big data: consider that a call to action.

Visit us for OSCON Data this summer. Then, move to Portland.

Three marketing areas where PDX11 fails

I had a short conversation yesterday about what I wish PDX11 was doing better with it’s marketing. I was the person who suggested we call this effort PDX11, and I bought the domains and asked the Open Source Lab to host our sites. I created the mailing lists and setup and ran the hackathon.

So, really, if anyone is to blame for this, it’s me. :)

But I’d like some help, so I’ll lay out to you what I think is wrong:

  1. Ensure that the Portland open source community knows what PDX11 is.
    Work in progress by the Knowledge Network is starting to address this issue. But much of the open source community has no idea what PDX11 is, or why they should care about it.
  2. Align some of the goals of open source community and the proprietary software community.
    Right now, it seems as though the values of one community diverge greatly from the other.

    The Mentor network seems like the most likely place for the groups to come together, but there’s still quite a significant cultural gap between the Software Association of Oregon and a very large open source community. It makes me wonder if the SAO hasn’t realized that open source developers are a meaningfully large and growing group that they should be serving the needs of.

    This reminds me of a post I did back in 2009 about growth in PostgreSQL job postings.

    So I looked up the relative growth rate of jobs with “open source” in the description:

    Seems like growth worth paying attention to, especially given that “Mobile App” has explosive growth, but still only represents .0007% of all job postings – whereas “open source” is in .004%:

  3. Marketing to the general public about why it’s important that the PDC focus effort on the software industry cluster.
    A friend asked the other day “Why should a non-geek care about PDX11?” And, I didn’t have a snappy answer. One thing that was said today during the unconference that I’m still mulling over was: “Software is the last growth industry we have in the US.” That’s maybe too agressive. :)

    Talking with Audrey today, I learned that it’s an established practice to target growing industries in a municipal area for public investment, and software industry was identified through an objective measure to be clustering and growing at a rate that warranted the city’s encouragement. The PDC has a site that shows some stats, but it’s far from clear to me, even as a person working on this project, exactly what is relevant to the general public. A bunch of this data was put into a PDF. One key number is that there are over 1,400 software companies in Oregon, and employment in this industry grew 19.2% from 2004-2009.

So, we could use your help. How should we address this?

Why should a non-geek care about the City’s and software community’s efforts to bring more software industry to Portland?

Weekly tweet digest for 2011-04-03

Weekly tweet digest for 2011-04-03

twittering on 2011-04-03

twittering on 2011-04-03