Category Archives: postgres

Postgres Open: next year (!), resources, video

Posted on by

Postgres Open is over!

I wanted to share a few resources, and remind attendees to fill out our survey. I really appreciate the detailed comments I’ve been getting! Keep them coming.

I wanted to specially thank our program committee:

Robert Haas
Josh Berkus
Gavin Roy
Greg Smith

They were the people who put together and edited the website, found sponsors, recruited speakers, voted on talks, gave talks and tutorials and executed the many tasks needed to make the conference a success. We plan to make key members of the Postgres community part of the operation of the conference going forward. We’re really just emulating the way that PgCon is run.

I have some more thoughts about what makes a conference “community-operated”, and once my budget numbers are settled, I’m going to share with you what running the conference costs in terms of my time, and in terms of dollars to operate. It’s important to both understand the costs involved, how much of my time is required and what that means for you as either a sponsor, speaker, attendee or volunteer supporting what we are doing.

NEXT YEAR: September 17-19, 2012

I’m pleased to announce that next year’s conference for September 17-19, 2012 at the Westin-Michigan Ave. So mark your calendars now!

The conference will continue to be operated as a non-profit, with proceeds going toward operation of the following year’s event, and a very small percentage going to Technocation, Inc – our fiscal sponsor and a 501(c)3 organization dedicated to developing educational opportunities and resources for software professionals.

We had fantastic support from our sponsors this year, and hope to expand that next year.

In particular: 2ndQuadrant, EnterpriseDB, Heroku and VMWare’s support were instrumental in pulling this event together. We really only started planning in May. It feels good to now have a whole year ahead of us!

With greater sponsor support, we can help fund some of the things that attendees asked for like: soda (which costs $8/soda – I feel as though we should get some kind of gold plating for this), conference tshirts, and a closing party.

Please get in touch if you or a company you know is interested in sponsorship for 2012!

Slides:

Speakers are uploading or linking their slides to the PostgreSQL wiki. If the slides you’re looking for aren’t there, please ping the speaker or me.

Streaming Video:

Streaming content will be available for about 30 days.

I will be getting all the video on flash drives this week. My plan is to upload it to either vimeo or youtube. I don’t really have the resources to provide individual copies of the videos, but if we find a location for raw data upload, I’ll pass that along to you all.

Looking toward Chicago: Postgres Open, local user groups, parties and on to October!

Posted on by

I’ve been incredibly busy this past month, and not blogging – being a free agent has possibly made me busier than I was before!

Postgres Open’s schedule is in near-final state. We’ve started adding talks to our Demo room on Thursday, and are looking forward to a keynote from Charles Fan, SVP at VMWare about recent developments in vmware’s cloud offerings for Postgres.

We’ll also be getting a more in-depth look at Heroku’s new postgres.heroku.com on-demand database service, as well as an open source tool they wrote called WAL-E.

Thanks to Heroku, we’ll be streaming much of the content from the conference live, so you’ll be able to catch the keynotes and many of the talks, even if you’re not there. And we’ll be sharing the videos after.

I believe we’re the first Postgres conference to do this! Someone correct me if I’m wrong. :)

While I’m in Chicago, I’m planning to drop by the Windy City Perl Mongers for a reprise of my 9.1 talk from OSCON.

We’re also planning a couple parties for Postgres Open, and hopefully inviting a few of the local user groups to join us.

After that, I’m headed in October to PostgreSQL Conference EU, and will be giving a talk about terabyte Postgres databases (and the problems you run into with them), and a database-specific “Mistakes were Made” talk, about operations and the tools we need to use to help us make fewer mistakes.

High availability and Postgres

Posted on by

A friend contacted me today, asking me “What are the best practices for failover with Postgres?” And he mentioned pgpool-II.

He was interested in 9.0, since 9.1 hasn’t been released yet. (but, it’s looking like we’re gearing up for a September release!)

My off-the-cuff response was:

There isn’t a single solution, although pgpool-II is a common one.

pgpool-II is what I’ve used in AWS. I’ve also seen people use heartbeat (I guess pacemaker now?). I think either works fine. The frustrating bit is that we don’t have the ability to refresh the failed system easily.

There’s also repmgr: https://github.com/greg2ndQuadrant/repmgr

It’s new, but might be worth exploring.

I started an High Availability page on the PostgreSQL wiki. We really need a canonical source of information for this. Devs are struggling to figure it out from our docs.

What are you doing for HA and Postgres?

The importance of doing things badly

Posted on by

Update: added “code review” to the things that we’re doing well below.

There were a couple themes for me from OSCON last week. One is transitions and change. I’ve got a whole slew of thoughts on this, particularly from my experience leaving the management team of Open Source Bridge.

But the other is the importance of doing things badly. In particular, the importance of doing things badly in open source.

Tim Anglade, at about 41:10, says that he thinks the reason why open source companies make money is because open source is kind of shitty (from an interview he did with Cliff Moon last fall). So, on one hand there’s a Money Making Opportunity. Probably not the one that we’d all prefer, but it is what it is.

When he said that, I immediately thought about the other things that we do badly (other than documentation) and the discussions I’d been having with people last week.

Basically, we had a problem in the Postgres community of experienced developers solving every small bug at nearly the moment it was reported. It’s sort of like a cat sitting at the entrance of the only mousehole.

The effect on the code is amazing – we have clearly documented, concise and consistent code. But the effect on the community is that we don’t have mid-level developers, and it is very difficult for inexperienced developers to build up a portfolio of small projects, based on bugs.

I don’t have a ready solution for this problem. And I do not mean this as a criticism of the thousands of hours our core teams have devoted to fixing bugs. We all benefit from the dedication. I am just pointing out that our system had a clear tradeoff – fewer contributors.

What we could do a bit worse (to address the point of this blog post) is lengthen our response time to solving bugs, and let some less experienced developers respond to the bugs queue. This probably involves creating a bug tracker and holding the tension a bit longer on fixes.

Our committers have made efforts toward spreading the load around more – with commitfest – meaning a greater support of code review, with Tom’s recent presentations about the planner, with our wiki-fied Todo list. And there are many more examples of our committers putting real effort into mentoring, tutoring and finding ways of bringing more people in.

The thing that’s missing from all of those efforts, however, is urgency. That’s what bug-fixing is great for. That’s why we have people who remain in operations work even if they hate being woken up at 3am. Urgent work is worthwhile work (mostly).

I’m sure there are other particular areas where we could do things worse, and thus invite more people to contribute. I’ll be thinking about this more in regard to our project event planning, as I think there’s a bit of a disconnect there, and a huge opportunity to involve more people.

I’m reminded again of David Eaves’ talks about how community management is the core competency of open source, not technology. I struggle with that thought every day, but it rings truer the more I try to work on the significant problems facing any particular open source project.

OSCON: We’re at the end…

Posted on by

I’m finally getting to blog, and here are a few highlights:

* “Mistakes were made” was a great time. Thank you everyone who shared stories. And those of you who attended, please connect with me – email or whatever, and let’s continue our discussions about failure.
* I have a little bit of editing to do left on the harder, better, faster, stronger slides. Talk ratings have been very high (thank you audience! :) Should have that up tomorrow!
* Not having a booth at OSCON was a real bummer for Postgres. We need to figure out a way to make this happen for us every year.
* Great having the time to connect with old friends in the hallways this week.
* Thanks O’Reilly for supporting our open source community.
* Thanks Google Open Source Programs office for bringing together open source leaders yet again this year for some important conversations.

Thank you everyone from the Postgres community who contributed to the Postgres day just before OSCON. All the speakers and their talks are listed here.

We need to keep having adjunct events like this! I think LCA has it right scheduling Mini-BoFs to provide networking opportunities for the distinct groups. I think OSCON should formalize this next year, and figure out a way of facilitating those groups in a more structured way.

I have another blog post brewing about difficult conversations.. but that’s going to have to wait until after I enjoy the brewers fest!

OSCON: Postgres represent! And my links for Harder, Better, Faster, Stronger talk

Posted on by

I’m giving a couple talks at OSCON this year. The first is on Tuesday, 10:40am room C123: Harder, Better, Faster, Stronger: Postgres 9.1. The other is Mistakes were Made, Wednesday at 1:40pm in room D136.

My colleague Robert Treat is giving a Pro PostgreSQL workshop Wednesday at 1:40pm too, room 204. He’s also giving a Scalability Patterns talk at 4:20pm Tuesday. I’m sure his talks will be awesome. :)

And here are the rest of the talks tagged with PostgreSQL.

Also remember — there’s a PgDay tomorrow at the Oregon Convention Center!

I’m pushing my examples for my 9.1 talk into a github repo. It should be populated with whatever I decide to use for the talk by Monday evening.

Building 9.1 for me on Mac OS X (leopard!) involved the following:


git tag -l | grep REL9_1
git checkout REL9_1_BETA2
./configure --with-perl --with-python --prefix=/opt/pg91beta2 --with-readline
make
make install

Normal caveats apply – you need X Code of a reasonably recent version, and a bunch of support libraries to make this happen. I haven’t rebuilt from scratch on OS X in a long time, but now I realize that maybe I aught to go through the pain and document this again.

But I digress!

I have a long list of resources for this talk and wanted to share. Probably in the slides for the talk, I’ll provide shortlinks so that people can pull them up and read instead of listening to me :D

Here’s my links:

And if you’re wondering about the title, I took it from an great Daft Punk song that fans have created some epic videos of:

PgCon Pub Track: Learning more about Synchronous Replication

Posted on by

So, we’re at the Pub and doing “create a billion tables” time trials with Jan Urbanski using Python and Josh Berkus using Perl.

We’re also hacking on a test framework the Slony developers have, specifically hacking with Steve Singer. What we discovered is that sync rep doesn’t wait for a WAL segment to be *replayed* before it returns. In the pg_stat_replication table, we see sent_location, write_location and flush_location synchronized, but not replay_location.

This makes sense from a database perspective, but may be surprising behavior for application developers. There are patches out there (according to what I just heard from Bernd) to make synchronous replication wait for replay on the slave, but it’s not certain when that will be committed. It definitely won’t be part of version 9.1.

I just wrote up configuration details from a database administrator’s perspective, and am planning on doing some additional work to make a highly condensed configuration tutorial for our main docs. We definitely need to explain this more clearly for users, who might be thinking of it more from an application perspective.

PgCon Day 1 – Cluster summit and catching up with folks

Posted on by

Yesterday, I spent my morning at the Clustering summit, catching up on what the cluster hackers have been up to for the last year. I was lucky enough to sit next to Jan Wieck and Kevin Grittner. You may remember Kevin from his work on serializable snapshot isolation.

There were some pretty awesome side conversations about where folks think work needs to be done next, and conflict resolution for multi- (or many-) master setups.

I gave a quick update on Bucardo 5, which had an alpha release last week, supports many-master and has has experimental support for non-Postgres targets. The first two targets are text and MongoDB.

The Postgres project has given the generic name “binary replication” to all the features like WAL shipping, streaming replication and synchronous replication. Simon Riggs also gave his update on these features at the Clustering Summit today. He observed that the 9.1 release is the culmination of 7 years of work on replication subsystems. Simon pointed out that synchronous replication is the best, and most obvious, use case for the binary replication at the core of Postgres. And also pointed out that he was quite pleased with the ultimate design.

For the afternoon, I spent some time with folks on the infrastructure team, giving Magnus well-deserved congratulations for his induction into -core, and meeting up with folks from all over at the Royal Oak and Keg, a reasonable steakhouse in town.

Looking forward to the developers meeting today!

At PgCon 2011 – day 0

Posted on by

I wrote my review of synchronous replication over on Emma’s Tech blog (It’ll probably be published mid-day Tuesday). I’m visiting Ottawa this year on behalf of Emma, one of many great sponsors of Postgres’ yearly international developer conference, pgCon.

This week will be packed for me – attending the Clustering summit, the developers meeting, presenting about Emma’s database systems, leading the lightning talks, and of course attending the many parties this week.

Because we are spread so far around the globe, pgCon is often our one chance to get together and really dig into problems in-person.

And, I’m pulling together our first ever Procedural Language summit. With the new extension system, over 30 procedural languages implemented, and a ton of new features being added to existing PLs, I thought it was time PL developers should come together and have a chat. I’ve still got a few details to work out before Saturday (sorry all that RSVP’d – final agenda coming soon!).

I’m hoping to also have another, unrelated, announcement this Wednesday. Hopefully all the details come together!

Anyway, with that cliffhanger, I’m off to get a good night’s rest before the clustering summit tomorrow.

9.1 beta 1 is out! Help us test.

Posted on by

Postgres released version 9.1 beta 1 today! This is a preview of 9.1, predicted to be available in the next 2-3 months, not a bugfix release for earlier versions of Postgres.

PostgreSQL 9.1 contains a huge volume of new features, possibly more any single release of PostgreSQL before. These features also include several innovations which PostgreSQL is the first database system to have. The most anticipated features in this version include:

  • Synchronous Replication
  • Per-column collations for multilingual databases
  • Unlogged Fast Tables
  • K-Nearest-Neighbor Indexing
  • Serializable Snapshot Isolation
  • Writeable Common Table Expressions
  • SE-Linux Integration
  • Extensions
  • SQL/MED attached tables

The PostgreSQL project now depends on you to test 9.1beta1 in order have a rapid and bug-free 9.1 release. If you are able to help with testing version 9.1, please see the Beta Testing HOWTO

Binary downloads are available, as is the source.

If you’d like to grab a copy of the latest from git, here is a quick set of instructions to compile 9.1beta1 from the git repo: 


git checkout REL9_1_BETA1
./configure --prefix=/opt/pg9.1beta1
make
sudo make install

And then to create a database: 

/opt/pg9.1beta1/bin/initdb -D mytestdb
/opt/pg9.1beta1/bin/pg_ctl -D mytestdb start

For a preview of features coming this fall, check out Depesz’s blog.