Puppet Faces are an extendable API for tricking out your Puppet instances. (“Faces” is just short for “Interfaces”.) Just a couple days ago I wrote about my survey of puppet + ec2 provisioning tools.

The problem I’m trying to solve, which I don’t feel like I’ve solved well, is how to give a type to a new system at bootstrap time, without using DNS. The type variable maps to a node manifest group, and determines the personality of a host – is it a database, webserver or development instance?

What I’d like to do is pass a type to puppet at install time and have the puppetmaster and the agent remember that mapping between host and type.

I did it with a really simple Facter plugin, install scripts named by type (passed in to puppet node install), and a file created by the install script in /etc/puppet.

Then, I wanted to be able to see which hosts were configured with which install type. Facter was aware of the type, so this seemed like it should be pretty easy…

I wrote a quick and dirty Face that pulls information out of $varlib/nodes/*.yaml on the puppet master. I imagine there are better ways to do this, but in the absence of documentation or someone to tell me not to do this, I forged ahead!

There were two things that I spent quite a bit of time chewing on before figuring it out:

1. If you want to make an :action in your Face the default, you just add default in the body of your :action block. I had to dig through a few cloudpack files before I found it!
2. If you are creating and terminating hosts frequently, you may end up with a bunch of certs and other annoying metadata laying around. To clean it up, the Puppet Node Face has a command you can run:
   
# puppet node clean [hostname]


   You’ll probably need to be the user that’s running puppet for this to work — it affects things that the puppetmaster owns in $varlib.

   If you’re doing this with code, it’s:
   
Puppet::Face[:node, :current].clean('hostname')


I put a little patch into a recent version of cloudprovisioner that invokes clean during a terminate. It’s quick and dirty, and only for AWS.

The resources I’ve found useful are:

• Kelsey’s talk at PuppetConf about Faces
• Kelsey Hightower’s Git repo for Puppet Dashboard Face
• Kartar’s Github face

And to a lesser extent, these blog posts were helpful for filling in a few gaps:

• What the heck are Faces?
• Creating a simple “backup” Face

I thought this would be easy.

I started using Vagrant, and was productive with it in about a day. Really a couple hours. Most of my time was spent downloading the correct version of VirtualBox, looking for starter images and then a small amount of time experimenting with the Vagrantfile scripting language (for multiple VMs).

And we made some Puppet configs.

Then I wanted to use those same Puppet configs with EC2.

So my ultimate goals were:

1. Reuse my existing puppet configs as much as possible
2. Have a completely automated deploy of a server system (including checkouts of code from a private github repo)
3. Have a puppetmaster in EC2
4. Be able to provision systems from EC2 or my laptop
5. Make the whole process easy for my coworkers

This is mostly a list of what I failed at using, and the thing I succeed with at the end.

Short aside:

Pro tip to people writing documentation: Most tutorials and sites that make recommendations for tools leave out the part where you run into all kinds of insane problems. Create a wiki page or a place where you collect the problems. Please.

For example: My Cloud Formation to Ubuntu AMI deploy was failing with an error in cfn.rb that said: “Unexpected return.” Um. Ok. *facepalm*

The problem was that a AWS-image specific JSON file wasn’t present (and couldn’t be created) on the target machine. So instead of noting (raise an exception, anyone?) that the file wasn’t present, the module just executed a bare return.

Because I don’t know much about Puppet internals, this was a very annoying problem to solve. (like, what gets installed in /var/lib/puppet/lib vs. in the gem install vs. the cloudpack library I was told to install in /etc/puppet/modules?)

Stepping back a bit – a useful note from the Cloud Formation folks would have been: “Hey – this probably won’t work if you try to deploy to non-Amazon Linux AMI distros of Linux.” It’s not obvious that’s the case! You’re supposed to be able to completely control the classes being installed on the target system, right? Bad assumption, apparently.

And we’re back!

Let me know in the comments if you’ve successfully navigated any of the tools I didn’t pick. Juju, in particular, I don’t think I gave a fair chance (since I didn’t try it at all).

Here’s my list:

1. Juju

   I just wasn’t sure this was a reasonable thing to install/use. No one I knew had ever heard of it. Didn’t try it.

2. Mcollective + tools ported to PHP

   I’m interested in Mcollective, but the configs looked overly complex, and I didn’t have anyone close by that was actively using it.

   The examples scared me away because of the PHP. I already had three languages at play in the deployment, and I didn’t need another language dependency. So, I didn’t bother trying it.

3. Custom scripts based on the ec2-tools packages

   This approach works, but is fragile and a PITA to keep updated. I tried it as a “getting oriented” exercise, and abandoned it.

4. Mccloud

   This looked awesome! I could reuse all my Vagrant configs and not really have to change anything… Except I had to maintain duplicate configs, just sub ‘Mccloud’. Eh.

   I may revisit this tool in the future, but it seemed to require pretty much the same things as the tool I ultimately decided to use, and didn’t seem as flexible. I also had a weird restriction where it wouldn’t allow me to spin up the correct type of image (I wanted m1.small in my testing). Could have been PEBKAC — I didn’t take good enough notes to say for sure.

5. cloud-init

   This looked very promising! We were already using Ubuntu so seemed like a good fit.

   Pros: easy – pass in a shell script when starting an EC2 instance from the web. Cons: required yet-another-configuration style. But there were command-line tools and it was looking very promising.

   In the end, using a supported package would have required me to be running a Linux desktop to start my puppetmaster. I didn’t search much harder than brew install cloud-init for a Mac-equivalent (that doesn’t exist). So, I moved on to the next thing.

6. AWS Cloud Formation

   I launched a puppetmaster pre-configured instance! I sort of got puppetmaster running! Then I tried to deploy an Ubuntu AMI from it… This does not work.

   So, I will save you a ton of time: Avoid trying to mix the pre-specified Cloud Formation images with other systems.

   Someone showed me the chunk of the config you can rip out and probably get it to work. I was frustrated at that point, and moved on. Too much tweaking was required, for what was uncertain gain at that point.

7. PuppetLab’s Cloud Provisioner

   This is what I am currently using! I’m running HEAD pulled directly from github. Older versions are not recommended. (I tried three versions.)

   The configuration is pretty straightforward and documented. The one thing (a very important thing) is that you have to amend your $RUBYLIB if you don’t install the code in your version of ruby’s default libdir. There’s no gem. Yet.

   I customized the deploy script to my liking – there is an unsupported option called --install-script you can pass in that will execute whatever .erb (a shell script!) you’d like if you put it in ~/.puppet/scripts. You can also pass in your puppetmaster hostname with --server.

   Totally sweet.

   The command-line is ok, but there’s also a programmatic interface in Ruby. Dan Bode showed me a short code snippet that worked (hostnames & keys sanitized):

   
irb(main):012:0> require 'puppet'
irb(main):013:0> require 'puppet/face'
irb(main):014:0> Puppet::Face[:node, :current].install('myserver.compute-1.amazonaws.com', :keyfile => 'mykey.pem', :login => 'ubuntu', :install_script => 'custom-puppetmaster', :server=>'myserver.compute-1.amazonaws.com')


   I so appreciate this! Faces is awesome.

I’ve got some additional tweaking to do yet, but I’m planning to commit a few amendments to the provisioner scripts included by default and the README. And I filed a couple bugs.

Overall, I’d bet that cloud-provisioner (if you use the version currently on github) will work for most people.

A few days ago, I asked:

I spend a lot of time thinking about the little details in systems – like the number of ephemeral ports consumed, number of open file descriptors and per-process memory utilization over time. Small changes across 50 machines can add up to a large overall change in performance.

And then, today, I saw this article:

One of the more telling comments I received was the idea that since the advent of virtualization, there’s no point in trying to fix anything anymore. If a weird error pops up, just redeploy the original template and toss the old VM on the scrap heap. Similar ideas revolved around re-imaging laptops and desktops rather than fixing the problem. OK. Full stop. A laptop or desktop is most certainly not a server, and servers should not be treated that way. But even that’s not the full reality of the situation.

I’m starting to think that current server virtualization technologies are contributing to the decline of real server administration skills.

There definitely has been a shift – “real server administration skills” are now more about packaging, software selection and managing dramatic shifts in utilization. It’s less important know to know exactly how to manage M4 with sendmail, and more important that you know you should probably use postfix instead. I don’t spend much time convincing clients that they need connection pooling; I debug the connection pooler that was chosen.

The available software for web development and operations is quite broad – the version of Linux you select, whether you are vendor supported or not, and the volume of open source tools to support applications.

Inevitably, the industry has shifted to configuration management, rather than configuration. And, honestly, the shift started about 15 years ago with cfengine.

Now we call this DevOps, the idea that systems management should be programmable. Burgess called this “Computer Immunology”. DevOps is a much better marketing term, but I think the core ideas remain the same: Make programmatic interfaces to manage systems and automate.

But, back to the broken window thing! I did some searching for development and broken windows and found that in 2007, a developer talked about Broken Window Theory:

People are reluctant to break something that works, but not so much when it doesn’t. If the build is already broken, then people won’t spend much time making sure their change doesn’t break it (well, break it further). But if the build is pristine green, then they will be very careful about it.

In 2005, Jeff Atwood mentioned the original source, and said “Maybe we should be sweating the small stuff.”

That stuck with me because I admit that I focus on the little details first. I try to fix and automate where I can, but for political or practical reasons, I often am unable to make the comprehensive system changes I’d like to see.

So, given that most of us live in the real world where some things are just left undone, where do we draw the line? What do we consider a bit of acceptable street litter, and what do we consider a broken window? When is it ok to just reboot the system, and when do you really need to figure out exactly what went wrong?

This decision making process is often the difference between a productive work day, and one filled with frustration.

The strategies that we use to make this choice are probably the most important aspects of system administration and devops today. There, of course, is never a single right answer for every business. But I’m sure there are some themes.

For example:

• My boss wrote about PEP8 and standards in formatting code, and automating this with a githook.
• PostgreSQL code goes through a reformatting pass before we release.
• Etsy tracks their PHP warnings for every release

James posted “Rules for Infrastructure” just the other day, which is a repost of the original gist. What I like about this is that they are phrased philosophically: here are the lines in the sand, and the definitions that we’re all going to agree to.

Where do you draw the line? And how do you communicate to your colleagues where the line is?

To pick up where Devrim left off in customizing RPMs, here are some more tips for getting your very own RPMs built:

• Create a VM with your favorite operating system (I’m using versions of CentOS). I need both 32-bit OS and 64-bit OS. This is much easier to manage with separate, local VMs.
• Install spectool (available here), and SVN
• The other dependancies were: gcc glibc-devel bison flex python-devel tcl-devel readline-devel zlib-devel openssl-devel krb5-devel e2fsprocs-devel libxml2-devel libxslt-devel pam-devel
• Edit the postgresql-$VERSION.spec file to your liking: If you’re adding patches, you need to add them in TWO places – first in the Patch#: group, and then again below where the %patch# series starts. Finally, if you’re adding an entirely new package (say in 8.2, pg_standby in contrib), you’ll need to also add the binary (or library, or whatever) to the appropriate %files clause later in the spec file. It’s also a good idea to modify ‘Release’. Here’s a sample diff of my spec file:


--- postgresql-8.2.spec (revision 188)
+++ postgresql-8.2.spec (working copy)
@@ -74,7 +74,7 @@
Summary: PostgreSQL client programs and libraries
Name: postgresql
Version: 8.2.17
-Release: 1PGDG%{?dist}
+Release: 1test%{?dist}
License: BSD
Group: Applications/Databases
Url: http://www.postgresql.org/
@@ -95,7 +95,9 @@
Patch4: postgresql-test.patch
Patch6: postgresql-perl-rpath.patch
Patch8: postgresql-prefer-ncurses.patch
+Patch7: postgresql-pgstat-dir.patch
Patch9: postgresql-use-zoneinfo.patch
+Patch10: pg_standby.patch

pushd doc
tar -zcf postgres.tar.gz *.html stylesheet.css
@@ -604,6 +608,7 @@
%{_bindir}/pg_controldata
%{_bindir}/pg_ctl
%{_bindir}/pg_resetxlog
+%{_bindir}/pg_standby
%{_bindir}/postgres
%{_bindir}/postmaster
%{_mandir}/man1/initdb.*


How have you customized RPMs using this repo? Share your .spec files!

Last night’s meeting was about ptop and Mark Wong’s efforts to make an interactive, command-line tool for monitoring the current status of a PostgreSQL database.

For our meeting, Mark set up a test operating system on a USB drive, and bravely demo’d his new software.

Mark got the idea for ptop a few months ago, and went looking for the source code to top to get started. After a few days of hacking, he had a some useful features he wanted to share. So, he’s set up a project and started gathering developers:

http://pgfoundry.org/projects/ptop

The features currently supported include displaying:

• Current queries
• Query plans
• Locks
• User table statistics
• User index statistics

One feature I particularly liked was showing deltas of statistics over time. So you set your ptop sampling interval to some value (i.e. 5 seconds) and then you can see, for example, user table statics
changing over time. This is something I wrote some perl scripts for, but would love to have as a standard utility. Particularly if I could log/graph it.

Currently, ptop is only compiling on Linux. I really want it on Mac OS X.

We came up with a short list of desired features for the next revision of ptop:

• Change command line options to match psql (-U for user, -p for port, etc)
• Add command to show table (basically \dt [tablename]) for non-query processes
• Enable arrow keys to move around in the content to select, instead
  of cut/paste process ID
• Log the stats deltas over time (i would be very interested in this)
• Change lock query to use an OUTER JOIN to show “waiting” locks that
  aren’t granted

And just a general question that came up:

• Is it possible to peek at a currently executing SELECT statement’s plan?
  – We all agreed that this would be awesome.

Thanks for a great meeting.

Thank you Mark & Gabrielle for sending out meeting announcements this month.

NEXT MEETING: DIFFERENT DATE AND TIME!

December Coders Bash
Tuesday, December 11, 2007
CubeSpace, 622 SE Grand Ave., Portland OR 97214

Sam from the PHP Users group started organizing a group event in
December, inviting folks from all the different users groups to come
and socialize.

ALSO a HACKATHON:
Gabrielle, Mark and I thought it would be awesome to have a ptop HackAThon the weekend after Thanksgiving. Some possible features we’d work on are below. Anyone interested? Get in touch. I was thinking the early afternoon on Saturday, or early evening on Sunday.

Gabrielle and I met to talk about some projects today. She brought up a couple questions that were raised about differences between MySQL and PostgreSQL syntax for data export.

She showed me \pset fieldsep and \pset format for controlling interactive output from SELECTS (see psql documentation). You might say: \pset fieldsep , (although that wouldn’t be CSV.. but it’s quick and dirty). And \pset format offers unaligned, aligned, html, latex, or troff-ms. There are several shortcuts available – \a for aligned/unaligned. A combination of \pset fieldsep and \a gets you nearly to CSV.

Then we took a look at the COPY command and our options there. That’s when we discovered this:


COPY { tablename [ ( column [, ...] ) ] | ( query ) }
TO { 'filename' | STDOUT }


See that query? Yeah, super sweet. This feature was new in version 8.2. (8.3 beta is out now!)
Now you can run a command like:


COPY (SELECT param1, param2, param3 from myview) TO STDOUT WITH CSV;


Or you can replace STDOUT with a file path. \copy supports the same syntax. This is a reasonable alternative to MySQL’s SELECT INTO OUTFILE. And the feature has been there for at least a year.

Wouldn’t it be great if the non-profit world could embrace free software? In my head, I’ve seen a giant Venn diagram labeled “VALUES” with Open Source/Free Software overlapping significantly with the of non-profits. Here’s a small one:

The problem is implementation and systems support. Administration is where the car goes off the rails for non-profits. Qualified open source admins are not necessarily available to non-profits – I’m not sure exactly why, but I’d bet cost is a big reason.

Non-profits often receive equipment and software donations from the community, with little technical experience to maintain them. Most of the donations are commercial software with expensive licenses. There are a few tech support groups popping up that cater to non-profits (lower prices, focus on maintaining – not upgrading).

We have FreeGeek here in Portland. But there are still many non-profits who don’t or can’t use their services. I wish that there was a “server-in-a-box” setup that office managers would feel comfortable maintaining. Filesharing is so ubiquitous and necessary, it is unreasonable to expect that every office that needs filesharing will have a “qualified” systems administrator to maintain the server.

I guess my question is – is there a set of software apps that could be given to small- to mid- size non-profits as a replacement for commercial/non-free software?

Off the top of my head, I would want:

* Ubuntu Linux for client/server
* Firefox for a web browser
* Zimbra or Chandler/Cosmo for email and calendaring
* WebDav and SMB filesharing
* Subversion auto-versioning support on sharepoints
* Something like once:radix for a Filemaker-like database interface
* Accounting software?
That would all be pretty tough for a non-profit to maintain. I wonder about packaging those things together. Would it be worth it?

And then, once it was put together, how do we create a system where non-profits either have access to qualified sysadmins or can administrate everything themselves?

I’ve done a little research into non-profit/tech forums and organizations, but not nearly enough to know everything about what’s already out there. I’m very interested in pursuing this idea, maybe just for the sake of the few non-profits I work with.