{"id":3420,"date":"2011-10-28T18:21:58","date_gmt":"2011-10-29T02:21:58","guid":{"rendered":"http:\/\/www.chesnok.com\/daily\/?p=3420"},"modified":"2012-03-26T02:21:51","modified_gmt":"2012-03-26T10:21:51","slug":"going-from-vagrant-and-puppet-into-ec2-a-short-survey-of-5-tools-and-two-i-didnt-bother-trying","status":"publish","type":"post","link":"https:\/\/www.chesnok.com\/daily\/2011\/10\/28\/going-from-vagrant-and-puppet-into-ec2-a-short-survey-of-5-tools-and-two-i-didnt-bother-trying\/","title":{"rendered":"Going from Vagrant and Puppet into EC2: A short survey of 5 tools (and two I didn&#8217;t bother trying)"},"content":{"rendered":"<p>I thought this would be easy. <\/p>\n<p>I started using <a href=\"http:\/\/vagrantup.com\/docs\/getting-started\/index.html\">Vagrant<\/a>, and was productive with it in about a day. Really a couple hours. Most of my time was spent downloading the correct version of VirtualBox, <a href=\"http:\/\/morethanseven.net\/2011\/05\/08\/Vagrant-plugin-for-interacting-with-vagrantboxes.html\">looking for starter images<\/a> and then a small amount of time experimenting with the Vagrantfile scripting language (<a href=\"http:\/\/vagrantup.com\/docs\/multivm.html\">for multiple VMs<\/a>).<\/p>\n<p>And we made some Puppet configs.<br \/>\n<!--more--><br \/>\nThen I wanted to use those same Puppet configs with EC2.<\/p>\n<p>So my ultimate goals were: <\/p>\n<ol>\n<li>Reuse my existing puppet configs as much as possible<\/li>\n<li>Have a completely automated deploy of a server system (including checkouts of code from a private github repo)<\/li>\n<li>Have a puppetmaster in EC2<\/li>\n<li>Be able to provision systems from EC2 or my laptop<\/li>\n<li>Make the whole process easy for my coworkers<\/li>\n<\/ol>\n<p>This is mostly a list of what I failed at using, and the thing I succeed with at the end.<\/p>\n<p><em>Short aside:<\/em><\/p>\n<p><b>Pro tip to people writing documentation<\/b>: Most tutorials and sites that make recommendations for tools leave out the part where you run into all kinds of insane problems. <b>Create a wiki page or a place where you collect the problems.<\/b> Please. <\/p>\n<p>For example: My Cloud Formation to Ubuntu AMI deploy was failing with an error in cfn.rb that said: &#8220;Unexpected return.&#8221;  Um. Ok. *facepalm*<\/p>\n<p>The problem was that a AWS-image specific JSON file wasn&#8217;t present (and couldn&#8217;t be created) on the target machine. So instead of noting (raise an exception, anyone?) that the file wasn&#8217;t present, the module just executed a bare <code>return<\/code>. <\/p>\n<p>Because I don&#8217;t know much about Puppet internals, this was a very annoying problem to solve. (like, what gets installed in <code>\/var\/lib\/puppet\/lib<\/code> vs. in the <code>gem install<\/code> vs. the cloudpack library I was told to install in <code>\/etc\/puppet\/modules<\/code>?)<\/p>\n<p>Stepping back a bit &#8211; a useful note from the Cloud Formation folks would have been: &#8220;Hey &#8211; this probably won&#8217;t work if you try to deploy to non-Amazon Linux AMI distros of Linux.&#8221; It&#8217;s not obvious that&#8217;s the case! You&#8217;re supposed to be able to completely control the classes being installed on the target system, right? Bad assumption, apparently.<\/p>\n<p><em>And we&#8217;re back!<\/em><\/p>\n<p>Let me know in the comments if you&#8217;ve successfully navigated any of the tools I didn&#8217;t pick. Juju, in particular, I don&#8217;t think I gave a fair chance (since I didn&#8217;t try it at all). <\/p>\n<p>Here&#8217;s my list: <\/p>\n<ol>\n<li><a href=\"http:\/\/www.slideshare.net\/derleiermann\/juju-puppetconf\">Juju<\/a>\n<p>I just wasn&#8217;t sure this was a reasonable thing to install\/use. No one I knew had ever heard of it. Didn&#8217;t try it.\n<\/li>\n<li><a href=\"http:\/\/www.devco.net\/archives\/2010\/07\/14\/bootstrapping_puppet_on_ec2_with_mcollective.php\">Mcollective + tools ported to PHP<\/a>\n<p>I&#8217;m interested in Mcollective, but the configs looked overly complex, and I didn&#8217;t have anyone close by that was actively using it. <\/p>\n<p>The examples scared me away because of the PHP. I already had three languages at play in the deployment, and I didn&#8217;t need another language dependency. So, I didn&#8217;t bother trying it.\n<\/li>\n<li><a href=\"http:\/\/www.codelord.net\/2010\/12\/19\/using-puppet-to-automatically-configure-new-ec2-instances\/\">Custom scripts based on the ec2-tools packages<\/a>\n<p>This approach works, but is fragile and a PITA to keep updated. I tried it as a &#8220;getting oriented&#8221; exercise, and abandoned it.\n<\/li>\n<li><a href=\"https:\/\/github.com\/jedi4ever\/mccloud\">Mccloud<\/a>\n<p>This looked awesome! I could reuse all my Vagrant configs and not really have to change anything&#8230; Except I had to maintain duplicate configs, just sub &#8216;Mccloud&#8217;. Eh. <\/p>\n<p>I may revisit this tool in the future, but it seemed to require pretty much the same things as the tool I ultimately decided to use, and didn&#8217;t seem as flexible. I also had a weird restriction where it wouldn&#8217;t allow me to spin up the correct type of image (I wanted m1.small in my testing). Could have been PEBKAC &#8212; I didn&#8217;t take good enough notes to say for sure.\n<\/li>\n<li><a href=\"https:\/\/help.ubuntu.com\/community\/CloudInit\">cloud-init<\/a>\n<p>This looked very promising! We were already using Ubuntu so seemed like a good fit. <\/p>\n<p>Pros: easy &#8211; pass in a shell script when starting an EC2 instance from the web. Cons: required yet-another-configuration style. But there were command-line tools and it was looking very promising.<\/p>\n<p>In the end, using a supported package would have required me to be running a Linux desktop to start my puppetmaster. I didn&#8217;t search much harder than <code>brew install cloud-init<\/code> for a Mac-equivalent (that doesn&#8217;t exist). So, I moved on to the next thing.\n<\/li>\n<li><a href=\"http:\/\/aws.amazon.com\/cloudformation\/\">AWS Cloud Formation<\/a>\n<p>I launched a puppetmaster pre-configured instance! I sort of got puppetmaster running! Then I tried to deploy an Ubuntu AMI from it&#8230; This does not work. <\/p>\n<p>So, I will save you a ton of time: <b>Avoid trying to mix the pre-specified Cloud Formation images with other systems.<\/b><\/p>\n<p>Someone showed me the chunk of the config you can rip out and probably get it to work. I was frustrated at that point, and moved on. Too much tweaking was required, for what was uncertain gain at that point.\n<\/li>\n<li><a href=\"https:\/\/github.com\/puppetlabs\/puppetlabs-cloud-provisioner\">PuppetLab&#8217;s Cloud Provisioner<\/a>\n<p>This is what I am currently using! I&#8217;m running <code>HEAD<\/code> pulled directly from github.  Older versions are not recommended. (I tried three versions.)<\/p>\n<p>The configuration is pretty straightforward and <a href=\"http:\/\/docs.puppetlabs.com\/guides\/cloud_pack_getting_started.html\">documented<\/a>. The one thing (a very important thing) is that you have to amend your <code>$RUBYLIB<\/code> if you don&#8217;t install the code in your version of ruby&#8217;s default libdir. There&#8217;s no gem. <a href=\"http:\/\/projects.puppetlabs.com\/issues\/10379\">Yet<\/a>. <\/p>\n<p>I customized the deploy script to my liking &#8211; there is an unsupported option called <code>--install-script<\/code> you can pass in that will execute whatever <code>.erb<\/code> (a shell script!) you&#8217;d like if you put it in <code>~\/.puppet\/scripts<\/code>. You can also pass in your puppetmaster hostname with <code>--server<\/code>.<\/p>\n<p>Totally sweet. <\/p>\n<p>The command-line is ok, but there&#8217;s also a programmatic interface in Ruby. Dan Bode showed me a short code snippet that worked (hostnames &#038; keys sanitized): <\/p>\n<p><code><br \/>\nirb(main):012:0> require 'puppet'<br \/>\nirb(main):013:0> require 'puppet\/face'<br \/>\nirb(main):014:0> Puppet::Face[:node, :current].install('myserver.compute-1.amazonaws.com', :keyfile => 'mykey.pem', :login => 'ubuntu', :install_script => 'custom-puppetmaster', :server=>'myserver.compute-1.amazonaws.com')<br \/>\n<\/code><\/p>\n<p>I so appreciate this! <a href=\"http:\/\/puppetlabs.com\/faces\/\">Faces<\/a> is awesome. <\/p>\n<\/li>\n<\/ol>\n<p>I&#8217;ve got some additional tweaking to do yet, but I&#8217;m planning to commit a few amendments to the provisioner scripts included by default and the README. And I filed a couple bugs. <\/p>\n<p>Overall, I&#8217;d bet that <a href=\"https:\/\/github.com\/puppetlabs\/puppetlabs-cloud-provisioner\">cloud-provisioner<\/a> (if you use the version currently on github) will work for most people. <\/p>\n","protected":false},"excerpt":{"rendered":"<p>I thought this would be easy. I started using Vagrant, and was productive with it in about a day. Really a couple hours. Most of my time was spent downloading the correct version of VirtualBox, looking for starter images and &hellip; <a href=\"https:\/\/www.chesnok.com\/daily\/2011\/10\/28\/going-from-vagrant-and-puppet-into-ec2-a-short-survey-of-5-tools-and-two-i-didnt-bother-trying\/\">Continue reading &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[507,505,661,503,506,216,232,504],"class_list":["post-3420","post","type-post","status-publish","format-standard","hentry","category-sysadmin","tag-bootstrap","tag-deployment","tag-devops","tag-ec2","tag-provisioning","tag-puppet","tag-tools","tag-vagrant"],"_links":{"self":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/3420","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/comments?post=3420"}],"version-history":[{"count":10,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/3420\/revisions"}],"predecessor-version":[{"id":3473,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/posts\/3420\/revisions\/3473"}],"wp:attachment":[{"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/media?parent=3420"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/categories?post=3420"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.chesnok.com\/daily\/wp-json\/wp\/v2\/tags?post=3420"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}