Running Puppet on CloudStack Instances

When we were building our CloudStack environment we wanted newly created instances to check into the puppetmaster and receive their configurations automatically. The goals for accomplishing this were:

• No human intervention
• Do not have to update a separate asset database/spreadsheet/text file/etc.
• Do NOT use Puppet’s auto-signing feature
• Instances receive all config via Puppet, thus minimizing the number of CloudStack templates we have to maintain by only having to keep base/minimal images for each OS that we are supporting (One el5 image, one el6 image, etc.)

We decided to bake a pre-signed Puppet certificate into the templates. We ran into a couple of issues though. To generate our pre-signed certificate, I took a client and modified it’s puppet.conf with the following:

[agent]
    certname = cert001
    node_name = facter
    node_name_fact = fqdn

If you are going to use my method of loading facts from user-data (described below) you will also need to enable pluginsync in puppet.conf.

The node_name and node_name_fact settings are necessary because the Puppet client thought it’s node name was “cert001” when using the pre-signed cert. After adjusting these settings, I had the client check into the master, manually signed the cert, then had the client check in again and pull down the certificate. James Turnbull (@kartar) pointed out that you can also generate the cert by running the following on the puppetmaster:

puppet cert generate <hostname>

So, for our example, it would be:

puppet cert generate cert001

Now that I have the signed cert, I copied it out of Puppet’s /var/lib/puppet/ssl directory on the client and put it on the other machines that I was using to build the other templates (also copying over the changes to puppet.conf to those machines). In the event the certificate gets compromised or we decide to rotate the cert on any sort of schedule, we would increment cert001 to cert002 and rebuild our templates. For existing instances, you would have to copy the signed cert to them and update their puppet.conf. Since our environment/instances are transient in nature, we would just rebuild the templates, deploy new instances and destroy the old instances.

I then had to modify /etc/puppet/auth.conf on the Puppet master. Details about why and what to change can be found in bug 2128.

# allow nodes to retrieve their own catalog
# (ie their configuration)
#path ~ ^/catalog/([^/]+)$
#method find
#allow $1

# This change allows us to use a common
# certificate across multiple nodes.
path ~ /catalog/.+
allow *

Make sure you have splay disabled in puppet.conf, otherwise your VM will boot up and wait for some random amount of time before checking into the puppetmaster…this pushes out how long it takes to provision the instance from end-to-end.

Don’t forget to chkconfig puppet on!

Once you’ve done all of that, turn the VM into a template inside CloudStack. I’m not going to cover that here.

Let’s take a look at what we have now. We have a template that when deployed will automatically have puppet start up and check into the puppetmaster. That’s half the battle.

Now, we need to somehow tell puppet what configs to deploy to this machine. I’m not a huge fan of using an ENC (external node classifier), but I suppose you could build out and ENC to intelligently decide what to do. Instead of doing that, I opted to create a fact called role that I could use in my manifests.

I use the user-data associated with a VM to store key value pairs of data that then get loaded into puppet facts. I have a separate post about this with links to the source code that I use.

So now we have a template we can deploy that automatically checks into puppet when it boots up and passes a $role fact to puppet. How does this help?

For my current site, all nodes are default nodes. This is a stripped down version of my site.pp (omitting defaults and other stuff irrelevant to this example):

import "base"

node default {
    include base
}

I’ve introduced a new manifest called base.pp. This class is applied to all nodes and is where the magic happens. My base.pp looks something like this:

class base {
    ...
    # Includes that apply to all machines
    ...

    # role-specific includes
    case $role {
        'somerole': {
            include somerole
        }
        'otherrole': {
            include otherrole
        }
    }
}

At the top of my base class, I include configurations that apply to all machines. Stuff like ntp, sshd, yum, etc. The case statement then includes any additional modules on a per-role basis.

To recap, we now have a CloudStack template that we can deploy that will automatically check into the puppetmaster and receive the configuration for whatever $role we specify in the user-data.

One of the big downsides to this approach is that the only way(as of CloudStack v2.2.13) to set user-data for an instance is by calling the deployVirtualMachine or updateVirtualMachine commands via CloudStack’s API.

The other big drawback that I haven’t yet accounted for is somehow passing in a puppet environment. You could have multiple flavors of templates with the only change being the puppet environment in puppet.conf, but that goes against supporting as few templates as possible. Ideally, I’d like to set an “env” or similar fact in the user-data and get something to read the value and update puppet.conf before puppet starts up. I’ll have to revisit this in the future once I have a solution.