Removing Destroyed CloudStack Instances From Puppet Stored Configurations DB

If you’re using Puppet, and using Puppet to automatically generate configurations for other parts of your infrastructure by using stored configurations, then one thing you have to do is clean up the stored configurations database once you have destroyed a node, otherwise whatever stored config you created will continue to persist despite you removing that node.

Luckily for us, the folks at PuppetLabs ship a script that we can use to clean up after a node. It’s called puppetstoredconfigclean.rb and was located at /usr/share/puppet/ext/puppetstoredconfigclean.rb on my system. You can pass this script the fqdn of a host, or a list of fqdn’s delimited by a space and the script will remove all references from the stored configs DB.

So that’s great, except now I’m leveraging this thing called the cloud(in my case, the cloud is a CloudStack based private cloud) and I have the ability to spawn and destroy nodes on a whim….I don’t want to have to manually run this script each time I destroy a node.

I’ve been working with the cloudstack-php-client, so I figured I could easily wrap a script that would poll the CloudStack API for a list of VM’s (listVirtualMachines) and I could reconcile that list against the list of hosts stored in Puppet’s DB, and if there were any nodes on the Puppet side that were missing or in a “Destroyed” state in CloudStack, simply call the puppetstoredconfigclean.rb script to remove them from Puppet’s DB.

One small problem…not all of my infrastructure is based in the cloud…I have a bunch of bare metal legacy machines that I still need to keep track of and just because they weren’t in CloudStack doesn’t mean I want them to be removed from Puppet’s DB. Luckily for me, for each of these bare metal legacy boxes, I have a file named with each box’s FQDN sitting in a directory that acts as part of my external node classification system. For example:

$ ls /path/to/enc/directory
node1.example.com node2.example.com node3.example.com

What this means is that I should only remove a machine from Puppet’s DB if there is:

    (NOT a corresponding file in the ENC directory)
        AND
    (
        (if there is NOT an entry in CloudStack's listVirtualMachines response that matches)
            OR
        (if there is a match in the listVirtualMachines response AND state == "Destroyed")
    )

You can find the script that I run to accomplish this task on Github.

I run this script on one of my puppetmasters every few minutes….you might need to adjust the frequency based on your requirements. It’s written in PHP because I only had the CloudStack PHP client working at the time (there are now Perl and Python versions of the client, I just haven’t had a chance to refactor this script yet). This script assumes that you’re using MySQL for the stored configurations database and requires permission to SELECT from the hosts table in order to work correctly.