Tutorial I gave at LISA 2014 http://goo.gl/G0TLfJ

This is a talk about running puppet in the enterprise, or at scale. The original title of Mastering Puppet was Puppet in the Enterprise, the talk was to present the ideas in the book.

The tutorial files are located on github at: https://github.com/uphillian/lisa2014

I uploaded the lisa2014.iso Live image used in the demo to dropbox, get it here https://www.dropbox.…


Talk I gave at puppetcamp seattle 2014 http://goo.gl/b2NISc

In the first part, Troubleshooting Puppet talks about problems with communication, how to make sure puppet is running and connecting to a master. In the second part, the talk moves on to compilation and catalog application issues. Real world solutions are presented throughout.


I've been running into the 800 node limit on mcollective and splitting up my nodes into subcollectives. I had a spot where I couldn't split up the nodes, so I started looking at why we were hitting this 800 node wall.

I'm using activemq with the ssl plugin, after turning on all the debugging I could find in activemq, it turns out it's just a simple resource limit problem.

With activemq running, I waited for my nodes to connect and watched the number of threads on the active java process. (This is after increasing the memory limits for activemq as described on…


While configuring OMD (yes, Orchestral Manoeuvers in the Dark, no, not really) I ran into a point at which apache was supposed to run as the OMD user for check_mk. Hard coded into the check_mk configuration is a call to

sudo su - check_mk -c check_mk\ --automation\ *

I've seen this many times, sysadmins doing sudo su -, but why? It reminds me of an admin who would routinely do cat file | less, or my favourite cat file | grep something | wc -l when grep -c something file would be just…


I'm not sure of the utility of this, but maybe it'll be useful to someone else. I was requested to output all the facts from a system in xml, not wanting to type much I made the following script...

#!/usr/bin/env ruby

require 'facter'
require 'rubygems'
require 'activesupport'

facts = {}

for fact in Facter.list.sort
facts[fact] = Facter.value(fact)

xml = facts.to_xml(:root => "facts")

print xml

The output looks like the following:…



machine A ( provides resource A on port 8888
machine B ( needs to access resource A

without modifying machine B (not allowed), create machine C and have any traffic to machine C on port 8888 forwarded to machine A. Then tell machine B that machine C is machine A and nobody is the wiser. None of the examples I found online had this working properly.

I eventually came up with the following.
machine C has two interfaces:
one that is on the same network as machine B - eth0
one that is on the…


I routinely used to transfer data between systems using rsync. Since I wanted the communication to be secure I used ssh-keys, I noticed that my trick for using a command in the key isn't terribly well documented, so here is how I do it...

Goal: Keep /opt/before on machine B in sync with /opt/after on machine A.

On machine A, create an ssh key for this

$ ssh-keygen -f id_rsync

Copy id_rsync.pub from machine A to machine B, create an rsync account for the transfer, place the key into the authorized_keys file on…


I was trying to allow a user to sudo to another account and run a specific command. I'm not a fan of getting them to run through su since it doesn't make much sense to involve a third tool in the equation. I could get it working with the following:

theiruser ALL=(runasuser) NOPASSWD:/usr/local/bin/script.sh

The user could run script.sh with sudo -u runasuser /usr/local/bin/script.sh and it worked as expected but if they tried sudo -iu runasuser /usr/local/bin/script.sh they got prompted for a password as the command…


Trying to fix an issue with snmp, I started by building an snmp module using audit2allow. It kept failing to load, and the error message is a little cryptic...

[root@host thomas]# semodule -i snmp.pp
libsepol.print_missing_requirements: snmp's global requirements were not met: type/attribute snmpd_t (No such file or directory).
libsemanage.semanage_link_sandbox: Link packages failed (No such file or directory).
semodule: Failed!

The .te file for the module looks like this:


Had an sssd process spinning and using 100% cpu. Did an strace on it and saw that it was complaining about too many open files.

pid accept(24, 0xaddress, [110]) = -1 EMFILE (Too many open files)

getting the number of open files for the process.

# lsof -p $(pidof sssd_pam) |wc -l

Looking at the limits for sssd, I saw that the nofile was set to 1024, which appears to be the default everywhere I tried.

# cat /proc/$(pidof…