I find this really useful, just a call out to sed so I don't have to remember the syntax.


ssh_delete_key() {
sed -i -e ${1}d ~/.ssh/known_hosts
}
alias sshdel=ssh_delete_key


Usage:

[thomas@laptop: ~] $ ssh 192.168.0.1
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
b7:da:35:ad:65:1c:5b:09:cf:a5:b3:e6:e7:0e:cf:43.
Please contact your system administrator.
Add correct host key in /home/thomas/.ssh/known_hosts to get rid of this message.
Offending RSA key in /home/thomas/.ssh/known_hosts:426
RSA host key for 192.168.0.1 has changed and you have requested strict checking.
Host key verification failed.
[thomas@laptop: ~] $ sshdel 426
[thomas@laptop: ~] $ ssh 192.168.0.1
The authenticity of host '167.143.72.26 (167.143.72.26)' can't be established.
RSA key fingerprint is b7:da:35:ad:65:1c:5b:09:cf:a5:b3:e6:e7:0e:cf:43.
Are you sure you want to continue connecting (yes/no)?


While at LinuxFest Northwest I picked up an Old Raspberry Pi from the Yard Sale. I didn't realise it was a Model A...took me a while to figure out a use for it, not much memory or processing power.

I installed RuneAudio on it but really just wanted Pandora. I found pianobar and the web interface, Patiobar and got them working. I did have to install a few things to get it working and it did involve some tricks with pacman. Probably could have ditched runeaudio but learning how to use arch again is fun.

pianobar had poor quality sound when I first started it, so I switched to the oss driver. Edit /etc/libao.conf and change default_driver to oss. Then run pianobar from the helper script aoss. I had to install libpulse as well to get that working.

Making them start automatically turned out to be a big hack so here's how...

Create a tmux startup script.

#!/bin/bash

export HOME=/root

#check if you can reach pandora.com
while /bin/true
do
getent hosts pandora.com
if [ $? == 0 ]; then
break
fi
done
tmux new-session -s pianobar -n pianobar -d
#tmux update-environment HOME
tmux send-keys -t 0 '/usr/bin/aopiano' enter
tmux new-window -t pianobar:1 -n Patiobar
tmux select-window -t pianobar:1
tmux send-keys -t 0 'cd /root/Patiobar; export HOME=/root; node index.js' enter

This checks to see if pandora.com is resolvable. The network-online.target wasn't working for me, the network was up but pandora.com wasn't yet resolvable, so pianobar would exit. The script starts tmux and creates two windows, in the first pianobar is run, in the second Patiobar. pianobar needs to be configured to load a station automatically or Patiobar will be sending the wrong input to it via the fifo.

The aopiano script is just the aoss script modified to start pianobar directly. I probably don't need it anymore, but already had it when I was testing.

#!/bin/sh

# A simple script to facilitate the use of the OSS compatibility library.
# Usage:
# aoss
LIB=lib
if [ -d /proc/asound ]; then
prefix=/usr
exec_prefix=${prefix}
LD_PRELOAD=${exec_prefix}/\$LIB/libaoss.so${LD_PRELOAD:+:$LD_PRELOAD} exec /usr/bin/pianobar
else
exec "$@"
fi
exit 1

So now on boot the raspberry waits for pandora to resolve, then starts tmux with pianobar and Patiobar running. The nice thing is that I can connect to the tmux session and control pianobar directly. I can also connect to the Patiobar session on port 3000 from any machine in the house.

I use a USB camera on my work laptop and the camera sometimes doesn't show up in my video conferencing app. I verified that the camera is showing in the usb list with lsusb (a script I got from https://github.com/jlhonora/lsusb which is great btw).

dtruss showed that the VDCAssistant was being accessed by my apps, so I tried killing that and boom the camera shows up.

So now I aliased camkill='sudo pkill VDCAssistant'

I spent far too long diagnosing why this worked in one set of hosts and not another. When the vrrp_script fails on one host, it's supposed to move the service to the other host, that's the whole point right?

This was working great in one pair of hosts, but not another. Turns out maths are hard and I'm dumb. The weight of the check script is subtracted from the priority. If priorityA - weight is still greater than priorityB, nothing happens. Someone thought it would be cool to change the weight of one of the hosts from 100 to 150. The weight of the test was only 2, so 148 was still a higher priority than the other host (100).

After changing the priority back to 101 everything started working normally. Alternatively I could have made the weight of the check 51.

Hopefully you didn't waste as much time as I did on that dumb a mistake.

Tutorial I gave at LISA2016

git crash course - slides
tutorial workbook

Talk I gave at PuppetConf 2016
Slides are here

Talk I gave at puppetcamp seattle 2016, slides at
goo.gl/Y61I0S

Some of this isn't a great idea for production...some.

Had a problem where I wanted to modify /etc/cups/cupsd.conf but wasn't sure who else might touch the file. I opted to use Augeas and quickly learned it wasn't as easy as I thought it might be...

The cupsd.conf configuration file uses a syntax similar to Apache configuration files, it uses the same Augeas lens (Httpd.lns). Changing directives is a bit of an issue, but I found the solution by reading the source for the Httpd.lns (/usr/share/augeas/lenses/dist/httpd.aug or /opt/puppetlabs/puppet/share/augeas/lenses/dist/httpd.aug)

What I wanted to do was turn off the port 631 listener, the line that starts with Listen localhost:631, in augtool this looks like the following:
[code type="shell"]
augtool> ls /files/etc/cups/cupsd.conf/
directive[1]/ = MaxLogSize
#comment[1] = Configuration file for the CUPS scheduler. See "man cupsd.conf" for a
#comment[2] = complete description of this file.
#comment[3] = Log general information in error_log - change "warn" to "debug"
#comment[4] = for troubleshooting...
directive[2]/ = LogLevel
directive[3] = PageLogFormat
#comment[5] = Only listen for connections from the local machine.
directive[4]/ = Listen
directive[5]/ = Listen
[/code]
So, this is where it gets a little weird, I just want to make sure that anything with a Listen *:631 matches, so I use regex.

augtool> get /files/etc/cups/cupsd.conf/directive[self::directive="Listen"]/arg[self::arg=~ regexp(".*:631")]
/files/etc/cups/cupsd.conf/directive[self::directive="Listen"]/arg[self::arg=~ regexp(".*:631")] = localhost:631

To remove that line, I just need to use rm:

augtool> rm /files/etc/cups/cupsd.conf/directive[self::directive="Listen"]/arg[self::arg=~ regexp(".*:631")]
rm : /files/etc/cups/cupsd.conf/directive[self::directive="Listen"]/arg[self::arg=~ regexp(".*:631")] 1
augtool> save
Saved 1 file(s)

Now when I view the cupsd.conf file I see the localhost:631 is gone but I still have a line with "Listen" on it.

# Only listen for connections from the local machine.
Listen
Listen /var/run/cups/cups.sock

That might work but it looks bad to me, so I opted to change my augeas code to remove all Listen lines and then add back the UNIX socket instead.

augtool> rm /files/etc/cups/cupsd.conf/directive[self::directive="Listen"]
rm : /files/etc/cups/cupsd.conf/directive[self::directive="Listen"] 3
augtool> set /files/etc/cups/cupsd.conf/directive[self::directive="Listen"] Listen
augtool> set /files/etc/cups/cupsd.conf/directive[self::directive="Listen"]/arg /var/run/cups/cups.sock
augtool> save
Saved 1 file(s)

Now when I look in the file, there's only one Listen line.

# grep Listen /etc/cups/cupsd.conf
Listen /var/run/cups/cups.sock

Going back and doing the same for Browsing and BrowseLocalProtocols, I ended up with the following Augeas resource for Puppet.

augeas { 'cups listen':
  incl    => '/etc/cups/cupsd.conf',
  context => '/files/etc/cups/cupsd.conf',
  lens    => 'Cups.lns',
  changes => [
    # Do not listen to anything but the unix socket
    "rm directive[self::directive='Listen']",
    "set directive[self::directive='Listen'] Listen",
    "set directive[self::directive='Listen']/arg /var/run/cups/cups.sock",
    # Don't browse local printers
    "set directive[self::directive='Browsing']/arg Off",
    "rm directive[self::directive='BrowseLocalProtocols']",
    "set directive[self::directive='BrowseLocalProtocols'] BrowseLocalProtocols",
    "set directive[self::directive='BrowseLocalProtocols']/arg none",
  ],
  require => Package['cups'],
  notify  => Service['cups'],
}

I ran into this problem recently, certificates were verifying ok but were revoked somewhere along the line. I wanted to check against the CRL but it's a somewhat undocumented feature (fixed in openssl 1.0.2). The -crl_check option checks your cert against the CRL listed in the certificate, but only if that is listed and accessible remotely.

To get the crl_check to work, append the CRL to your CA and then specify the -CAfile option to whatever openssl command you are using (I used s_client and verify successfully).

$ cat /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem > /tmp/ca_combined.pem
$ openssl verify -crl_check -CAfile /tmp/ca_combined.pem /tmp/yourface.pem
/tmp/yourface.pem: CN = yourface.localdomain
error 23 at 0 depth lookup:certificate revoked
$ openssl verify -CAfile /tmp/ca_combined.pem /tmp/yourface.pem
/tmp/yourface.pem: OK

Without the -crl_check, the certificate comes back valid.

(Puppet did tell me that the certificate was revoked, but I didn't believe it, had to verify with OpenSSL, if OpenSSL says it's revoked, I'll believe it.)

It came up twice that I had to do this, so I decided to see if I could make something simple that solved my problem. I'm sure there's a project to do this already, but here goes. I have a list of machines, I want to see if they respond to a ping so I can determine if they are up or not.

I wanted to do this two ways, either by feeding in a list from stdin or by giving hostnames as arguments. This took a few revisions to get just right, but here's what I came up with:


#!/bin/bash
# ping many hosts and return up or down for each

usage () {
echo "Ping Many Hosts"
echo "Usage: $0 host1 [host2 host3]"
echo "echo host1 | $0"
echo "ping all hosts, return up or down for each"
}
# read hostnames from stdin or command line arguments
if [ $# -eq 0 ]; then
#hosts=$(cat /dev/stdin)
read -t 2 hosts
else
hosts="$*"
fi

if [ "$hosts" == "" ]; then
usage
exit 1
fi
fail=0
# loop over hosts, ping once, wait 1 second
for host in $hosts
do
echo -n "$host "
ping -c 1 -t 1 -q $host >/dev/null 2>&1
if [ $? == 0 ]; then
echo up
else
echo down
# return a fail for the whole run if you can't reach any of the hosts
fail=1
fi
done
exit $fail

The secret sauce is giving the timeout to read, so that if you don't give any input, it gives up after the timeout and gives you the usage information.
Now I have to do the same thing with netcat for a specific port, of course the port would be an argument, so I'll need to modify the logic to make that work too...