Sysadmin Ramblings

using rsync with ssh keys via authorized_keys and command=”rsync …”

uphill — Fri, 29 Jan 2010 18:58:18 +0000

Scenario I

Backup directory /mnt/one from server pris to client directory /home/user/two on client deckard by initiating the the copy from the client deckard. (i.e. send files from the server to the client)

create ssh keys using ssh-keygen
[user@deckard ~]$ cd .ssh [user@deckard .ssh]$ ssh-keygen -t dsa -f deckard Generating public/private dsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in deckard. Your public key has been saved in deckard.pub. The key fingerprint is: 17:f4:69:30:6c:67:5a:73:2e:6f:ba:4b:8b:94:2a:f9 user@deckard.example.com The key's randomart image is: +--[ DSA 1024]----+ | .+ | | .o+=.. | | ..=++ | | .o. . | | S . o | | . . o | | . o .o | | o o o.. | | oE . +o | +-----------------+
create a new user for the sync operation and copy the public key you just created to pris
[root@pris ~]# useradd deckardsync [root@pris ~]# su - deckardsync [deckardsync@pris ~]$ mkdir .ssh [deckardsync@pris ~]$ cd .ssh [deckardsync@pris .ssh]$ scp user@deckard:.ssh/deckard.pub authorized_keys deckard.pub 100% 1194 1.2KB/s 00:00
make the /mnt/one directory accessible to deckardsync (or ideally owned by this user)
[root@pris ~]# chown deckardsync /mnt/one
edit authorized_keys and add the commands for rsync into the key
command="rsync --server --sender -vlogDtprCz . /mnt/one/" ssh-dss AAAAB3NzaC1kc3MAAACBANbyPA4Vkem1tXrBcmkc9+SHeBrgHKbeBdS2MZKMBT/CsPWPSwMFQGg3GzX2KFrIVlZW/+OfkFcrZabMxtLb4CfvFgZsK18hcyYWZobhtpzqfsoolVnWbHdcmxFqyUq9fIK5iPA2UnvLoLRCDuklQNZ+V8o7fiCiPzXw5sqw3weRAAAAFQDKbAINhyt3OzJhP680PqrA9vHNFwAAAIB0mmnu9rfUKnSAH8UV068H28NEaNuIvSzQchvsPpBZmLpN/yr0mUbWdUJtVfFO72fbhQW+gQmEydCoPgehAGCx0g5jcs+0J7nhDlCqCqYAluD/79jJvEr7Tc33u0QTJSEX9My5X6OVtKByGfGPIyeLdhdsM2s70xbXKpfV4j8KpgAAAIEAnbxqsdbxpZ/vKZMJCW4TuHzOk76By5HHHHRb6XIUTsImQmoHrH1T2ioVil6eNp+V02hbYzbs8OuMqj6ne3gLzIyPqIP1OHuusisrLKgtWTC74lZnZ48d9QCyUHI48yZyoISs0HvEvC08LHYqsq1z1ntCHAde4iszE2TAMsYBat4= user@deckard.example.com
Note: there are no line breaks in the above key… the file has only one line.
copy something into /mnt/one on pris
[root@pris ~]# cd /mnt/one [root@pris one]# cp -a /usr/share/doc/rsync-2.6.8 .
start rsync on deckard using the ssh key
[user@deckard .ssh]$ rsync -e 'ssh -i deckard -l deckardsync' -Cavz pris:/mnt/one /home/user/two/ receiving file list ... done ./ rsync-2.6.8/ rsync-2.6.8/COPYING rsync-2.6.8/README rsync-2.6.8/tech_report.tex sent 98 bytes received 14416 bytes 29028.00 bytes/sec total size is 36507 speedup is 2.52 [user@deckard .ssh]$ ls ~/two rsync-2.6.8

Scenario II

Backup directory /home/user/two from client deckard to server directory /mnt/three on server pris

The steps involved here are essentially the same with only one small change in the authorized_keys, drop the —sender option to rsync (since pris is no longer the sender)

create new ssh key for the transfer in this direction.
[user@deckard .ssh]$ ssh-keygen -t dsa -f pris Generating public/private dsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in pris. Your public key has been saved in pris.pub. The key fingerprint is: 75:67:7a:ca:2f:b2:11:f4:83:50:27:07:50:0b:55:a3 user@deckard.math.ias.edu The key's randomart image is: +--[ DSA 1024]----+ | o+*o= | | o * . | | . E . o | | + + + | | S o + . | | o + | | . o | | .... | | .o .. | +-----------------+
copy the key to pris and append it to authorized_keys
[root@pris .ssh]# scp user@deckard:.ssh/pris.pub . user@deckard's password: pris.pub 100% 617 0.6KB/s 00:00 [root@pris .ssh]# echo -n 'command="rsync --server -vlogDtprCz . /mnt/three" ' >>authorized_keys [root@pris .ssh]# cat pris.pub >>authorized_keys
initiate the transfer from deckard, this time acting as the sender not the receiver (flip sender for receiver)
[user@deckard .ssh]$ rsync -e 'ssh -i pris -l deckardsync' -Cavz /home/user/two/ pris:/mnt/three building file list ... done ./ rsync-2.6.8/ rsync-2.6.8/COPYING rsync-2.6.8/README rsync-2.6.8/tech_report.tex sent 14422 bytes received 98 bytes 29040.00 bytes/sec total size is 36507 speedup is 2.51

Now you just need to put that rsync line in a cronjob and you’ll have automatic syncing. (if you do, remember to use the full path for the ssh-keys you generated). The nice thing here is that if the key should be discovered, the only thing the attacker can do is run rsync.

puppetserver certificates being signed in the future

uphill — Fri, 29 Jan 2010 16:37:11 +0000

We had a problem where new clients couldn’t get their keys signed properly by the puppetmaster. Both the client and the server were in perfect sync with our ntp server. date on both machines returned the expected results. We are running mongrel so I went down the wrong path of thinking apache was to blame for the time problem. It wasn’t until I started going through the certificate_factory stuff that I found the problem. We’d errors on the certs like this:

[root@puppet ~]# cd /var/lib/puppet/ssl [root@puppet ssl]# openssl verify > -CAfile ./certs/ca.pem ./certs/client.example.com.pem > ./certs/client.example.com.pem: /CN= client.example.com > error 9 at 1 depth lookup:certificate is not yet valid

Outputing the certificate showed that the cert was being signed for a future date, even though the time on the machines is correct.

[root@puppet ssl]# date Fri Jan 29 11:31:32 EST 2010 [root@puppet ssl]# openssl x509 -text -in ca/signed/client.example.com.pem |grep -A2 Valid Validity Not Before: Feb 17 13:28:04 2010 GMT Not After : Feb 16 13:28:04 2015 GMT

Going through the code I found that the date was being set in certificate_factory.rb

def set_ttl # Make the certificate valid as of yesterday, because # so many people's clocks are out of sync. from = Time.now - (60*60*24) @cert.not_before = from @cert.not_after = from + ttl end

Just for fun I ran the command through interactive ruby (irb) and discovered the source of the problem.

[root@puppet ~]# ntpdate time.example.com 29 Jan 09:02:45 ntpdate[9117]: step time server 192.168.0.1 offset -6377207.794727 sec [root@puppet ~]# irb irb(main):001:0> Time.now => Tue Apr 13 05:25:50 -0400 2010 irb(main):002:0> quit [root@puppet ~]# date Fri Jan 29 08:59:07 EST 2010

I still don’t know why this happened, it’s not a puppet bug, it’s a ruby bug. date was returning the expected results. I checked Timezones, everything, all were good. It was time for a kernel upgrade, so I did the upgrade and rebooted. I haven’t seen the problem since :-/ The machine in question is a kvm running on version 88, I know there are some clock skew problems with earlier kvm’s but this is not really a skew, it’s far in the future…and the date was still being show as correct. So ruby must’ve been calculating the date wrong somehow, it doesn’t really make sense…comments welcome. Anyway, if this happens to you, maybe try irb and see if ruby thinks the date is wrong.

extract private key and cert from pkcs12 (cert8.db/key3.db)

uphill — Mon, 25 Jan 2010 18:55:41 +0000

using fedora-ds/redhat-ds it creates cert8.db and key3.db to store the certs. I wanted to extract the private key as PEM so I could import it elsewhere.

[root@ldap] cd /etc/dirsrv/slapd-ldap [root@ldap] pk12util -o cert.p12 -n 'server-cert' -d . Enter Password or Pin for "NSS Certificate DB": Enter password for PKCS12 file: Re-enter password: pk12util: PKCS12 EXPORT SUCCESSFUL [root@ldap] openssl pkcs12 -in cert.p12 -out cert.pem -nodes -clcerts Enter Import Password: MAC verified OK [root@ldap] cat cert.pem Bag Attributes friendlyName: server-cert localKeyID: 10 F4 C2 F6 01 3C 66 AA 72 35 C9 A7 DA B9 12 3F 11 A1 98 F6 Key Attributes: -----BEGIN PRIVATE KEY----- MIICdwIBADANBgkqhkiG9w0BAQEFAASCAmEwggJdAgEAAoGBALA7rSWdSk4CVHef ... BnevX/uQwZ3L1Qo= -----END PRIVATE KEY----- Bag Attributes friendlyName: server-cert localKeyID: 10 DD CC EE BB 3C 33 AC 72 35 C9 A7 DA B9 12 3F 11 A1 98 F6 subject=/C=US/ST=Any State/L=Any Town/O=Example/CN=ldap.example.com issuer=/C=US/ST=Any State/L=Any Town/O=Example/CN=certmaster.example.com -----BEGIN CERTIFICATE----- ChMcSW5zdGl0dXRlIGZvciBBZHZhbmNlZCBTdHVkeTEeMBwGA1UECxMVU2Nob29s ... gIP23WbaOw4DygMwXfbJwF5K0xxv+NALlpoaZw== -----END CERTIFICATE-----

I couldn’t figure out how to do it with pk12util and certutil alone, the key was using openssl after exporting with pk12util…

puppetmaster Error 400 on SERVER: Too many connections

uphill — Fri, 15 Jan 2010 19:30:52 +0000

Was getting this error on our puppetmaster that only had a few clients. Turns out it’s just an error from mysql being passed down the line. We share our mysql for puppet with multiple servers. Restarting mysql saved the day… Read http://dev.mysql.com/doc/refman/5.1/en/too-many-connections.html for instructions on increasing the limit…

mysql> show variables like 'max_connections'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | max_connections | 100 | +-----------------+-------+ 1 row in set (0.00 sec) mysql>

increasing the limit in /etc/my.cnf

[mysqld] datadir=/var/lib/mysql ... max_connections=200 mysql> show variables like 'max_connections'; +-----------------+-------+ | Variable_name | Value | +-----------------+-------+ | max_connections | 200 | +-----------------+-------+ 1 row in set (0.00 sec) mysql>

weird device names like devXXXXX with ifconfig on Fedora/RHEL

uphill — Wed, 13 Jan 2010 20:44:46 +0000

We had a machine that would keep coming up with devXXXXX where XXXXX is a seemingly random number. We tried ifrename to no avail, modprobe -r didn’t help, modules.conf didn’t help. It turned out that the HWADDR line in the ifcfg-ethX file was wrong. After fixing the line we were able to do ifup ethX and the devXXXXX went away.

rooting the droid (Motorola A855) rooted

uphill — Thu, 10 Dec 2009 15:56:40 +0000

This is my first android phone, so I thought I’d share my experience of rooting the phone. All credit to Zinx Verituse over on Alldroid.org

Note: I just updated to the update released December 10th for the droid and was able to reapply the update.zip.

The instructions are here

Download this file and rename it update.zip on the sdcard.
turn off the phone
turn on the phone holding down the x key on the keyboard
When you see the Motorola symbol, press Vol+ and Camera buttons together.
An onscreen menu will appear, use the d-pad to navigate to update.zip and type return to apply
After the phone reboots, Go to Settings -> Applications -> Development and enable USB Debugging
Connect the droid to your desktop, download the sdk
unpack the sdk and navigate to tools directory
as root/administrator run adb start-server
next run adb shell
su to root /system/bin/su
remount the root filesystem rw
# mount rootfs / rootfs ro 0 0 tmpfs /dev tmpfs rw,mode=755 0 0 devpts /dev/pts devpts rw,mode=600 0 0 proc /proc proc rw 0 0 sysfs /sys sysfs rw 0 0 tmpfs /sqlite_stmt_journals tmpfs rw,size=4096k 0 0 none /dev/cpuctl cgroup rw,cpu 0 0 /dev/block/mtdblock4 /system yaffs2 ro 0 0 /dev/block/mtdblock6 /data yaffs2 rw,nosuid,nodev 0 0 /dev/block/mtdblock5 /cache yaffs2 rw,nosuid,nodev 0 0 /dev/block/mtdblock0 /config yaffs2 ro 0 0 /dev/block//vold/179:1 /sdcard vfat rw,dirsync,nosuid,nodev,noexec,uid=1000,gid=1015,fmask=0702,dmask=0702,allow_utime=0020,codepage=cp437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro 0 0 # mount -o remount -o rw -t yaffs2 /dev/block/mtdblock4 /system
I don’t know if the device will always be the same (/dev/block/mtdblock4), best to check first.
now make a copy of sh and call it su-something or replace su
# cd /system/bin # cat sh > su or # cat sh >su-fake # chmod 4775 su or # chmod 4775 su-fake # exit

You now have either su or su-fake that you can run from terminal emulator on the phone and become root. To start using your new privileges, install busybox and symlink to it for all the tools that are missing…

puppet upgrade problem 0.25 “could not convert from pson”

uphill — Fri, 20 Nov 2009 21:35:56 +0000

After upgrading to 0.25, the following error occurs:

Could not retrieve catalog from remote server: Could not intern from pson: Could not convert from pson: Could not find relationship target ”

This turned out to be because of recipes using exec without naming the exec. Example

exec { "cat /that/file": unless => "something", path => "/bin", refreshonly => false }

should be rewritten as

exec { "cat that file": command => "cat /that/file", unless => "something", path => "/bin", refreshonly => false }

The error goes away after making the change and all is well.

no more adsense

uphill — Wed, 04 Nov 2009 11:44:53 +0000

I don’t really see the benefit of using adsense anymore. Everyone knows they’re ads, no one clicks. It doesn’t pay my isp bills anymore, so I’m dropping it. They were only annoying anyway.

hard coding a library path into an executable while building an rpm

uphill — Tue, 06 Oct 2009 21:03:18 +0000

The problem I was trying to fix here was that I had a package that required a much newer version of a library than the system had installed on it. I didn’t want to ruin the stability of the system by updating the library so I build it and placed it in a non standard location (harkening back to the solaris days I guess). If you can’t guess what the problem was, I called the library package qt4-vlc…hint hint.

That part went fine, but whenever I tried to build my package that was supposed to use qt4-vlc, it would use the system libs in %{_libdir}/qt4…I tried to use rpath as a solution but couldn’t get the syntax right. I looked at the gnu documentation but that didn’t work because gcc kept complaining that it didn’t know what the -rpath option meant.

The solution was to escape all the -rpath options that are for the linker (ld) and not gcc. Using -Wl, passes arguments to the linker and ignores them in the compiler. The final line I arrived at is:

LDFLAGS='-Wl,-rpath -Wl,%{_libdir}/qt4-vlc' $LDFLAGS export LDFLAGS

I put this in the %build and the %install stages. In %build I put it before %configure and in %install it’s before %makeinstall. How to read this is that the -Wl, just says, pass the next argument to ld, so

-Wl,-rpath -Wl,%{_libdir}/qt4-vlc

is sent to the linker as

-rpath %{_libdir}/qt4-vlc

Once I figured that out, it was all good. But it seems a bit silly to have two of the -Wl, clauses, since you can just put in another comma. In the end I just used this:

-Wl,-rpath,%{_libdir}/qt4-vlc

It’s not only more compact, it’s easier to read..

building rpms, spec files, rpmbuild, simple tutorial on making rpms by example

uphill — Tue, 22 Sep 2009 15:04:56 +0000

I posted some additional material on building rpms in the appendix of the book. I cover how to build an example spec file from scratch. This is similar to what I did in my presentation, just with a lot more detail. I hope to expand this section to cover nested packages and kernel modules. Those sections are not done yet…read it here.