Source Control for Server Admin

So you manage a server, or a lot of servers alone or in a team, however you are doing this, you are going to be tweaking configuration files often and creating custom scripts for automation.  There are two tools I use for revision control - RCS for configuration files (generally) and SVN for scripts (generally).

RCS.  The classic.  All the documentation you will ever need is in the man pages.  Well that and some context for how to use it.  RCS creates revision files in place.  So if you change /etc/dhcpd.conf, it will create /etc/dhcpd.conf,v.  This is a very useful setup when controlling local files in arbitrary locations - like most of /etc on most of your servers.  There are a few caveats to keep in mind:

• RCS will put revision files (the ,v files) in an RCS folder if present
• The default behaviour is to remove a file from its current path on check-in

Keeping these in mind, this is my general pattern for working with files under /etc.

1. If there is no RCS folder (e.g. /etc/RCS), create it first
• mkdir -m 700 ./RCS
• Assuming your working folder is where the file in question is, this will create an RCS folder and protect it from other users (typically non-root)
2. If a file doesn't exist in RCS, check it in first
• ci -u dhcpd.conf && co -l dhcpd.conf
• ci is short for "check-in", unlike SVN or CVS, "ci" is the command and not an argument to "rcs"
• The -u "unlocks" the file leaving it in place (so dhcpd can read it)
• co is "check-out" and -l "locks" the file for editing
• I always leave files checked out to capture changes by other users or by the system (like rpm)
3.  If the file does exist in RCS, check for any un-committed changes
• rcsdiff dhcpd.conf
• This does a diff against the last checked-in version by default but you can specify a version if you want to compare against earlier changes
• Check-in any un-committed changes or find the person who made the changes and make them do it
4. The file should always be left checked-out (per above comment), otherwise check it out
5. Make changes
6. Check-in changes, and check-out the file for the next user
• ci -u dhcpd.conf && co -l dhcpd.conf
• Give a brief log message indicating what the changes were and again, leave the file checked-out to capture changes by the system or other users

Now the last useful command I'll mention there is rlog which lets you read the revision history log.

Now SVN is a proper centralized source control system.  They have excellent documentation on setting up a repository.  This is very useful for system admin scripts. 

Although most system administration related scripts won't ever have "releases" or "branches", you probably still want to follow the SVN guide and create at least a trunk in case you ever do need to tag a specific version.   There's no cost, so I use a trunk even though I've never used it because changing later is a problem.

With SVN you'll want to keep an updated local working copy ("tip") either on a shared NFS location or locally on each server.  How you do it is up to you, just create a cronjob to run "svn update /path/to/tip" and then you can always run scripts from that path.

RapidSVN is a great tool, well maybe not great, but works very well for sys admin anyhow and its readily available.  So check out your own working copy of the trunk with RapidSVN.  I configured RapidSVN to use gedit as my standard editor and meld as my diff tool.  

This gives you everything you need for day-to-day creating and maintain system configuration files and your toolbox of scripts for automated system maintenance.

Debugging Python Scripts

This is really just props for a site I found with a nice walk-through of using the Python Debugger - pdb.

http://pythonconquerstheuniverse.wordpress.com/2009/09/10/debugging-in-python/

pdb your built-in step-through debugger allowing you to inspect objects and all the usual things you need in developing a program.

Running the numbers

Two interesting tools popped up recently.

Good old Linux Counter has been passed down to a new maintainer. This is a classic project which attempts to get Linux usage data from user input. Its hard to tell if its particularly relevant, but it is interesting to see relative usage across platforms and by region. As for estimating global Linux use? Hard to be convinced this provides a good enough sampling to be very convincing. Nevertheless, I keep my machines at home registered there. Or at least some of them :P

Another one I really like is Debian Popcon which tracks popularity of Debian packages by installs and by "votes". Popcon is actually just a Debian package which phones home your installed package list and it is installed by default on some distros while not others. What I like about popcon is that when there are a wide variety of F/OSS tools available, you can check the list to see which tools are ranked highest so you can at least start by trying the most used tool rather than taking a total wild guess. For example, in looking for a SVN GUI tool, I did a "yum search svn" and there were a lot of hits. So I opened up popcon, search the list top to bottom for "svn" and took the highest hit which was a GUI tool which was RapidSVN. Well, then I checked with Dante which tool he used, but lo and behold, it was RapidSVN :)

Reorganizing Ubuntu Partitions

My personal PC at home died. It was an old PC no matter which way you look at it. Every part had been replaced or upgraded over time (case, PSU, optical drive, hard drive, memory, CPU, mainboard, NIC, video card) so knowing it's actual age is pretty hard, but it looks like "Friday" as a PC existed for 7 years. Checking my blog, the first reference I found was October 31, 2004 indicating Friday was the new name for an old PC called Michael.

Time for a new new PC. I've reused the optical drive but everything else is new in Agnes (from Immortality by Milan Kundera). I did your basic "install Windows first, Ubuntu second" so pretty much just a mommy-install. Until I realized I really hadn't made a big enough Windows partition.

I figured it would be a pain, moving the first Ubuntu partition back on the drive so I backed everything up and booted from the Live CD. "gparted" is included on the live CD and it was painless to shrink the Ubuntu partition, move it "right" and extend the Windows partition. I didn't have to reinstall grub or do anything else, it pretty much just worked - for both OSes. It's always so nice when things just work.

But I will say, it's pretty dumb that Ubuntu doesn't use LVM. As I have posted before, LVM is very useful. What would be nice is if I could have just lumped most of the free space into LVM and then just carved out an LV for home and another for media so I could grow them as needed. Rather than fiddle too much with that though, I ended up just going with a relatively large /home partition and will just grow that as needed and then if I need space for other things - like more storage under the 'doze, I can put a partition at the end of the disk.

Heartbeat

I recently have tested out running Heartbeat (finally, took too long to get to this, but that's another story). This is a cluster resource manager (CRM) which polls nodes in a cluster and brings resources up when a node failure is detected.

It's interesting. I wouldn't call it elegant really, maybe the newer Pacemaker would seem cleaner. But it is simple and at least in testing it is effective especially when combined with DRBD which I posted on earlier. The thing is where DRBD really seems built for top-notch resiliency and flexibility, Heartbeat seems it will work, but it's not obvious that you'll get what you expected - maybe it's just the documentation on DRBD was really well done.

At any rate, there is great documentation on getting Heartbeat up with DRBD both from the CentOS wiki and from DRBD. I used heartbeat with drbd83 in CentOS.

What Heartbeat will do is listen for heartbeats from a peer node in a cluster and if a peer goes down, it will bring up services on the working node. There's a handful of important things about this to keep in mind.

First is the heartbeat - this is just a stand-alone network connection between two nodes so if that connection goes down or the heartbeats get choked out by competing traffic, Heartbeat may well decide you have a node failure. This is not a trivial problem because now that Heartbeat can kill services on an active node, this is potentially an new point of failure. And this is common to many HA configurations including DRBD itself though as we know, it will identify split-brain and gives you some recourse for repairs. So the suggestions here are to use a dedicated connection, preferably a point-to-point connection with a cross-over cable or a serial port - and this is not uncommon for clusters (the point-to-point connection) like in this white paper for Microsoft Storage Server.

Then there is the issue of resource management - when the CRM is managing the resources, the usual OS procedures should not. If Heartbeat is in charge of bringing up MySQL, you shouldn't be starting MySQL from the init scripts when the OS boots. Now the nice thing with DRBD is that it's behaviour is consistent with this paradigm - when DRBD resources start up, they are in "secondary" mode and cannot be accessed by the OS. So if you have a file share protected by DRBD, Samba wouldn't be started by the OS, and likewise, that file system would be unavailable when the OS starts (by default at least). So here, Heartbeat makes a lot of sense. You take a 2 node cluster for example, when the nodes start up, Heartbeat looks for the peer, picks someone to become active, and then would make "primary" the DRBD resource on that peer, mount the file system, start smb. On the stand-by node, you would both have 'smb' off and the file system would not be writeable which helps ensure consistency.

I guess I could go on about Heartbeat quite a bit, but there's one last thing to mention specifically here and that's the style of cluster. There are "R1" style clusters which are simple but limited to 2 resources (and other limitations) and then there are CRM enabled clusters which are more robust but more complicated to configure. I have only used R1 because it was sufficient for my needs - 2 nodes, one was known "preferred", keeping cluster configuration in sync "manually" wasn't onerous. But CRM enabled clusters are more interesting because you can add more nodes and it will distribute the cluster configuration automatically, etc.

The one thing I haven't really touched on is the quorum which others who are more familiar with cluster management will be more familiar with than I. Basically with Heartbeat in an R1 style cluster, there isn't going to be a quorum. Your configuration is maintained pretty much manually, services are only running on one node, etc. In CRM style Heartbeat or other application clusters, the quorum is what all the nodes agree on and typically stored in a file. On Windows Storage Server and other clusters, the quorum is stored on the shared disk which means any problem there means the cluster fails. With Heartbeat, the quorum file is copied among the nodes but this is susceptible to becoming out of sync like if there is a communication failure on the heartbeat channel leading to a split brain. Or this is my limited understanding of this. At any rate, it is a problem and it isn't trivial in working with active/active or multi-node configurations.

Cache in Openfire

In the course of troubleshooting the office Jabber server the other day, I came across some interesting info about the various caches that Openfire has. If you log on to the admin console of your Openfire server and go to the cache summary page, you can see what the usage and effectiveness of your various caches are. Specifically, I found that a couple caches were full - Roster and VCard. The Roster cache was limited to 0.50MB by default it seemed and it's effectiveness was less than 20% at the time.

It is a fairly common issue and it has been discussed in the Ignite Realtime forums. The solution posted is to set a couple system properties to override the default:

cache.username2roster.size
cache.vcardCache.size

Both of these are given in bytes. The post in that thread says to go to 5MB, I found that my Vcard cache didn't need to be much bigger than the default and the roster cache only needed 2 or 3 MB.

After changing this, both cache hit rates are closer to 90%.

Our system is very small (less than a couple hundred users total), so the effect is not big on regular usage. But well worth checking on your server as it is a quick and easy optimization.

Again with the tapes

In a previous post I said that to get around devices changing their numbering, it was useful to use the /dev/tape/by-id instead of the generic /dev/nst0. Unfortunately, this is also imperfect I've just learned as the device which was previously "scsi-35000e11138aa0001-nst" this time came up as "scsi-35000e11138aa0001". And you can guess how gracefully the software handled that (not at all). Now I don't know if was a driver update (possible) or if the device was switched to a different SAS interface (also possible), or maybe just the gremlins. Whatever it was, once again, I had to reconfigure the software to find the new device ID.