Off you go... into the purple yonder! » Sysadmin http://ward.vandewege.net/blog Tue, 29 Jun 2010 19:11:16 +0000 http://wordpress.org/?v=abc en hourly 1 nagios, mdadm and snmp http://ward.vandewege.net/blog/2009/11/nagios-mdadm-and-snmp/ http://ward.vandewege.net/blog/2009/11/nagios-mdadm-and-snmp/#comments Mon, 09 Nov 2009 20:47:22 +0000 ward http://ward.vandewege.net/blog/?p=510 I found this script while looking for a simple script to monitor mdadm arrays. The script is fine, but it has a subtle bug – it will never report an error because the –detail parameter is missing in the call to mdadm. I modified the script a bit, like so:

#!/bin/sh
# (c) 2008 Jasper Spaans 

worst=0
msg=""

for dev in /dev/md?* ; do \
  mdadm --misc -t --detail $dev >/dev/null
  status=$?
  if [ $status == 0 ]; then
    msg="${msg} ${dev}: ok"
  elif [ $status == 1 ] ; then
    if [ worst != 2 ] ; then
      worst=1
    fi
    msg="${msg} ${dev}: degraded"
  elif [ $status == 2 ] ; then
    worst=2
    msg="${msg} ${dev}: degraded - unusable"
  fi
done

echo "mdadm:$msg"
exit $worst

which I saved as /usr/local/bin/check-mdadm.sh.

Add in a bit of snmpd.conf config (and set up sudo accordingly, of course):

...
exec   mdadm /usr/bin/sudo /usr/local/bin/check-mdadm.sh

and a small script on the nagios side (/usr/local/bin/nagios-check-mdadm):

#!/bin/sh

SNMP=`snmpwalk -v1 -c YOUR-PUBLIC $1 extOutput |grep mdadm`
TMP1=`echo $SNMP |grep degraded`
TMP2=`echo $SNMP |sed -e 's/^.*mdadm: //'`

if [ "$TMP1" = "" ]; then
  echo "OK: $TMP2"
  return 0
else
  echo "ERROR: $TMP2"
  return 2
fi

add a bit of nagios config:

define command {
       command_name check_mdadm
       command_line /usr/local/bin/nagios-check-mdadm $HOSTADDRESS$
}

define service {
       use      defaults
       name     check_mdadm
       description   MDADM
       check_command check_mdadm
}

And voila, nagios notifications when disks fall out of the array.

]]> http://ward.vandewege.net/blog/2009/11/nagios-mdadm-and-snmp/feed/ 1 a new home server http://ward.vandewege.net/blog/2009/09/a-new-home-server/ http://ward.vandewege.net/blog/2009/09/a-new-home-server/#comments Sun, 06 Sep 2009 03:01:59 +0000 ward http://ward.vandewege.net/blog/?p=480 I’ve been running an old Shuttle with a 2.4GHz celeron CPU, 512MB of ram and two 500GB disks in raid-1 as home server for the past 5 years or so. Well, I upgraded the disks in February 2008, before that it had 2x 200GB in raid-1. The thing has no UPS and runs in the closet here at home. And yet:

13:40:34 up 569 days, 17:04,  2 users,  load average: 1.26, 0.94, 0.45

Yeah, home power is pretty reliable around here.

This machine serves as the central network storage for our home, and I also use it to back up a bunch of servers that live at a nearby colo facility, with the rather fantastic BackupPC. The Shuttle has served well over the years but it is getting a bit old – I was starting to expect it to fail. Its power draw is rather high: 78W while idle (that’s after applying all of powertop’s suggestions), and a whopping 100W while doing heavy disk activity.

I was running out of disk space again, so I bought two 1TB ‘green’ WD drives (WD10EADS-00L) that are rated at 5.4W active, 2.8W idle, and 0.4W standby/sleep.

Next – a replacement for the Shuttle. First I looked at a QNAP TS-219p which is a rather awesome little NAS device. It’s based on Marvell’s Kirkwood ARM core, which is the same as the one used in the Sheevaplug, clocked at 1.2GHz. This thing is pretty fast. Its power specs are also impressive: 

Sleep mode: 5W
In operation: 21W (with 2 x 500GB HDD installed)

I was of course looking to run Debian on it, which is perfectly possible. People like the firmware that the thing comes with, but it’s proprietary so I’d rather not use that. Plus, I need to be able to run BackupPC.

The major downside is price – the TS-219P costs about $400, without disks. Since the Sheevaplug costs about $100, I would have thought a price in the $200-250 range for the TS-219P would have been reasonable.

Meanwhile I came across some really good NAS reviews over at SmallNetBuilder, and in particular their price/performance NAS chart.

Looking at that chart, the MSI Wind PC performance is pretty much on par with the TS-219P, for a fraction of the price. Extra bonus: it does not come with proprietary software preinstalled, because the Wind is really a bare-bones PC. The Wind has one 3.5″ bay, and one 5.15″ bay. It also has an on-board CF adapter. It has a dual-core Intel Atom 230 (1.6GHz).

I purchased 

$134.99    MSI Wind PC
 $26.99    G.SKILL 2GB 200-Pin DDR2 SO-DIMM DDR2 533
 $43.99    Transcend 16GB Compact Flash (CF) Flash Card Model TS16GCF133
  $9.99    StarTech BRACKET Metal 3.5" to 5.25" Drive Adapter Bracket

Total: $215.96 + shipping

The drive bay adaptor turned out to be not only severely overpriced, but also not practical for the Wind – I had to drill a few holes in the damn thing to make the second hard drive fit in the Wind. Don’t buy this kind, or don’t pay $10 for it!

I installed Debian on the CF card (leaving it read-only during normal operation) and use the two disks purely for data – in raid-1 of course. If I did this again I’d buy a smaller CF card – 8GB would be plenty, even 4GB would be enough for the non-volatile bits of /.

Power use, as tested: idle 27W, with heavy disk activity 33W. In other words, this will take 50-70W off our household power budget, which should work out to a savings of $7 to $10/month.

]]> http://ward.vandewege.net/blog/2009/09/a-new-home-server/feed/ 0 capistrano, svn and webserver timeouts http://ward.vandewege.net/blog/2009/07/capistrano-svn-and-webserver-timeouts/ http://ward.vandewege.net/blog/2009/07/capistrano-svn-and-webserver-timeouts/#comments Fri, 03 Jul 2009 15:54:34 +0000 ward http://ward.vandewege.net/blog/?p=475 A customer reported problems with capistrany deploys that would just die like this:

 ** [XXX.XXX.XXX :: err] svn: REPORT request failed on '/!svn/vcc/default'
 ** svn: REPORT of '/!svn/vcc/default': Chunk delimiter was invalid (http://XXX.XXX.XXX)
    command finished

After disabling gzip compression on the server for text/xml documents, the error became

 ** [XXX.XXX.XXX :: err] svn: REPORT request failed on '/!svn/vcc/default'
 ** svn: REPORT of '/!svn/vcc/default': Could not read response body: connection was closed by server (http://XXX.XXX.XXX)

The server side logs said:

[Fri Jul 03 10:53:57 2009] [error] [client XX.XX.XX.XX] Provider encountered an error while streaming a REPORT response.  [500, #0]
[Fri Jul 03 10:53:57 2009] [error] [client XX.XX.XX.XX] A failure occurred while driving the update report editor  [500, #190004]

Googling was not very helpful – there are many reports of these errors going back years, and many different solutions, none of which applied to my setup. In general, these errors seem to mean that there was some sort of network problem.

I tried to reproduce the problem by running the offending svn command manually. Out of hundreds of tries, I only managed to make it fail like that just once. And yet running cap deploy, which in turn calls the svn command, it would happen much more often.

I finally tracked this down to an agressive send/receive timeout in Apache’s config. It was set to 3 seconds to prevent too many inactive connections from taking up server resources. Apparently the subversion client sometimes takes a while to get back to the http server its talking to – in this particular situation when run via capistrano, more than 3 seconds. So the server would disconnect the svn client, which would then just fall over with that obscure error message.

In other words, check your server timeouts if you see this kind of intermittent error…

]]> http://ward.vandewege.net/blog/2009/07/capistrano-svn-and-webserver-timeouts/feed/ 0 df and zettabytes http://ward.vandewege.net/blog/2009/06/df-and-zettabytes/ http://ward.vandewege.net/blog/2009/06/df-and-zettabytes/#comments Tue, 23 Jun 2009 02:35:19 +0000 ward http://ward.vandewege.net/blog/?p=458 This is a very confused filesystem. But check it out – df supports zettabytes!

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0               19G  -64Z   22G 101% /

Surprisingly, the machine is up and runs just fine.

]]> http://ward.vandewege.net/blog/2009/06/df-and-zettabytes/feed/ 4 on the importance of gem cleanup http://ward.vandewege.net/blog/2009/06/on-the-importance-of-gem-cleanup/ http://ward.vandewege.net/blog/2009/06/on-the-importance-of-gem-cleanup/#comments Tue, 16 Jun 2009 16:35:01 +0000 ward http://ward.vandewege.net/blog/?p=448 I have a monit config that tries to stop/start mongrel instances like this:

  start program = "/usr/bin/mongrel_rails cluster::start -C path-to-mongrel_cluster.yml --clean --only PORT"
  stop program = "/usr/bin/mongrel_rails cluster::stop -C path-to-mongrel_cluster.yml --clean --only PORT"

I have the latest mongrel_cluster gem installed (1.0.5), and yet mongrel_rails kept throwing errors about –clean and –only:

invalid option: --clean for command 'cluster::start'
invalid option: --only for command 'cluster::start'

Turns out I had an older mongrel_cluster gem installed as well:

$ sudo gem cleanup mongrel_cluster
Cleaning up installed gems...
:0:Warning: Gem::SourceIndex#search support for Regexp patterns is deprecated
Attempting to uninstall mongrel_cluster-0.2.1
Successfully uninstalled mongrel_cluster-0.2.1
Clean Up Complete

After running gem cleanup, the mongrel_rails commands above started working.

This kind of code behaviour irks me – it’s not intuitive. It does not help that ‘gem list’ suggests that having multiple versions of a gem installed is not a problem – and it usually is not. I guess the mongrel_cluster gem is an exception. File this one under ‘good to know’…

]]> http://ward.vandewege.net/blog/2009/06/on-the-importance-of-gem-cleanup/feed/ 0 checking dns glue records http://ward.vandewege.net/blog/2009/06/checking-dns-glue-records/ http://ward.vandewege.net/blog/2009/06/checking-dns-glue-records/#comments Sun, 14 Jun 2009 19:54:33 +0000 ward http://ward.vandewege.net/blog/?p=446 This is mostly a reminder to myself. Here’s a way to check a domain’s DNS glue records. For example, for google:

 dig +trace +nosearch +all +norecurse google.com
]]> http://ward.vandewege.net/blog/2009/06/checking-dns-glue-records/feed/ 0 counting known_hosts http://ward.vandewege.net/blog/2009/05/counting-known_hosts/ http://ward.vandewege.net/blog/2009/05/counting-known_hosts/#comments Thu, 14 May 2009 00:46:40 +0000 ward http://ward.vandewege.net/blog/?p=432 All right folks, time to fess up. What’s your count?

$ wc ~/.ssh/known_hosts -l
1470

Obviously there are some old machines in there – this file goes back at least five years. There are some duplicates as well as there is one line per host/port/user combination. It’s annoying that I can’t weed out old entries easily, as ssh now encrypts the entries in known_hosts by default. Of course that is a very sensible thing to do from a security perspective, but it’s a shame from a statistics point of view.

]]> http://ward.vandewege.net/blog/2009/05/counting-known_hosts/feed/ 1 recycling a ton of old computing gear http://ward.vandewege.net/blog/2009/03/recycling-a-ton-of-old-computing-gear/ http://ward.vandewege.net/blog/2009/03/recycling-a-ton-of-old-computing-gear/#comments Sat, 14 Mar 2009 03:22:29 +0000 ward http://ward.vandewege.net/blog/?p=390 We recycled at work this week – literally over a metric ton (estimated) of old servers. This is what about two thirds of that amount looked liked:

old_computers

Apologies for the low quality picture – I only had a crappy cell phone camera on hand…

]]> http://ward.vandewege.net/blog/2009/03/recycling-a-ton-of-old-computing-gear/feed/ 2 ff3 sets downloaded files as readonly when using the ‘open with’ feature http://ward.vandewege.net/blog/2009/02/ff3-sets-downloaded-files-as-readonly-when-using-the-open-with-feature/ http://ward.vandewege.net/blog/2009/02/ff3-sets-downloaded-files-as-readonly-when-using-the-open-with-feature/#comments Wed, 25 Feb 2009 14:25:09 +0000 ward http://ward.vandewege.net/blog/?p=361 Firefox 3 has a new ‘feature’ where downloaded files are marked readonly when they are opened on the fly with the ‘open with’ feature.

The reasoning behind this is that there were many people who lost information because they didn’t realize Firefox deletes those temporary downloaded files when it closes. So Firefox 3 was patched to save the temporary files as readonly, thus forcing the applications to prompt the user to store the document somewhere permanent if any changes are attempted.

Of course this causes all sorts of unexpected breakage. I have an application where users download an Openoffice document that’s generated on the fly, and need to make some local edits. They now get the document in read-only, and need to hit ’save’ before they can start editing it. That’s annoying in their workflow.

The workaround is to create a new boolean attribute called 

browser.helperApps.deleteTempFileOnExit

in about:config, and set it to false. Obviously this will lead to information leakage, but if you are on a GNU/Linux system it won’t be too bad. The files will end up in /tmp/, which will be wiped on your next boot.

All in all I think this change in behavior makes sense, but it is annoying for people who know what they are doing. Maybe Openoffice could be modified to deal with readonly files in a smarter way when it is called from Firefox. In that case, perhaps Openoffice could move the file elsewhere and mark it read-write while opening the document, without user intervention. Seems like that would be a nice (optional?) feature. No local changes would be lost ever, and power users would not be inconvenienced.

]]> http://ward.vandewege.net/blog/2009/02/ff3-sets-downloaded-files-as-readonly-when-using-the-open-with-feature/feed/ 1 what’s going on with the .org nameservers? http://ward.vandewege.net/blog/2009/01/whats-going-on-with-the-org-nameservers/ http://ward.vandewege.net/blog/2009/01/whats-going-on-with-the-org-nameservers/#comments Wed, 21 Jan 2009 02:00:40 +0000 ward http://ward.vandewege.net/blog/?p=321 I got rather strange error messages from two servers this evening, one at 18:00:21 EST, and another one at 18:34:01 EST. One server lives in Brussels, the other in Boston. On both servers a (totally different) script executed via cron complained it could not resolve a hostname. Both hostnames exist, and both are in .org domains (in different domains).

Both machines run dnscache locally for resolving. Sadly, the logs had already rotated on the first server, but the other one still had the relevant entry. This is what I found:

2009-01-20 18:34:01.729172500 query #1221279 127.0.0.1:45933 (id 50134) a REDACTED.iofc.org.
2009-01-20 18:34:01.729221500 cached ns org. b2.org.afilias-nst.org.
2009-01-20 18:34:01.729223500 cached ns org. a2.org.afilias-nst.info.
2009-01-20 18:34:01.729225500 cached ns org. b0.org.afilias-nst.org.
2009-01-20 18:34:01.729277500 cached ns org. d0.org.afilias-nst.org.
2009-01-20 18:34:01.729279500 cached ns org. c0.org.afilias-nst.info.
2009-01-20 18:34:01.731611500 cached ns org. a0.org.afilias-nst.info.
2009-01-20 18:34:01.731654500 cached a b2.org.afilias-nst.org.
2009-01-20 18:34:01.731657500 cached a a2.org.afilias-nst.info.
2009-01-20 18:34:01.731674500 cached a b0.org.afilias-nst.org.
2009-01-20 18:34:01.731676500 cached a d0.org.afilias-nst.org.
2009-01-20 18:34:01.731678500 cached a c0.org.afilias-nst.info.
2009-01-20 18:34:01.731694500 cached a a0.org.afilias-nst.info.
2009-01-20 18:34:01.731696500 tx g=0 a REDACTED.iofc.org. org. 199.19.53.1 199.249.112.1 199.249.120.1 199.19.57.1 199.19.54.1 199.19.56.1
2009-01-20 18:34:01.814058500 nxdomain 199.19.53.1 TTL=0 REDACTED.iofc.org.

What happens here is that the Afilias (the organization that runs the .org registry) nameserver at 199.19.53.1 returned NXDOMAIN (no such domain) when asked about the host REDACTED.iofc.org, rather than returning the nameservers that are authoritative for the iofc.org domain.

A bit later, things were fine again when another Afilias nameserver replied properly:

2009-01-20 19:04:01.715579500 query #1221422 127.0.0.1:45936 (id 46625) a prometheus.iofc.org.
2009-01-20 19:04:01.715627500 cached ns org. b2.org.afilias-nst.org.
2009-01-20 19:04:01.715629500 cached ns org. a2.org.afilias-nst.info.
2009-01-20 19:04:01.715631500 cached ns org. b0.org.afilias-nst.org.
2009-01-20 19:04:01.715634500 cached ns org. d0.org.afilias-nst.org.
2009-01-20 19:04:01.715635500 cached ns org. c0.org.afilias-nst.info.
2009-01-20 19:04:01.715637500 cached ns org. a0.org.afilias-nst.info.
2009-01-20 19:04:01.715639500 cached a b2.org.afilias-nst.org.
2009-01-20 19:04:01.715641500 cached a a2.org.afilias-nst.info.
2009-01-20 19:04:01.715660500 cached a b0.org.afilias-nst.org.
2009-01-20 19:04:01.715662500 cached a d0.org.afilias-nst.org.
2009-01-20 19:04:01.715664500 cached a c0.org.afilias-nst.info.
2009-01-20 19:04:01.715666500 cached a a0.org.afilias-nst.info.
2009-01-20 19:04:01.715683500 tx g=0 a REDACTED.iofc.org. org. 199.19.54.1 199.249.112.1 199.19.56.1 199.19.53.1 199.19.57.1 199.249.120.1
2009-01-20 19:04:01.754217500 rr 199.19.54.1 TTL=172800 a ns1.iofc.org. 62.49.196.44
2009-01-20 19:04:01.754224500 rr 199.19.54.1 TTL=86400 a ns4.iofc.org. 208.94.48.44
2009-01-20 19:04:01.754266500 rr 199.19.54.1 TTL=86400 ns iofc.org. ns1.iofc.org.
2009-01-20 19:04:01.754269500 rr 199.19.54.1 TTL=86400 ns iofc.org. ns4.iofc.org.
2009-01-20 19:04:01.754271500 stats count=1221422 motion=98951375 udp-active=1 tcp-active=0
2009-01-20 19:04:01.754273500 cached a ns1.iofc.org.
2009-01-20 19:04:01.754274500 cached a ns4.iofc.org.
...

This kind of thing is not supposed to happen. What’s going on, Afilias?

Update: This sans.org report might be related. And there were a few threads on the nanog mailing list about a DDOS amplification attack using DNS servers.

]]> http://ward.vandewege.net/blog/2009/01/whats-going-on-with-the-org-nameservers/feed/ 1