I found a migration script at norwinter.com, which I improved a bit. It will handle wiki pages with full history as well as attachments. It won’t preserve who committed revisions, however – that is hardcoded in the script. So, this is still a hack.

Usage instructions:

a) copy your MoinMoinWiki data/pages directory to the server that runs your redmine install
b) put the migrate_from_moinmoin.rake script in lib/tasks/ in your Redmine install
c) edit the migrate_from_moinmoin.rake script, replace both instances of YOUR@EMAIL.ADDRESS
d) run rake redmine:migrate_from_moinmoin RAILS_ENV=”production”
e) provide a unique redmine project id and the path to your MoinMoinWiki data/pages directory

And here is my version of migrate_from_moinmoin.rake.

This script worked well enough for me to import MoinMoinWiki version 1.5.7 to Redmine 1.2.0.

The machine booted into Grub’s rescue mode. Grub 2′s rescue mode is nice, but not exactly intuitive (no ‘help’ or ‘?’ command), and documentation for it is apparently not easily found via Google.

This is what was eluding me for a while:

In rescue mode, only the insmod, ls, set, and unset commands are
normally available. If you end up in rescue mode and do
not know what to do, then see "GRUB only offers a rescue shell."

That’s from the Grub manual at gnu.org.

The steps involved in recovery are documented in the same manual. Basically you need to set the prefix and root variables, and then insmod the ‘normal’ module and run it.

The packaging was not too great; while all disks were well packaged individually, the big boxes that contained the individual drive boxes were flimsy. As a consequence, one of the disks got rather damaged (the one on the right):

I don’t know what it got hit with, but it must have been a pretty serious blow. The aluminium enclosure of the drive is severely dented and even cracked; the white line in the image below is an actual crack in the metal:

Back in October 2009 I added 130 TB of disk to another cluster, which looked like this, prior to install:

That was 65 times WD2002FYPS.

So this time around – almost 18 months later – we get 27% more capacity using 15% fewer drives. Got to love the computer industry and the progress it makes.

The ‘Average CPU Power’ is based on ‘average’ use, which is explained on Wikipedia.

According to Wikipedia, the Thermal Design Power is the maximum power consumption for thermally significant periods running worst-case non-synthetic workloads (cf. this article). If we assume that the bulk of the electrical power consumed by a CPU is converted into waste heat, then the TDP can be a reasonable approximation for the amount of electrical power a CPU would consume under a worst-case, real-world load.

I used cpuburn to generate such a load. There was no IO load on the system during the tests. I measured power draw with an off-the-shelf Kill-a-watt, so these results should be taken with a grain of salt.

The data indicates that the idle vs. full load power consumption difference for one CPU is 70 to 75W.

If we assume the power consumption under full load is 115W (the TDP for the processor), then idle power consumption would be 40 to 45W per CPU. That would put idle power consumption at 35-39% of its TDP for this particular CPU.

-rw-r--r-- 1 root  root  1.1G 2010-10-31 20:19 10125-127-2010-10.error

After:

-rw-r--r-- 1 root  root   11M 2010-10-31 20:19 10125-127-2010-10.error.bz2

Bzip2 reduced the file to 1% of its original size. Not bad!

smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%        66         -
# 2  Extended offline    Completed without error       00%     65434         -
# 3  Extended offline    Completed without error       00%     65266         -

Interestingly, the Power_On_Hours field did not wrap. Bug in smartctl? Bug in the drive firmware?

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   095   095   060    Pre-fail  Always       -       458761
  2 Throughput_Performance  0x0005   100   100   050    Pre-fail  Offline      -       210
  3 Spin_Up_Time            0x0007   105   105   024    Pre-fail  Always       -       160 (Average 154)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       27
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   111   111   020    Pre-fail  Offline      -       43
  9 Power_On_Hours          0x0012   091   091   000    Old_age   Always       -       65616
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       27
192 Power-Off_Retract_Count 0x0032   099   099   050    Old_age   Always       -       1387
193 Load_Cycle_Count        0x0012   099   099   050    Old_age   Always       -       1387
194 Temperature_Celsius     0x0002   189   189   000    Old_age   Always       -       29 (Lifetime Min/Max 21/39)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

#!/bin/sh
# (c) 2008 Jasper Spaans 

worst=0
msg=""

for dev in /dev/md?* ; do \
  mdadm --misc -t --detail $dev >/dev/null
  status=$?
  if [ $status == 0 ]; then
    msg="${msg} ${dev}: ok"
  elif [ $status == 1 ] ; then
    if [ worst != 2 ] ; then
      worst=1
    fi
    msg="${msg} ${dev}: degraded"
  elif [ $status == 2 ] ; then
    worst=2
    msg="${msg} ${dev}: degraded - unusable"
  fi
done

echo "mdadm:$msg"
exit $worst

which I saved as /usr/local/bin/check-mdadm.sh.

Add in a bit of snmpd.conf config (and set up sudo accordingly, of course):

...
exec   mdadm /usr/bin/sudo /usr/local/bin/check-mdadm.sh

and a small script on the nagios side (/usr/local/bin/nagios-check-mdadm):

#!/bin/sh

SNMP=`snmpwalk -v1 -c YOUR-PUBLIC $1 extOutput |grep mdadm`
TMP1=`echo $SNMP |grep degraded`
TMP2=`echo $SNMP |sed -e 's/^.*mdadm: //'`

if [ "$TMP1" = "" ]; then
  echo "OK: $TMP2"
  return 0
else
  echo "ERROR: $TMP2"
  return 2
fi

add a bit of nagios config:

define command {
       command_name check_mdadm
       command_line /usr/local/bin/nagios-check-mdadm $HOSTADDRESS$
}

define service {
       use      defaults
       name     check_mdadm
       description   MDADM
       check_command check_mdadm
}

And voila, nagios notifications when disks fall out of the array.

13:40:34 up 569 days, 17:04,  2 users,  load average: 1.26, 0.94, 0.45

Yeah, home power is pretty reliable around here.

This machine serves as the central network storage for our home, and I also use it to back up a bunch of servers that live at a nearby colo facility, with the rather fantastic BackupPC. The Shuttle has served well over the years but it is getting a bit old – I was starting to expect it to fail. Its power draw is rather high: 78W while idle (that’s after applying all of powertop’s suggestions), and a whopping 100W while doing heavy disk activity.

I was running out of disk space again, so I bought two 1TB ‘green’ WD drives (WD10EADS-00L) that are rated at 5.4W active, 2.8W idle, and 0.4W standby/sleep.

Next – a replacement for the Shuttle. First I looked at a QNAP TS-219p which is a rather awesome little NAS device. It’s based on Marvell’s Kirkwood ARM core, which is the same as the one used in the Sheevaplug, clocked at 1.2GHz. This thing is pretty fast. Its power specs are also impressive:

Sleep mode: 5W
In operation: 21W (with 2 x 500GB HDD installed)

I was of course looking to run Debian on it, which is perfectly possible. People like the firmware that the thing comes with, but it’s proprietary so I’d rather not use that. Plus, I need to be able to run BackupPC.

The major downside is price – the TS-219P costs about $400, without disks. Since the Sheevaplug costs about $100, I would have thought a price in the $200-250 range for the TS-219P would have been reasonable.

Meanwhile I came across some really good NAS reviews over at SmallNetBuilder, and in particular their price/performance NAS chart.

Looking at that chart, the MSI Wind PC performance is pretty much on par with the TS-219P, for a fraction of the price. Extra bonus: it does not come with proprietary software preinstalled, because the Wind is really a bare-bones PC. The Wind has one 3.5″ bay, and one 5.15″ bay. It also has an on-board CF adapter. It has a dual-core Intel Atom 230 (1.6GHz).

I purchased

$134.99    MSI Wind PC
 $26.99    G.SKILL 2GB 200-Pin DDR2 SO-DIMM DDR2 533
 $43.99    Transcend 16GB Compact Flash (CF) Flash Card Model TS16GCF133
  $9.99    StarTech BRACKET Metal 3.5" to 5.25" Drive Adapter Bracket

Total: $215.96 + shipping

The drive bay adaptor turned out to be not only severely overpriced, but also not practical for the Wind – I had to drill a few holes in the damn thing to make the second hard drive fit in the Wind. Don’t buy this kind, or don’t pay $10 for it!

I installed Debian on the CF card (leaving it read-only during normal operation) and use the two disks purely for data – in raid-1 of course. If I did this again I’d buy a smaller CF card – 8GB would be plenty, even 4GB would be enough for the non-volatile bits of /.

Power use, as tested: idle 27W, with heavy disk activity 33W. In other words, this will take 50-70W off our household power budget, which should work out to a savings of $7 to $10/month.

 ** [XXX.XXX.XXX :: err] svn: REPORT request failed on '/!svn/vcc/default'
 ** svn: REPORT of '/!svn/vcc/default': Chunk delimiter was invalid (http://XXX.XXX.XXX)
    command finished

After disabling gzip compression on the server for text/xml documents, the error became

 ** [XXX.XXX.XXX :: err] svn: REPORT request failed on '/!svn/vcc/default'
 ** svn: REPORT of '/!svn/vcc/default': Could not read response body: connection was closed by server (http://XXX.XXX.XXX)

The server side logs said:

[Fri Jul 03 10:53:57 2009] [error] [client XX.XX.XX.XX] Provider encountered an error while streaming a REPORT response.  [500, #0]
[Fri Jul 03 10:53:57 2009] [error] [client XX.XX.XX.XX] A failure occurred while driving the update report editor  [500, #190004]

Googling was not very helpful – there are many reports of these errors going back years, and many different solutions, none of which applied to my setup. In general, these errors seem to mean that there was some sort of network problem.

I tried to reproduce the problem by running the offending svn command manually. Out of hundreds of tries, I only managed to make it fail like that just once. And yet running cap deploy, which in turn calls the svn command, it would happen much more often.

I finally tracked this down to an agressive send/receive timeout in Apache’s config. It was set to 3 seconds to prevent too many inactive connections from taking up server resources. Apparently the subversion client sometimes takes a while to get back to the http server its talking to – in this particular situation when run via capistrano, more than 3 seconds. So the server would disconnect the svn client, which would then just fall over with that obscure error message.

In other words, check your server timeouts if you see this kind of intermittent error…

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md0               19G  -64Z   22G 101% /

Surprisingly, the machine is up and runs just fine.