* October 2004 – purchase Shuttle home server with 2x 200GB disk (200GB useable)
* May 2007 – upgrade to 2x 500GB disk (500GB useable)
* September 2009 – upgrade to MSI Wind with 2x 1TB disk (1TB useable)
* June 2012 – upgrade to 2x 2TB disk (2TB useable)

Looks like my upgrades are roughly 2.5 years apart. Interesting!

I bought one Hitachi HDS723020BLA642 (that’s model ’0f12115′, a 2TB SATA3 drive with 64MB of cache). Well, actually I ordered model ’0S02861′ which is SATA2 and only has 32MB of cache, but for some reason Amazon shipped me the faster one. Ah, well, I’m not complaining. The other drive is a Western Digital WD20EARX (that’s also 2TB SATA3 drive with 64MB of cache).

It’s worth noting that the two 1TB drives that are being replaced (WD10EADS-00L) have been rock solid over the past 2.75 years, despite a *lot* of data being written to them (nightly backups of more and more servers, which is also why I was running out of space…).

So now I have two SATA3 drives in a system that is SATA2 only. That’s too bad. On the other hand, this machine now has 10x the amount of disk space that its first incarnation had, back in 2004. Nice!

The packaging was not too great; while all disks were well packaged individually, the big boxes that contained the individual drive boxes were flimsy. As a consequence, one of the disks got rather damaged (the one on the right):

I don’t know what it got hit with, but it must have been a pretty serious blow. The aluminium enclosure of the drive is severely dented and even cracked; the white line in the image below is an actual crack in the metal:

Back in October 2009 I added 130 TB of disk to another cluster, which looked like this, prior to install:

That was 65 times WD2002FYPS.

So this time around – almost 18 months later – we get 27% more capacity using 15% fewer drives. Got to love the computer industry and the progress it makes.

So… how would this work? Presumably the computer would run some code that is not under the control of the user/owner of the machine, and protected by the TPM module. That code would then validate if the machine is free of malware or not – somehow. I have no idea how that could possibly be foolproof, but let’s assume for a moment there is a way to do this.

First problem: your computer would have to run code that most likely comes without source, is hard or impossible to inspect, and cannot be changed.

Let’s say for the sake of argument that this validation code is somehow optional. Or perhaps you are an enterprising person, and you’ve managed to kick this stuff off your computer (TPM-ectomy, anyone?). Next problem: now you can’t validate your computer with your ISP to prove that it is free of malware. To do that, you need access to the secret encryption key buried in the TPM.

This is called remote attestation: the machine(s) your computer communicates with can see information about your computer – say, what operating system you run, and what patch level – and because that data is signed or encrypted by your TPM chip, you can not change it.

Note that it’s already pretty easy for remote machines to see what (version of) an operating system a computer runs, for instance with TCP/IP fingerprinting, but that is easy to fake.

Remote attestation is the real danger of ‘trustworthy’ computing. They can try to put all sorts of things in the hardware; if people have physical access, someone will find a way around it. But if they make it impossible to network your computer without an operational TPM chip then we might as well kiss all our free software and free hardware goodbye. It won’t be any good to run a computer with GNU/Linux, if we can’t go online with it… Or if our online banking refuses to talk to our computer because our machine is not deemed to be running a fully patched version of Windows.

Given that this position paper comes from Microsoft, it’s not too hard to see where they want to go. Microsoft would love to be in a position where ISPs and banks require certain patchlevels of its software. Can you imagine a better way to force people to keep upgrading their Windows licenses? Or to force people to stop using free operating systems?

I have a better idea to combat the malware problem, mr. Charney. Why don’t we ask people to stop using Windows. Without Windows, the malware/botnet problem would not be nearly as bad as it is today.

I’m running out of CPU – quick, head over to the corner store for an Intel scratch card!

Is this an alternate universe? And, how long before they sell cards that will unlock extra features, but only for a limited time?

I guess now we know at least one concrete reason why Intel does not like coreboot. You can’t restrict people like this when their computer does not have a proprietary BIOS.

13:40:34 up 569 days, 17:04,  2 users,  load average: 1.26, 0.94, 0.45

Yeah, home power is pretty reliable around here.

This machine serves as the central network storage for our home, and I also use it to back up a bunch of servers that live at a nearby colo facility, with the rather fantastic BackupPC. The Shuttle has served well over the years but it is getting a bit old – I was starting to expect it to fail. Its power draw is rather high: 78W while idle (that’s after applying all of powertop’s suggestions), and a whopping 100W while doing heavy disk activity.

I was running out of disk space again, so I bought two 1TB ‘green’ WD drives (WD10EADS-00L) that are rated at 5.4W active, 2.8W idle, and 0.4W standby/sleep.

Next – a replacement for the Shuttle. First I looked at a QNAP TS-219p which is a rather awesome little NAS device. It’s based on Marvell’s Kirkwood ARM core, which is the same as the one used in the Sheevaplug, clocked at 1.2GHz. This thing is pretty fast. Its power specs are also impressive:

Sleep mode: 5W
In operation: 21W (with 2 x 500GB HDD installed)

I was of course looking to run Debian on it, which is perfectly possible. People like the firmware that the thing comes with, but it’s proprietary so I’d rather not use that. Plus, I need to be able to run BackupPC.

The major downside is price – the TS-219P costs about $400, without disks. Since the Sheevaplug costs about $100, I would have thought a price in the $200-250 range for the TS-219P would have been reasonable.

Meanwhile I came across some really good NAS reviews over at SmallNetBuilder, and in particular their price/performance NAS chart.

Looking at that chart, the MSI Wind PC performance is pretty much on par with the TS-219P, for a fraction of the price. Extra bonus: it does not come with proprietary software preinstalled, because the Wind is really a bare-bones PC. The Wind has one 3.5″ bay, and one 5.15″ bay. It also has an on-board CF adapter. It has a dual-core Intel Atom 230 (1.6GHz).

I purchased

$134.99    MSI Wind PC
 $26.99    G.SKILL 2GB 200-Pin DDR2 SO-DIMM DDR2 533
 $43.99    Transcend 16GB Compact Flash (CF) Flash Card Model TS16GCF133
  $9.99    StarTech BRACKET Metal 3.5" to 5.25" Drive Adapter Bracket

Total: $215.96 + shipping

The drive bay adaptor turned out to be not only severely overpriced, but also not practical for the Wind – I had to drill a few holes in the damn thing to make the second hard drive fit in the Wind. Don’t buy this kind, or don’t pay $10 for it!

I installed Debian on the CF card (leaving it read-only during normal operation) and use the two disks purely for data – in raid-1 of course. If I did this again I’d buy a smaller CF card – 8GB would be plenty, even 4GB would be enough for the non-volatile bits of /.

Power use, as tested: idle 27W, with heavy disk activity 33W. In other words, this will take 50-70W off our household power budget, which should work out to a savings of $7 to $10/month.

The difference in performance between the old mechanical drive and this SSD is *spectacular*. The machine feels 100 times faster. In other words, you want one of these for your desktop/laptop

As for servers – I can’t wait until these things become big enough/cheap enough to replace mechanical drives there, too…

Here are some bonnie++ results:

Hitachi 7K200-160:

$ bonnie++ 
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
countzero        4G 14755  55 15887  20 13568  13 18775  58 31430  11  72.4   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 21341  83 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
countzero,4G,14755,55,15887,20,13568,13,18775,58,31430,11,72.4,0,16,21341,83,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Intel X25-M gen 1:

$ bonnie++
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
countzero        4G 51217  98 76395  31 42853  25 43666  80 150391  45 14762  61
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
countzero,4G,51217,98,76395,31,42853,25,43666,80,150391,45,14762.2,61,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

Apologies for the low quality picture – I only had a crappy cell phone camera on hand…

The method basically consists of cutting off the legs of the PLCC chip as close to the package as possible, so as to minimize strain on the paths on the motherboard while cutting. After that, the legs can easily be de-soldered from the board, and it’s trivial to solder on a PLCC socket. Obviously you loose the PLCC chip in the process, but this method is a lot easier than removing the PLCC chip with a heat gun. It’s also safer for the surrounding components.

Here are some photos:

The chip prior to modification.

Legs cut off on two sides of the chip. It’s important to use the right cutter for this job – it needs to be able to cut very closely to the package of the chip. I used a Hakko CHP Ergonomic Micro Cutter, part number TR20M.

PLCC chip removed.

PLCC chip without legs.

The legs have been de-soldered. If you look carefully near pins 1 and 2 on the top row, you’ll see I was foolish and used too much force and a too small soldering tip while de-soldering. I damaged the board. Luckily the pins involved are all on the same big pad, and everything still works.

This is the socket prior to soldering. I had to remove the plastic ‘floor’ of the socket in the center to allow easy access to the pins for soldering. You can put that piece of plastic back in the socket after it has been soldered onto the board, even loose. It will prevent PLCC chips from being pushed too deep into the socket, where not all pins might make contact with the socket.

The PLCC socket, installed.

http://support.euro.dell.com/support/downloads/download.aspx?c=uk&l=en&s=gen&releaseid=R155882&formatcnt=2&libid=0&fileid=208066

Man – what a pile of crap. This thing is a self-extracting archive for rpm-based systems (redhat/suse). Fine – except that I run Debian on that machine. Anyway, you can run it with –extract to extract the contents:

./delldset_v1.4.0.8.bin --help


Command-line options for Dell System E-Support Tool (DSET)

Usage:  [options...]

Options:

-h,--help        : Display command-line usage help
-i,--install     : Install/upgrade the DSET application
-v,--version     : Display version information
--list           : Display contents of package (+)
--extract  : Extract files to specified path (+)

Using no options will display the interactive user
interface which will guide you through using DSET.

These contents are basically an rpm file and some support files:

-r-xr-xr-x  1 ward ward     4133 Apr  2  2007 Dell_License*
-r-xr-xr-x  1 ward ward     9909 May  8  2007 README*
-r-xr-xr-x  1 ward ward       58 May  8  2007 Version.txt*
-r-xr-xr-x  1 ward ward 15200341 May  9  2007 delldset-1.4.0-8.i386.rpm*
-rw-r--r--  1 ward ward 15187240 Oct 27 15:57 delldset_1.4.0-9_i386.deb
-r-xr-xr-x  1 ward ward    10271 Apr  2  2007 install.sh*
-r-xr-xr-x  1 ward ward      472 Apr  2  2007 sphelp.txt*
-r-xr-xr-x  1 ward ward       24 May  4  2005 test.sh*
-r-xr-xr-x  1 ward ward     9877 Apr  2  2007 utility.sh*

The rpm can be transformed into a .deb with alien:

$ alien delldset-1.4.0-8.i386.rpm 
Warning: Skipping conversion of scripts in package delldset: postinst postrm preinst prerm
Warning: Use the --scripts parameter to include the scripts.
delldset_1.4.0-9_i386.deb generated
$ alien delldset-1.4.0-8.i386.rpm --scripts
delldset_1.4.0-9_i386.deb generated

Which installs fine – but you really want to be using a kernel that has ipmi functionality compiled as modules; otherwise the package will not install properly until you add

hapi.allow.user.mode=yes

to /etc/omreg.cfg. In any case, you need to have ipmi as modules for the rest of this sequence of commands, so you might as well boot into a stock Debian kernel.

You’ll probably want to have installed Dell’s open manage before you run the DSET report. To do that, add

  deb ftp://ftp.sara.nl/pub/sara-omsa dell sara

to your /etc/apt/sources.list file and

  apt-get update
  apt-get install dellomsa

So, then you would be ready to run the dellsysteminfo command but WAIT – don’t do it yet. That script is dangerous. It does stuff like this, probing for the running services (from /opt/dell/dset/bin/dell-sysreport.sh):

#Dell: Extra OS stuff we want

echo "     Getting status of services ..." ${TEE2LOG}

#Dell: SuSE doesn't have a "service status" method so do it the manual way 
#catifexec "/sbin/service" "--status-all" 
 
rm -f $ROOT/service > /dev/null 2>&1 
 
SERVICEDIR=/etc/init.d 
 
#pushd ${SERVICEDIR} > /dev/null 
          for SERVICE in `ls -1 ${SERVICEDIR}` ; do 
 
            case "${SERVICE}" in 
#Skip over these "services" since they tend to hang some systems when requested for status 
              functions | halt | killall | single| linuxconf| kudzu | bgpd | boot | reboot | single | nscd | \ 
                  halt.* | rc | boot.* | *rpmorig | *rpmnew | *rpmsave | *~ | *.orig) 
      ;; 
              *) 
                if [ -x "${SERVICEDIR}/${SERVICE}" -a -f "${SERVICEDIR}/${SERVICE}" ]; then 
        echo "Service Name: ${SERVICE}" >> $ROOT/service 
                 /usr/bin/env -i LANG=$LANG PATH=$PATH TERM=$TERM "${SERVICEDIR}/${SERVICE}" status >> $ROOT/service 2>&1 
    echo >> $ROOT/service 
    echo "---------------------------------------------------------------------------" >> $ROOT/service 
               fi 
                ;; 
            esac 
          done 
#popd > /dev/null

That didn’t go over too well on my system – a bunch of things stopped working (like the ‘w’ command), samba started segfaulting, etc. It’s clearly calling some init scripts that don’t deal well with being asked for a status.

So I commented out that part of the script, which made things a lot happier – but of course services are now not mentioned in the report.

The other problem with the dset package is that it is riddled with calls to awk, sort, etc with absolute paths. Seriously, this is not 1995. You’ll need to fix tons of calls – just grep for ‘/bin/awk’, ‘/bin/grep’, ‘/bin/sort’ etc in /opt/dell/dset.

When all that’s done, you can generate the report

  dellsysteminfo

which will drop a zip file with a horribly long name (including the use of round brackets, sigh) in /root/ with the password ‘dell’. Awesome security guys. Why is there even a password if it’s so trivial?

The Dell tech I’m e-mailing with (who is very helpful btw, this post is not a criticism about Dell’s support people – only about their GNU/Linux tools) said I’d need to upgrade the system bios and the ‘ESM’, which is the ‘system firmware’. The former was easy with the help of this blog post. In a nutshell:

aptitude install libsmbios-bin libsmbios1 libsmbiosxml1
getSystemId

That will print out your ‘System ID’. Look up the latest system bios for that System ID at

http://linux.dell.com/repo/software/bios-hdrs/

and download it. Then

modprobe dell_rbu
dellBiosUpdate -u -f bios.hdr

and reboot, at which point your bios will be upgraded if all goes well. Note that if your cmos battery is dead, you can’t upgrade your bios anymore because all the command line tool does is load it into ram and flip a switch to tell the bios to upgrade itself on the next boot; without cmos battery either that switch or the bios in ram may not survive a reboot.

So that worked fine. The ESM update was a little trickier. I downloaded another .BIN file from

http://support.euro.dell.com/support/downloads/download.aspx?c=fr&l=fr&s=gen&releaseid=R147948&SystemID=PWE_PNT_2800&servicetag=8YGYQ1J&os=WNET&osl=fr&deviceid=5814&devlib=0&typecnt=0&vercnt=7&catid=-1&impid=-1&formatcnt=4&libid=29&fileid=196749

At least that thing didn’t try to do all sorts of rpm manipulations, it just runs a few scripts (again, you can verify with –extract).

The tricky thing here is that running the BIN file conflicts with the ipmi modules. Here’s what I did:

Boot into single user mode, and stop openmanage and ipmievd:

/etc/init.d/instsvcdrv stop
/etc/init.d/ipmievd stop

Then unload all ipmi modules:

rmmod ipmi_devintf
rmmod ipmi_si 
rmmod ipmi_msghandler

Then start screen, and start the .BIN file. It will look around, cause the ipmi modules to be loaded again and if all is well ask you if you really want to update your ESM, [Y/N].

At this point, break out of screen with ctrl-a d, and unload the ipmi modules again (I kid you not!):

rmmod ipmi_devintf
rmmod ipmi_si 
rmmod ipmi_msghandler

Now return to screen

screen -r

and press ‘Y’. Don’t worry, it’s going to load the ipmi modules yet again, and then starts printing dots on the screen as the upgrade progresses. It takes a while – at least 5 minutes on my system. So just wait until it is done.

Eventually the script said ‘update complete’. I read this trick with screen and the unloading of the ipmi modules somewhere on a dell wiki, but I can’t find the link now (sorry!) so I can’t give credit.

Anyway, and I was able to verify the upgrade with ipmitool (I upgraded to firmware revision 1.72):

# ipmitool -I open mc info
Device ID                 : 32
Device Revision           : 0
Firmware Revision         : 1.72
IPMI Version              : 1.5
Manufacturer ID           : 674
Manufacturer Name         : Unknown (0x2a2)
Product ID                : 0 (0x0000)
Device Available          : yes
Provides Device SDRs      : yes
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    Chassis Device
Aux Firmware Rev Info     :
    0x00
    0x00
    0x00
    0x00

So in the end I got almost everything to work (except for the dset output which is still partially missing). But seriously, this was way harder than it should be.

That DSET package is a total mess and needs some serious reworking.

The system bios upgrade procedure is elegant and simple – there Dell deserves credit.

The ESM upgrade… not so much. Seriously – what’s up with those ipmi modules getting in the way twice?

In a nutshell, I’d like Dell to hire some people to clean up the GNU/Linux tools they provide. But they have to be people who know about more than just rpm-based distros, and who are interested in truly cross-distribution solutions and want to work with all (popular) distributions. Basically, if Dell can get their tools into Debian, Fedora and OpenSuse, things will trickle down to almost every other popular flavor of GNU/Linux. There is a lot of work to do…