-rwxr-xr-x 1 ward ward 60097588 Jan 31 06:18 prometheus
-rwxr-xr-x 1 ward ward 38359349 Jan 31 06:19 promtool

Fillippo Valsorda has a post that describes shrinking golang binaries.

Prometheus uses the promu tool for building its binaries. Adding the necessary ld flags is done by editing .promu.yml file like this:

--- .promu.yml.orig	2019-02-20 08:57:29.413554323 -0500
+++ .promu.yml	2019-02-19 17:44:33.451539440 -0500
@@ -17,6 +17,8 @@
         -X github.com/prometheus/common/version.Branch={{.Branch}}
         -X github.com/prometheus/common/version.BuildUser={{user}}@{{host}}
         -X github.com/prometheus/common/version.BuildDate={{date "20060102-15:04:05"}}
+        -s
+        -w
 tarball:
     files:
         - consoles

After building the prometheus binaries with the latest stable go (1.11.5) using make build, we end up with:

-rwxr-xr-x  1 ward ward 46379136 Feb 19 17:45 prometheus
-rwxr-xr-x  1 ward ward 29391136 Feb 19 17:46 promtool

The binaries are now reduced to about 77% of their original size. Better, but still not great.

We can confirm they are now indeed statically linked and stripped of debug info:

$ file prometheus
prometheus: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped
$ file promtool
promtool: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped

Applying upx takes a while but yields excellent results:

$ upx --brute prometheus 
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  46379136 ->   9370100   20.20%  linux/ElfAMD   prometheus                    

Packed 1 file.

$ upx --brute promtool
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  29391136 ->   6271396   21.34%  linux/ElfAMD   promtool                      

Packed 1 file.

And here are the resulting binaries:

-rwxr-xr-x  1 ward ward 9370100 Feb 19 17:45 prometheus
-rwxr-xr-x  1 ward ward 6271396 Feb 19 17:46 promtool

The binaries are now down to roughly 16% of their original size. There is a noticeable startup delay at runtime for the upx decompression, but for a long-running daemon like prometheus that doesn’t matter at all.

For promtool, which is used more interactively, you might find the startup delay annoying. This is the original binary:

$ time ./promtool.orig 
usage: promtool.orig []  [ ...]

Tooling for the Prometheus monitoring system.
...

real	0m0.055s
user	0m0.036s
sys	0m0.013s

And here’s the UPX-compressed one:

$ time ./promtool
usage: promtool []  [ ...]

Tooling for the Prometheus monitoring system.
...

real	0m0.536s
user	0m0.521s
sys	0m0.016s

I’m upgrading my laptop to SSD number four (an Intel 530, 480GB) and I decided it was time to set up TRIM properly. This is a fairly straightforward configuration with LUKS handling encryption for an LVM physical volume, inside of which the logical volumes live that I need to be able to run TRIM on.

Caveat emptor: because TRIM essentially makes it clear which blocks on the drive are in use and which are not, TRIM is disabled by default at the LUKS layer. Enabling TRIM will leak a bit of information (how many bytes are used by your encrypted partition), and should not be done in a ‘plausible deniability’ scenario.

There’s a lot of information out there on this subject – the best docs appear to be for Arch – but it seems that Debian does things slightly differently from most other distributions. Here’s what I had to do to make TRIM work.

1. In /etc/default/grub, add to the GRUB_CMDLINE_LINUX_DEFAULT and GRUB_CMDLINE_LINUX lines:

  cryptdevice=/dev/sdaX:vg_name:discard

Replace /dev/sdaX with the path to the partition that contains your LUKS volume. Replace vg_name with the name of your LVM volume group.

2. Run update-grub:

  # update-grub

3. In /etc/lvm/lvm.conf, set

  issue_discards = 1

in the devices section.

4. update /etc/crypttab: add ‘,discard’ to the fourth field for each encrypted partition:

sdaX_crypt UUID=abcdefgh-1234-5678-90ab-cdefghijklmn none luks,discard

5. update your initramfs:

  # mkinitramfs -o /boot/initrd.img-`uname -r`

6. it is sufficient to run lstrim weekly. So do not add it to the options in /etc/fstab, but rather install the following in /etc/cron.weekly/fstrim:

  #! /bin/sh
  for mount in / /boot /data; do
    fstrim -v $mount
  done

You should adjust the list of mount points to your situation. You’ll get a weekly e-mail with the amount of space that was trimmed; if you don’t like that, drop the -v argument to fstrim.

And that’s it. After a reboot, you can verify that TRIM is supported with:

# dmsetup table
sdaX_crypt: 0 929880064 crypt aes-xts-plain64 0...0 0 8:X 4096 1 allow_discards

If everything worked, you will see ‘allow_discards’ at the end of the line for your LUKS-backed device. And fstrim will not complain with ‘discard operation not supported’.

scontrol show daemons

command to figure out what it should run on the machine it is being invoked on. If you have a config file error, that command will return a blank line.

And /etc/init.d/slurm-llnl will exit without starting anything up.

scontrol will give you no indication of the nature of the configuration error. For that, use

slurmctld -Dvvvv

It’ll probably boil down to some dns/hostname related issue.

Sometimes one needs to make a PDF with fancy features (e.g. color gradients) available for online viewing. In Firefox, a click on the link to the PDF will open it in PDF.js, which will warn you if it has features it can’t quite deal with. PDF.js will still try to render it. That can lead to visual artefacts and/or be very slow, neither of which are desirable and can be confusing for users.

There is a way to disable PDF.js server-side, and force Firefox to open a download popup box when user clicks on a PDF:

AddType application/octet-stream .pdf

Unfortunately, that makes Chrome (on Windows only, for some reason) really unhappy: it will open a blank new tab and not download the PDF file.

So, here’s how you tell Apache 2.2 to only change the Content-type header for PDF files when the User Agent says ‘Firefox’:

  <IfModule mod_setenvif.c>
    BrowserMatchNoCase firefox pdf=stream
  </IfModule>

  <IfModule mod_rewrite.c>
    RewriteCond %{ENV:pdf} stream
    RewriteRule .pdf$ - [T=application/octet-stream]
  </IfModule>

The BrowserMatchNoCase sets an environment variable called ‘pdf’ to the value ‘stream’ (which is an arbitrary choice). That environment variable is checked by the RewriteCond line; if it matches ‘stream’, the RewriteRule on the next line changes the Content-type header.

This can be done in a less convoluted way on Apache 2.4 with the <If> directive.

* October 2004 – purchase Shuttle home server with 2x 200GB disk (200GB useable)
* May 2007 – upgrade to 2x 500GB disk (500GB useable)
* September 2009 – upgrade to MSI Wind with 2x 1TB disk (1TB useable)
* June 2012 – upgrade to 2x 2TB disk (2TB useable)

Looks like my upgrades are roughly 2.5 years apart. Interesting!

I bought one Hitachi HDS723020BLA642 (that’s model ’0f12115′, a 2TB SATA3 drive with 64MB of cache). Well, actually I ordered model ’0S02861′ which is SATA2 and only has 32MB of cache, but for some reason Amazon shipped me the faster one. Ah, well, I’m not complaining. The other drive is a Western Digital WD20EARX (that’s also 2TB SATA3 drive with 64MB of cache).

It’s worth noting that the two 1TB drives that are being replaced (WD10EADS-00L) have been rock solid over the past 2.75 years, despite a *lot* of data being written to them (nightly backups of more and more servers, which is also why I was running out of space…).

So now I have two SATA3 drives in a system that is SATA2 only. That’s too bad. On the other hand, this machine now has 10x the amount of disk space that its first incarnation had, back in 2004. Nice!

I found a migration script at norwinter.com, which I improved a bit. It will handle wiki pages with full history as well as attachments. It won’t preserve who committed revisions, however – that is hardcoded in the script. So, this is still a hack.

Usage instructions:

a) copy your MoinMoinWiki data/pages directory to the server that runs your redmine install
b) put the migrate_from_moinmoin.rake script in lib/tasks/ in your Redmine install
c) edit the migrate_from_moinmoin.rake script, replace both instances of YOUR@EMAIL.ADDRESS
d) run rake redmine:migrate_from_moinmoin RAILS_ENV=”production”
e) provide a unique redmine project id and the path to your MoinMoinWiki data/pages directory

And here is my version of migrate_from_moinmoin.rake.

This script worked well enough for me to import MoinMoinWiki version 1.5.7 to Redmine 1.2.0.

The machine booted into Grub’s rescue mode. Grub 2′s rescue mode is nice, but not exactly intuitive (no ‘help’ or ‘?’ command), and documentation for it is apparently not easily found via Google.

This is what was eluding me for a while:

In rescue mode, only the insmod, ls, set, and unset commands are 
normally available. If you end up in rescue mode and do 
not know what to do, then see "GRUB only offers a rescue shell."

That’s from the Grub manual at gnu.org.

The steps involved in recovery are documented in the same manual. Basically you need to set the prefix and root variables, and then insmod the ‘normal’ module and run it.

The packaging was not too great; while all disks were well packaged individually, the big boxes that contained the individual drive boxes were flimsy. As a consequence, one of the disks got rather damaged (the one on the right):

I don’t know what it got hit with, but it must have been a pretty serious blow. The aluminium enclosure of the drive is severely dented and even cracked; the white line in the image below is an actual crack in the metal:

Back in October 2009 I added 130 TB of disk to another cluster, which looked like this, prior to install:

That was 65 times WD2002FYPS.

So this time around – almost 18 months later – we get 27% more capacity using 15% fewer drives. Got to love the computer industry and the progress it makes.

The ‘Average CPU Power’ is based on ‘average’ use, which is explained on Wikipedia.

According to Wikipedia, the Thermal Design Power is the maximum power consumption for thermally significant periods running worst-case non-synthetic workloads (cf. this article). If we assume that the bulk of the electrical power consumed by a CPU is converted into waste heat, then the TDP can be a reasonable approximation for the amount of electrical power a CPU would consume under a worst-case, real-world load.

I used cpuburn to generate such a load. There was no IO load on the system during the tests. I measured power draw with an off-the-shelf Kill-a-watt, so these results should be taken with a grain of salt.

The data indicates that the idle vs. full load power consumption difference for one CPU is 70 to 75W.

If we assume the power consumption under full load is 115W (the TDP for the processor), then idle power consumption would be 40 to 45W per CPU. That would put idle power consumption at 35-39% of its TDP for this particular CPU.

-rw-r--r-- 1 root  root  1.1G 2010-10-31 20:19 10125-127-2010-10.error

After:

-rw-r--r-- 1 root  root   11M 2010-10-31 20:19 10125-127-2010-10.error.bz2

Bzip2 reduced the file to 1% of its original size. Not bad!