Off you go… into the purple yonder!

Out of the box, the Prometheus binaries in the latest release (v2.7.1) clock in rather large, at:

-rwxr-xr-x 1 ward ward 60097588 Jan 31 06:18 prometheus
-rwxr-xr-x 1 ward ward 38359349 Jan 31 06:19 promtool

Fillippo Valsorda has a post that describes shrinking golang binaries.

Prometheus uses the promu tool for building its binaries. Adding the necessary ld flags is done by editing .promu.yml file like this:

--- .promu.yml.orig	2019-02-20 08:57:29.413554323 -0500
+++ .promu.yml	2019-02-19 17:44:33.451539440 -0500
@@ -17,6 +17,8 @@
         -X github.com/prometheus/common/version.Branch={{.Branch}}
         -X github.com/prometheus/common/version.BuildUser={{user}}@{{host}}
         -X github.com/prometheus/common/version.BuildDate={{date "20060102-15:04:05"}}
+        -s
+        -w
 tarball:
     files:
         - consoles

After building the prometheus binaries with the latest stable go (1.11.5) using make build, we end up with:

-rwxr-xr-x  1 ward ward 46379136 Feb 19 17:45 prometheus
-rwxr-xr-x  1 ward ward 29391136 Feb 19 17:46 promtool

The binaries are now reduced to about 77% of their original size. Better, but still not great.

We can confirm they are now indeed statically linked and stripped of debug info:

$ file prometheus
prometheus: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped
$ file promtool
promtool: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped

Applying upx takes a while but yields excellent results:

$ upx --brute prometheus 
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  46379136 ->   9370100   20.20%  linux/ElfAMD   prometheus                    

Packed 1 file.

$ upx --brute promtool
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  29391136 ->   6271396   21.34%  linux/ElfAMD   promtool                      

Packed 1 file.

And here are the resulting binaries:

-rwxr-xr-x  1 ward ward 9370100 Feb 19 17:45 prometheus
-rwxr-xr-x  1 ward ward 6271396 Feb 19 17:46 promtool

The binaries are now down to roughly 16% of their original size. There is a noticeable startup delay at runtime for the upx decompression, but for a long-running daemon like prometheus that doesn’t matter at all.

For promtool, which is used more interactively, you might find the startup delay annoying. This is the original binary:

$ time ./promtool.orig 
usage: promtool.orig []  [ ...]

Tooling for the Prometheus monitoring system.
...

real	0m0.055s
user	0m0.036s
sys	0m0.013s

And here’s the UPX-compressed one:

$ time ./promtool
usage: promtool []  [ ...]

Tooling for the Prometheus monitoring system.
...

real	0m0.536s
user	0m0.521s
sys	0m0.016s

Even though I have been using SSDs for more than 5 years on my main machine, I’ve never bothered to set up TRIM properly. At first, that was because the drive didn’t support it – it was the first Intel X25M model, an 80GB drive. After that I moved on to an Intel X25M 120GB, and then an Intel 520 240GB. Both of those drives supported TRIM, but I didn’t configure my machine to use it.

I’m upgrading my laptop to SSD number four (an Intel 530, 480GB) and I decided it was time to set up TRIM properly. This is a fairly straightforward configuration with LUKS handling encryption for an LVM physical volume, inside of which the logical volumes live that I need to be able to run TRIM on.

Caveat emptor: because TRIM essentially makes it clear which blocks on the drive are in use and which are not, TRIM is disabled by default at the LUKS layer. Enabling TRIM will leak a bit of information (how many bytes are used by your encrypted partition), and should not be done in a ‘plausible deniability’ scenario.

There’s a lot of information out there on this subject – the best docs appear to be for Arch – but it seems that Debian does things slightly differently from most other distributions. Here’s what I had to do to make TRIM work.

1. In /etc/default/grub, add to the GRUB_CMDLINE_LINUX_DEFAULT and GRUB_CMDLINE_LINUX lines:

  cryptdevice=/dev/sdaX:vg_name:discard

Replace /dev/sdaX with the path to the partition that contains your LUKS volume. Replace vg_name with the name of your LVM volume group.

2. Run update-grub:

  # update-grub

3. In /etc/lvm/lvm.conf, set

  issue_discards = 1

in the devices section.

4. update /etc/crypttab: add ‘,discard’ to the fourth field for each encrypted partition:

sdaX_crypt UUID=abcdefgh-1234-5678-90ab-cdefghijklmn none luks,discard

5. update your initramfs:

  # mkinitramfs -o /boot/initrd.img-`uname -r`

6. it is sufficient to run lstrim weekly. So do not add it to the options in /etc/fstab, but rather install the following in /etc/cron.weekly/fstrim:

  #! /bin/sh
  for mount in / /boot /data; do
    fstrim -v $mount
  done

You should adjust the list of mount points to your situation. You’ll get a weekly e-mail with the amount of space that was trimmed; if you don’t like that, drop the -v argument to fstrim.

And that’s it. After a reboot, you can verify that TRIM is supported with:

# dmsetup table
sdaX_crypt: 0 929880064 crypt aes-xts-plain64 0...0 0 8:X 4096 1 allow_discards

If everything worked, you will see ‘allow_discards’ at the end of the line for your LUKS-backed device. And fstrim will not complain with ‘discard operation not supported’.

Slurm uses the

scontrol show daemons

command to figure out what it should run on the machine it is being invoked on. If you have a config file error, that command will return a blank line.

And /etc/init.d/slurm-llnl will exit without starting anything up.

scontrol will give you no indication of the nature of the configuration error. For that, use

slurmctld -Dvvvv

It’ll probably boil down to some dns/hostname related issue.

The built-in PDF.js viewer in Firefox is nice, but it still has quite a few bugs. Most of the rendering issues are caused by it not supporting certain PDF features yet.

Sometimes one needs to make a PDF with fancy features (e.g. color gradients) available for online viewing. In Firefox, a click on the link to the PDF will open it in PDF.js, which will warn you if it has features it can’t quite deal with. PDF.js will still try to render it. That can lead to visual artefacts and/or be very slow, neither of which are desirable and can be confusing for users.

There is a way to disable PDF.js server-side, and force Firefox to open a download popup box when user clicks on a PDF:

AddType application/octet-stream .pdf

Unfortunately, that makes Chrome (on Windows only, for some reason) really unhappy: it will open a blank new tab and not download the PDF file.

So, here’s how you tell Apache 2.2 to only change the Content-type header for PDF files when the User Agent says ‘Firefox’:

  <IfModule mod_setenvif.c>
    BrowserMatchNoCase firefox pdf=stream
  </IfModule>

  <IfModule mod_rewrite.c>
    RewriteCond %{ENV:pdf} stream
    RewriteRule .pdf$ - [T=application/octet-stream]
  </IfModule>

The BrowserMatchNoCase sets an environment variable called ‘pdf’ to the value ‘stream’ (which is an arbitrary choice). That environment variable is checked by the RewriteCond line; if it matches ‘stream’, the RewriteRule on the next line changes the Content-type header.

This can be done in a less convoluted way on Apache 2.4 with the <If> directive.