shrinking Prometheus binaries

Out of the box, the Prometheus binaries in the latest release (v2.7.1) clock in rather large, at:

-rwxr-xr-x 1 ward ward 60097588 Jan 31 06:18 prometheus
-rwxr-xr-x 1 ward ward 38359349 Jan 31 06:19 promtool

Fillippo Valsorda has a post that describes shrinking golang binaries.

Prometheus uses the promu tool for building its binaries. Adding the necessary ld flags is done by editing .promu.yml file like this:

--- .promu.yml.orig	2019-02-20 08:57:29.413554323 -0500
+++ .promu.yml	2019-02-19 17:44:33.451539440 -0500
@@ -17,6 +17,8 @@
         -X{{date "20060102-15:04:05"}}
+        -s
+        -w
         - consoles

After building the prometheus binaries with the latest stable go (1.11.5) using make build, we end up with:

-rwxr-xr-x  1 ward ward 46379136 Feb 19 17:45 prometheus
-rwxr-xr-x  1 ward ward 29391136 Feb 19 17:46 promtool

The binaries are now reduced to about 77% of their original size. Better, but still not great.

We can confirm they are now indeed statically linked and stripped of debug info:

$ file prometheus
prometheus: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped
$ file promtool
promtool: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, stripped

Applying upx takes a while but yields excellent results:

$ upx --brute prometheus 
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  46379136 ->   9370100   20.20%  linux/ElfAMD   prometheus                    

Packed 1 file.

$ upx --brute promtool
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
  29391136 ->   6271396   21.34%  linux/ElfAMD   promtool                      

Packed 1 file.

And here are the resulting binaries:

-rwxr-xr-x  1 ward ward 9370100 Feb 19 17:45 prometheus
-rwxr-xr-x  1 ward ward 6271396 Feb 19 17:46 promtool

The binaries are now down to roughly 16% of their original size. There is a noticeable startup delay at runtime for the upx decompression, but for a long-running daemon like prometheus that doesn’t matter at all.

For promtool, which is used more interactively, you might find the startup delay annoying. This is the original binary:

$ time ./promtool.orig 
usage: promtool.orig []  [ ...]

Tooling for the Prometheus monitoring system.

real	0m0.055s
user	0m0.036s
sys	0m0.013s

And here’s the UPX-compressed one:

$ time ./promtool
usage: promtool []  [ ...]

Tooling for the Prometheus monitoring system.

real	0m0.536s
user	0m0.521s
sys	0m0.016s
Posted in Sysadmin | Leave a comment

Debian Jessie: TRIM on LVM on LUKS on SSD

Even though I have been using SSDs for more than 5 years on my main machine, I’ve never bothered to set up TRIM properly. At first, that was because the drive didn’t support it – it was the first Intel X25M model, an 80GB drive. After that I moved on to an Intel X25M 120GB, and then an Intel 520 240GB. Both of those drives supported TRIM, but I didn’t configure my machine to use it.

I’m upgrading my laptop to SSD number four (an Intel 530, 480GB) and I decided it was time to set up TRIM properly. This is a fairly straightforward configuration with LUKS handling encryption for an LVM physical volume, inside of which the logical volumes live that I need to be able to run TRIM on.

Caveat emptor: because TRIM essentially makes it clear which blocks on the drive are in use and which are not, TRIM is disabled by default at the LUKS layer. Enabling TRIM will leak a bit of information (how many bytes are used by your encrypted partition), and should not be done in a ‘plausible deniability’ scenario.

There’s a lot of information out there on this subject – the best docs appear to be for Arch – but it seems that Debian does things slightly differently from most other distributions. Here’s what I had to do to make TRIM work.

1. In /etc/default/grub, add to the GRUB_CMDLINE_LINUX_DEFAULT and GRUB_CMDLINE_LINUX lines:


Replace /dev/sdaX with the path to the partition that contains your LUKS volume. Replace vg_name with the name of your LVM volume group.

2. Run update-grub:

  # update-grub

3. In /etc/lvm/lvm.conf, set

  issue_discards = 1

in the devices section.

4. update /etc/crypttab: add ‘,discard’ to the fourth field for each encrypted partition:

sdaX_crypt UUID=abcdefgh-1234-5678-90ab-cdefghijklmn none luks,discard

5. update your initramfs:

  # mkinitramfs -o /boot/initrd.img-`uname -r`

6. it is sufficient to run lstrim weekly. So do not add it to the options in /etc/fstab, but rather install the following in /etc/cron.weekly/fstrim:

  #! /bin/sh
  for mount in / /boot /data; do
    fstrim -v $mount

You should adjust the list of mount points to your situation. You’ll get a weekly e-mail with the amount of space that was trimmed; if you don’t like that, drop the -v argument to fstrim.

And that’s it. After a reboot, you can verify that TRIM is supported with:

# dmsetup table
sdaX_crypt: 0 929880064 crypt aes-xts-plain64 0...0 0 8:X 4096 1 allow_discards

If everything worked, you will see ‘allow_discards’ at the end of the line for your LUKS-backed device. And fstrim will not complain with ‘discard operation not supported’.

Posted in Sysadmin | Leave a comment

Happy 30th, GNU!

RMS launched the GNU project on September 27th, 1983.

MIT hackers generously lent us the use of the Green building tonight. It needs some of its pixels repaired, though!

Posted in Free Software/Open Source, Personal | Leave a comment

I object to the new Massachusetts Tax on Software Services

I object to the new “Sales and Use Tax on Computer and Software Services” (“the Tax”) which was passed as part of the Transportation Act.

I don’t mind paying higher taxes if the Commonwealth needs more money to fund public services. I support more tax dollars for public transportation.

However; I believe that taxes should be levied in a way that is fair. I also believe that the collection of taxes should be as efficient as possible, and that the collection of taxes should not cause undue burden on the people and companies of the Commonwealth.

The Tax is not a fair tax – why are ‘Computer system design services’ singled out for sales/use tax?

The Tax wil be complicated and expensive to comply with. It is administratively very complicated because the definition is complex and vaguely worded, and has a number of exceptions. It is complicated to collect because it is a state sales tax.

The Tax makes doing business more complicated for all IT businesses in the Commonwealth. The Tax will particularly harm the small businesses in our Commonwealth. That includes startups and aspiring entrepreneurs – the Tax will reduce our competitiveness. As such, the Tax will cause our Commonwealth long-term harm.

The Tax was passed under the radar, in the middle of the summer, and goes into effect 7 days (!) after the Transportation Act was passed. This is unreasonable.

The Tax needs to be repealed.

If you live or work in Massachusetts, please contact your State Representative and State Senator. The Department of Revenue is also inviting comments or suggestions at

1. The Tax is not fair.

Apart from telecommunication services, no other services are subject to sales/use tax in Massachusetts. Why are “Computer system design services” singled out?

2. The wording of the Tax is vague and unclear.

From TIR 13-10:

“Pursuant to newly-enacted legislation, An Act Relative to Transportation
Finance, St. 2013, c. 46 (“the Act”), which became law on July 24, 2013,
Chapters 64H and 64I of the General Laws have been amended to apply
the sales and use tax to certain services relating to computer system
design and to modification, integration, enhancement, installation, or
configuration of standardized or prewritten software.”

This wording makes no sense. Almost all software is ‘prewritten’. It is industry best practice to use existing software where it exists, and modify it to suit where necessary.

Taken literally, the Tax applies to practically every action of every IT professional. We are always modifying, integrating, enhancing, installing or configuring software – be it software we wrote ourselves or something that was acquired.

The Tax would require us to keep detailed accounting records of how much time was spent on software that was ‘prewritten’ vs. software that was written from scratch. Of course, almost no software is written entirely from scratch, most software builds on ‘prewritten’ components and libraries.

Doing this kind of accounting is absurd – it would be a ridiculous waste of time.

In real life – particularly so for free and open source software – there is no clear line between software that was ‘prewritten’ and software that was not. It does not make sense to create that artificial distinction in the Tax – it makes everything very complicated.

TIR 13-10 further defines:

“’Computer system design services’, the planning, consulting or designing of computer systems that integrate computer hardware, software or communication technologies and are provided by a vendor or a third party.”

This definition does not help.

What does this mean? What is included? What is not? What is a ‘communication technology’? Does there always need to be a hardware sale involved?

How does it apply to cloud computing services?

How does it apply to consulting services on free and open source software?

3. the Tax is complicated and unworkable.

The Tax suddenly subjects a whole class of small companies – startups and freelancers – to a vague and complicated tax rule. Now every IT freelancer will likely need to register with the Commonwealth to collect state sales tax.

We’ll need to hire lawyers and accountants to make sure we comply with this complicated, vaguely worded and unclear rule. This will suck up resources that
would be better spent growing our businesses and innovating.

This is an unreasonable burden that will hinder entrepreneurs and put a brake on innovation in the Commonwealth.

4. the Tax is unreasonable

The Tax was passed under the radar, in the middle of the summer, as part of the Transportation Act. It goes into effect tomorrow, July 31st 2013, 7 days after the Transportation Act was passed. This is unreasonable.

The Tax needs to be repealed.

If you live or work in Massachusetts, please contact your State Representative and State Senator. The Department of Revenue is also inviting comments or suggestions at

Posted in Politics | 1 Comment

Welcome Oscar!

Posted in Personal | Leave a comment

when `scontrol show daemons` returns nothing…

Slurm uses the

scontrol show daemons

command to figure out what it should run on the machine it is being invoked on. If you have a config file error, that command will return a blank line.

And /etc/init.d/slurm-llnl will exit without starting anything up.

scontrol will give you no indication of the nature of the configuration error. For that, use

slurmctld -Dvvvv

It’ll probably boil down to some dns/hostname related issue.

Posted in Sysadmin | Leave a comment

at the White House

I visited the White House last week to present a poster on during a Champions of Change event about Open Science.

There are more photos on Flickr.

Posted in Free Software/Open Source, Personal | Leave a comment

Selectively forcing PDF downloads for Firefox

The built-in PDF.js viewer in Firefox is nice, but it still has quite a few bugs. Most of the rendering issues are caused by it not supporting certain PDF features yet.

Sometimes one needs to make a PDF with fancy features (e.g. color gradients) available for online viewing. In Firefox, a click on the link to the PDF will open it in PDF.js, which will warn you if it has features it can’t quite deal with. PDF.js will still try to render it. That can lead to visual artefacts and/or be very slow, neither of which are desirable and can be confusing for users.

There is a way to disable PDF.js server-side, and force Firefox to open a download popup box when user clicks on a PDF:

AddType application/octet-stream .pdf

Unfortunately, that makes Chrome (on Windows only, for some reason) really unhappy: it will open a blank new tab and not download the PDF file.

So, here’s how you tell Apache 2.2 to only change the Content-type header for PDF files when the User Agent says ‘Firefox’:

  <IfModule mod_setenvif.c>
    BrowserMatchNoCase firefox pdf=stream

  <IfModule mod_rewrite.c>
    RewriteCond %{ENV:pdf} stream
    RewriteRule .pdf$ - [T=application/octet-stream]

The BrowserMatchNoCase sets an environment variable called ‘pdf’ to the value ‘stream’ (which is an arbitrary choice). That environment variable is checked by the RewriteCond line; if it matches ‘stream’, the RewriteRule on the next line changes the Content-type header.

This can be done in a less convoluted way on Apache 2.4 with the <If> directive.

Posted in Sysadmin | Leave a comment

more home server disk

I blogged almost 3 years ago about my home server upgrade. The thing has been running very reliably ever since, but I am running out of disk space again. So, time for another upgrade – disk only though this time. Notice the pattern?

* October 2004 – purchase Shuttle home server with 2x 200GB disk (200GB useable)
* May 2007 – upgrade to 2x 500GB disk (500GB useable)
* September 2009 – upgrade to MSI Wind with 2x 1TB disk (1TB useable)
* June 2012 – upgrade to 2x 2TB disk (2TB useable)

Looks like my upgrades are roughly 2.5 years apart. Interesting!

I bought one Hitachi HDS723020BLA642 (that’s model ’0f12115′, a 2TB SATA3 drive with 64MB of cache). Well, actually I ordered model ’0S02861′ which is SATA2 and only has 32MB of cache, but for some reason Amazon shipped me the faster one. Ah, well, I’m not complaining. The other drive is a Western Digital WD20EARX (that’s also 2TB SATA3 drive with 64MB of cache).

It’s worth noting that the two 1TB drives that are being replaced (WD10EADS-00L) have been rock solid over the past 2.75 years, despite a *lot* of data being written to them (nightly backups of more and more servers, which is also why I was running out of space…).

So now I have two SATA3 drives in a system that is SATA2 only. That’s too bad. On the other hand, this machine now has 10x the amount of disk space that its first incarnation had, back in 2004. Nice!

Posted in Everything else, Hardware, Sysadmin | Leave a comment

International Day Against DRM

May 4th is the International Day Against DRM.

To celebrate, get 50% off all e-books at – today only! Use the code DRMFREE.

Posted in DRM | Leave a comment