a new record

This morning, a user on one of our machines (inadvertently) created a mail loop with a bad procmail script:

09:42:05 up 120 days, 9:23, 20 users, load average: 3367.40, 3265.08, 2751.75

I had seen machines go up to about 200 before, but never this high. If you ever wonder about the stability of the 2.6 kernel – and this is a Xen setup! – here’s your answer. Even with the load this high, the machine was responsive enough on a couple of ssh sessions to solve the problem remotely.