Off you go... into the purple yonder! » Xen

kexec’ing into a Xen kernel

ward — Fri, 08 Aug 2008 01:13:29 +0000

I’ve got a number of servers that run coreboot + Xen. I like to run coreboot with a linux-as-a-bootloader (LAB) payload. That means that coreboot, after bringing up the machine, boots into a small linux kernel + busybox environment, entirely contained in rom. That environment can serve as an emergency fallback to resolve booting problems remotely – remember, coreboot has excellent serial support from power-on. Anyway; on a normal boot the machine kexecs into a Xen kernel.

And that works great. Recently, on deploying a new server, I discovered that the machine would just hang trying to kexec into Xen 3.2.1.

After a lot of digging and some fun setting up a test environment with qemu, I discovered that this broke somewhere between Xen 3.1.0 and 3.1.3. A message to the Xen-devel mailing list today yielded a quick response from a Xen developer suggesting I try the ‘no-real-mode’ parameter as an argument to the hypervisor.

The no-real-mode option stops Xen from doing some bios calls, and it also tells it to ignore the e820 bios table.

Here’s an lab.conf that works:

CMDLINE="no-real-mode com1=115200,8n1 cdb=com1"
INITRD=""
KERNEL="/xen-3.1.4.gz"
MODULE1="/vmlinuz-2.6.18.8-xen root=/dev/md1 ro xencons=ttyS0 console=tty0 console=ttyS0,115200"
MODULE2="/initrd.img-2.6.18.8-xen"

Note that you need to pass no-real-mode as the first xen command line argument, otherwise it won’t work.

And somehow that solves the problem for Xen 3.1.3 and 3.1.4, both under qemu + coreboot and qemu + bochs. Kexec’ing into Xen 3.1.2 triple-faults qemu under qemu + coreboot, but works fine under qemu + bochs. That smells like a coreboot bug. I’ll try on real hardware tomorrow to see if this bug is specific to our qemu port, or a more general coreboot problem. To be continued..

xen 3.2 serial

ward — Mon, 18 Feb 2008 22:36:45 +0000

Getting access to the serial port in a Xen 3.2 dom0 is somewhat complicated. This is the magic incantation for your grub menu.lst file to get console at 115200 bps on the first physical serial port, as well as on the screen.

serial --unit=0 --speed=115200
terminal --timeout=5 serial console

title Xen 3
root    (hd0,0)
kernel    /boot/xen-3.gz com1=115200,8n1 console=com1,vga
module    /boot/vmlinuz-2.6.18.8-xen root=/dev/md0 ro xencons=ttyS0 console=tty0 console=ttyS0,115200n8
module    /boot/initrd.img-2.6.18.8-xen
boot

The ‘com1=115200,8n1 console=com1,vga’ arguments on the kernel line make sure that xen writes its output to the serial port (with the right speed, stop bit etc) as well as to the vga device. The ‘console=tty0 console=ttyS0,115200n8′ on the first module line tells the kernel to do the same. Xen controls the serial port hardware at this point (that’s the default in 3.2: XEN_DISABLE_CONSOLE is set in the dom0 kernel config!), and in order for the dom0 kernel and Xen to share that serial port, we have to tell the kernel to use the xencons virtual console driver – hence the ‘xencons=ttyS0′ parameter.

Debugging Xen with Serial Console has it almost right, but the xencons parameter is missing. Without xencons you’ll see serial output until the dom0 kernel taks over; that kernel won’t see any serial ports, which makes (m)getty very unhappy…

Xen 3.2.0

ward — Sat, 26 Jan 2008 20:55:16 +0000

So Xen 3.2.0 was released last week.

Oddly enough there are no precompiled 64-bit binaries anymore. That does not make much sense to me – running Xen on 32 bit is just… painful. Problems with accessing ram beyond 4GB, issues with libc6, etc. On 64 bit all that stuff just works.

Sadly, the 3.2 release is broken. It simply does not build – make world dies with this mercurial error:

select-repository: Searching `.:..' for linux-2.6.18-xen.hg
select-repository: Ignoring `.'
not found!
select-repository: Unable to determine Xen repository parent.

Google returns some hits suggesting that this is caused by using a version of mercurial that is too old, but that is nonsense – I’m seeing this even with 0.9.5. It’s a bug in the Xen 3.2.X release.

The fix is easy; just check out the linux-2.6.18-xen mercurial repository yourself in the parent directory to where you have downloaded Xen:

  apt-get install gawk libssl-dev libncurses5-dev pciutils-dev
  cd /usr/src/
  hg clone http://xenbits.xensource.com/linux-2.6.18-xen.hg
  cd xen-3.2.X
  make world
  make install

Of course you can use make -jX to compile things faster. Time to give 3.2 a spin now

a new record

ward — Thu, 19 Apr 2007 13:57:52 +0000

This morning, a user on one of our machines (inadvertently) created a mail loop with a bad procmail script:

09:42:05 up 120 days, 9:23, 20 users, load average: 3367.40, 3265.08, 2751.75

I had seen machines go up to about 200 before, but never this high. If you ever wonder about the stability of the 2.6 kernel – and this is a Xen setup! – here’s your answer. Even with the load this high, the machine was responsive enough on a couple of ssh sessions to solve the problem remotely.