Off you go… into the purple yonder!

I’ve got a number of servers that run coreboot + Xen. I like to run coreboot with a linux-as-a-bootloader (LAB) payload. That means that coreboot, after bringing up the machine, boots into a small linux kernel + busybox environment, entirely contained in rom. That environment can serve as an emergency fallback to resolve booting problems remotely - remember, coreboot has excellent serial support from power-on. Anyway; on a normal boot the machine kexecs into a Xen kernel.

And that works great. Recently, on deploying a new server, I discovered that the machine would just hang trying to kexec into Xen 3.2.1.

After a lot of digging and some fun setting up a test environment with qemu, I discovered that this broke somewhere between Xen 3.1.0 and 3.1.3. A message to the Xen-devel mailing list today yielded a quick response from a Xen developer suggesting I try the ‘no-real-mode’ parameter as an argument to the hypervisor.

The no-real-mode option stops Xen from doing some bios calls, and it also tells it to ignore the e820 bios table.

Here’s an lab.conf that works:

CMDLINE="no-real-mode com1=115200,8n1 cdb=com1"
INITRD=""
KERNEL="/xen-3.1.4.gz"
MODULE1="/vmlinuz-2.6.18.8-xen root=/dev/md1 ro xencons=ttyS0 console=tty0 console=ttyS0,115200"
MODULE2="/initrd.img-2.6.18.8-xen"

Note that you need to pass no-real-mode as the first xen command line argument, otherwise it won’t work.

And somehow that solves the problem for Xen 3.1.3 and 3.1.4, both under qemu + coreboot and qemu + bochs. Kexec’ing into Xen 3.1.2 triple-faults qemu under qemu + coreboot, but works fine under qemu + bochs. That smells like a coreboot bug. I’ll try on real hardware tomorrow to see if this bug is specific to our qemu port, or a more general coreboot problem. To be continued..

Ars Technica describes a pilot project in Ottawa that puts a new twist on telecommunication infrastructure: the customer-owned last mile.

From the article: A private company has recently completed a project to string dark fiber from a colocation facility under the Ottawa City Hall to a neighborhood of 400 older, upper-middle-class homes.

The idea is to sell the fiber to the home-owners. The larger the uptake, the lower the cost of the fiber will be since the bulk of the run is shared between all home owners. They estimate a cost of $2700 (Canadian $, I assume) if there is an uptake of 10%. If 50% of all homes sign up, the cost could be as low as $1000, and it can be paid with a lump sum or in installments.

The colocation facility is carrier-neutral, allowing the home owners to sign up with any ISP that has a presence there. A fiber management company will look after the fiber for a small fee, similar to how condo associations are often organized.

This is a really great idea. It puts the whole last mile problem on its head, and puts power in the hands of the home owners. Best of all - local communities can start doing this without having to wait for any state-wide or national laws.

So you’ve done quite a few dapper to hardy upgrades, and you’re ready for the next one.

You

  apt-get update
  apt-get upgrade
  apt-get install update-manager-core

and then

  do-release-upgrade -d

You go get a cup of tea, and come back to

# do-release-upgrade -d
Checking for a new ubuntu release
Done Upgrade tool signature
Done Upgrade tool
Done downloading
extracting 'hardy.tar.gz'
authenticate 'hardy.tar.gz' against 'hardy.tar.gz.gpg' 

Reading cache

Checking package manager
Reading package lists: Done
...
...
Done http://archive.ubuntu.com hardy-updates/restricted Sources
Done http://archive.ubuntu.com hardy-updates/universe Sources
Done downloading            

Error during update 

A problem occurred during the update. This is usually some sort of
network problem, please check your network connection and retry. 

W:Failed to fetch
http://security.ubuntu.com/ubuntu/dists/hardy-security/main/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://security.ubuntu.com/ubuntu/dists/hardy-security/restricted/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://security.ubuntu.com/ubuntu/dists/hardy-security/universe/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://security.ubuntu.com/ubuntu/dists/hardy-security/main/source/Sources.bz2
Hash Sum mismatch
, W:Failed to fetch
http://security.ubuntu.com/ubuntu/dists/hardy-security/restricted/source/Sources.bz2
Hash Sum mismatch
, W:Failed to fetch
http://security.ubuntu.com/ubuntu/dists/hardy-security/universe/source/Sources.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy/main/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy/restricted/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy/universe/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy/multiverse/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy/main/source/Sources.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy/restricted/source/Sources.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy/universe/source/Sources.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy-updates/main/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy-updates/restricted/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy-updates/universe/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy-updates/multiverse/binary-amd64/Packages.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy-updates/main/source/Sources.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy-updates/restricted/source/Sources.bz2
Hash Sum mismatch
, W:Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/hardy-updates/universe/source/Sources.bz2
Hash Sum mismatch
, E:Some index files failed to download, they have been ignored, or
old ones used instead. 

Restoring original system state

Aborting
Reading package lists: Done
Building dependency tree: Done
Building dependency tree: Done
Building dependency tree: Done

So you check the documentation at http://www.ubuntu.com/getubuntu/upgrading and http://help.ubuntu.com/community/HardyUpgrades. Nothing different. You try a different mirror. Same problem. What’s going on?

You google around a bit until you stumble across https://bugs.launchpad.net/ubuntu/+source/update-manager-core/+bug/223741 with a comment from 2008-07-17:

Using the ‘-d’ flag now causes Packages.bz2 and Sources.bz2 Hash Sum mismatch’s. “sudo do-release-upgrade” by itself now works. Please modify the upgrade instructions on http://www.ubuntu.com/getubuntu/upgrading as this is causing considerable problems with folks trying to upgrade. I’ve just tested on a 6.04.2 (fresh) install and the following works:

sudo aptitude upgrade
sudo aptitude dist-upgrade
sudo aptitude install update-manager-core
sudo do-release-upgrade

Ergo, drop the ‘-d’ and all will be well. Would sure be nice if the docs could be updated…

It’s official - we have a new logo!

Many thanks to the folks who made this happen:

* Konsult Stuge
* coresystems GmbH
* Kitchener Waterloo Linux User Group and Richard Weait
* Breakfast Design

So you’re happily developing a rails app, and for whatever reason need to restart your devel server.

And then you get this (paths removed to improve readability):

$ script/server
=> Booting Mongrel (use 'script/server webrick' to force WEBrick)
=> Rails application starting on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
** Starting Mongrel listening at 0.0.0.0:3000
** Starting Rails with development environment...
Exiting
.../vendor/rails/railties/lib/commands/servers/mongrel.rb:16: warning: already initialized constant OPTIONS
.../vendor/rails/railties/lib/commands/servers/mongrel.rb:19: undefined method `options' for []:Array (NoMethodError)
  from /usr/lib/ruby/1.8/rubygems/custom_require.rb:32:in `gem_original_require'
  from /usr/lib/ruby/1.8/rubygems/custom_require.rb:32:in `require'
  from .../vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in `require'
  from .../vendor/rails/activesupport/lib/active_support/dependencies.rb:342:in `new_constants_in'
  from .../vendor/rails/activesupport/lib/active_support/dependencies.rb:496:in `require'
  from .../vendor/rails/railties/lib/commands/server.rb:39
  from script/server:3:in `require'
  from script/server:3

Googling for this error shows that Mongrel is eating the real error message. There are patches that have been committed to work around this - and guess what, your rails version already has those patches *and yet you still can’t see what’s going on*.

A number of underlying problems are suggested - missing dependencies/gems, problems with gems being renamed between rails versions, etc.

As Murphy would have it, the error could well be caused by something totally different.

There’s an easy way to find out. Don’t use ’script/server’ to debug this. Just start your mongrels with mongrel_rails, which will tell you the problem right away, like it did for me:

$ mongrel_rails start
** Starting Mongrel listening at 0.0.0.0:3000
** Starting Rails with development environment...
.../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:249:in `load_missing_constant': Expected .../app/controllers/admin/key_hosts_controller.rb to define Admin::KeyHostsController (LoadError)
  from .../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:453:in `const_missing'
  from .../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/inflector.rb:257:in `constantize'
  from .../vendor/rails/activerecord/lib/../../activesupport/lib/active_support/core_ext/string/inflections.rb:148:in `constantize'
  from .../vendor/plugins/hobo/lib/hobo/model_router.rb:91:in `add_routes_for'
  from .../vendor/plugins/hobo/lib/hobo/model_router.rb:82:in `each'
  from .../vendor/plugins/hobo/lib/hobo/model_router.rb:82:in `add_routes_for'
  from .../vendor/plugins/hobo/lib/hobo/model_router.rb:70:in `add_routes'
  from .../vendor/plugins/hobo/lib/hobo/model_router.rb:70:in `each'
   ... 28 levels...
  from /var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:in `run'
  from /var/lib/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
  from /var/lib/gems/1.8/bin/mongrel_rails:16:in `load'
  from /var/lib/gems/1.8/bin/mongrel_rails:16

I made a stupid mistake in app/controllers/admin/key_hosts_controller.rb. No problems with dependencies or gems at all.

I don’t know what’s up with script/server, but it needs some love. Badly.

$ host ftp.us.debian.org.
ftp.us.debian.org has address 128.30.2.36
ftp.us.debian.org has address 64.50.238.52
ftp.us.debian.org has address 64.50.236.52
ftp.us.debian.org has address 35.9.37.225

$ telnet ftp.us.debian.org 80
Trying 35.9.37.225...
Connected to ftp.us.debian.org.
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.0 200 OK
Connection: close
Content-Type: text/html
Content-Length: 2949
Date: Tue, 03 Jun 2008 22:36:34 GMT
Server: Microsoft-IIS/6.0

Connection closed by foreign host.

Excuse me?

$ whois 35.9.37.225
Merit Network Inc. MICH-1 (NET-35-0-0-0-1)
                                  35.0.0.0 - 35.255.255.255
Michigan State University MICH-618 (NET-35-8-0-0-1)
                                  35.8.0.0 - 35.10.255.255

# ARIN WHOIS database, last updated 2008-06-02 19:10
# Enter ? for additional hints on searching ARIN's WHOIS database.

Why, exactly, is Michigan State running an official Debian mirror on IIS?

The three other hosts run a real OS/webserver.

WBUR, one of the two NPR stations in Boston now streams in unencumbered and patent-free Ogg Vorbis!

Check out the WBUR listen page. Here’s the direct link to the m3u. Also see the FSF press release about this.

DRM died on April 29th, 2008 when Microsoft announced it would be shutting down its ‘Plays for sure’ DRM validation servers on August 31st, 2008. All DRM’d music bought under the ‘Plays for sure’ system will become permanently tethered to the device it is currently ‘authorized’ on. When those devices fail or are replaced the DRM’ed music on them will be permanently lost.

While there are still many services with DRM out there, the de-activation of the ‘plays for sure’ system illustrates the core problem with DRM very clearly - somebody else holds the keys to your DRM’d music/movies/books. If they decide you can’t use them anymore - for whatever reason - your media become unusable.

Even those people who don’t know or don’t care about DRM will now think twice, particularly if they are burned by ‘plays for sure’.

Ergo, DRM is dead.

DRM’s funeral - the zuneral - was held this evening in Cambridge, MA. About 20 people attended the tasteful ceremony that ended with the submersion of the deceased in the Charles.

A plaque was revealed on the Weeks bridge to commemorate the event:

DRM is dead, rejoice!

More info about the death of DRM will become available on http://drmisdeadtome.org shortly. Check out my zuneral photos or all zuneral photos on flickr.

The coreboot symposium in Denver was a success. As I announced during my talk there, Silicon Mechanics has pledged to ship servers with coreboot preinstalled - more in particular, their Rackform nServ A236 model, which is a 1U pizzabox with a nice dual-socket F Opteron board by supermicro. I’ve got a bunch of those machines in production running coreboot, and they’re pretty awesome.

Marc’s AMD talk [PDF] was fascinating - AMD’s really behind coreboot, and there is a lot of hardware support in the pipeline. Awesome stuff.

Finally, we also ported a new board to v3: support for the PC Engines Alix.2C3 is now in the coreboot v3 tree. It’s not perfect yet - there’s an irq conflict between the usb ports and eth2 that needs to be sorted out, but we hope to fix that shortly and other than that it works great.

Update 2008-04-18: this is the Silicon Mechanics press release.

Update 2008-05-09: the irq issues on the alix.2c3 are resolved. Get revision 680 or higher.

I came across a nice PDF today. It caused an HP 4100 to just hang ‘processing job’; a Xerox 4500 printed it but overlayed the page with a lovely postscript error, and an Imagistics im2520f (a rebadged Minolta, really) just printed “Error: configurationerror”.

So I ran it through pdf2ps, after which it printed fine. Nice error message, no?

$ pdf2ps SAMSUNG13MIRsApr01Apr3008cd12.pdf

   **** Warning: Fonts with Subtype = /TrueType should be embedded.
                 The following fonts were not embedded:
                        Arial-Black
                        Arial-BoldMT
                        ArialNarrow-Bold
                        ArialNarrow-BoldItalic
                        TimesNewRomanPSMT

   **** This file had errors that were repaired or ignored.
   **** The file was produced by:
   **** >>>> Acrobat Distiller 7.0.5 (Windows) < <<<
   **** Please notify the author of the software that produced this
   **** file that it does not conform to Adobe's published PDF
   **** specification.