rails, mongrel and mysterious gpgme errors

Mongrel changes its user and group id to the user you specify when started as another user (say, root).

However, it does not change the environment variables from the user it was started as. In particular, ENV["USER"] and ENV["HOME"] remain set to, say, ‘root’ and ‘/root’ respectively.

If you then try to use the ruby gpgme wrappers you’ll run into strange errors. No Ruby exceptions are triggered, but things go wrong in the gpgme_n C wrapper code. From there, gpg is called as the mongrel user, but with environment variables pointing to root’s home directory. Ouch.

Of course you won’t see the problem in development mode (because typically there you run mongrel as the user you are working as).

The solution is simple: force those two variables in your environment.rb file to the correct values for the user you are running as, but not for development mode.

Posted in Rails | Leave a comment

remote install of a PE2900

I got a new Dell PowerEdge 2900 delivered at a customer site thousands of miles away. I installed it remotely after instructing the people onsite to plug in the network interfaces and power. That worked, but it would have been a whole lot easier if it wasn’t for those problems:

Problem 1: stupid default firmware settings.

The PE2900 came with a DRAC5 remote management card. First problem: this card defaults to a fixed IP address of 192.168.0.120. Whose idea was that? What’s wrong with defaulting to dhcp? Anyway, I got in by tweaking my network config a bit – but I’m glad I avoid 192.168.0.0/24 subnets, because if you were in 192.168.0.0/24 and you had a system on 192.168.0.120…

At that point I could ssh into the DRAC with the default username and password. Good. The ‘connect com2′ command connects to the serial port to which the console output can be redirected from the BIOS, allowing full remote text-based interaction. Perfectly sufficient if you run a unix based operating system. Except… that redirection is disabled by default. Seriously – this is supposed to be enterprise hardware?

Here’s how to fix this:

$ racadm config -g cfgSerial -o cfgSerialCom2RedirEnable 1
$ racadm getconfig -g cfgSerial
cfgSerialBaudRate=115200
cfgSerialConsoleEnable=1
cfgSerialConsoleQuitKey=^\
cfgSerialConsoleIdleTimeout=300
cfgSerialConsoleNoAuth=0
cfgSerialConsoleCommand=
cfgSerialHistorySize=8192
cfgSerialCom2RedirEnable=1
cfgSerialTelnetEnable=0
cfgSerialSshEnable=1

Here’s another one: all serial settings default to 115200 baud – except for the one setting that you need to get serial after the bootloader hands off to the kernel. That one is set to 57600. Why?

$ racadm getconfig -g cfgIpmiSol
cfgIpmiSolEnable=0
cfgIpmiSolBaudRate=57600
cfgIpmiSolMinPrivilege=4
cfgIpmiSolAccumulateInterval=10
cfgIpmiSolSendThreshold=220
$ racadm config -g cfgIpmiSol -o cfgIpmiSolBaudRate 115200
Object value modified successfully
$ racadm getconfig -g cfgIpmiSol
cfgIpmiSolEnable=0
cfgIpmiSolBaudRate=115200
cfgIpmiSolMinPrivilege=4
cfgIpmiSolAccumulateInterval=10
cfgIpmiSolSendThreshold=220

So if you get proper output from the BIOS, and grub, but only garbage from the kernel and your getty, this is why. Note: even setting (m)getty to 57200 produced garbage, seems like cfgIpmiSolBaudRate really needs to match the speed of the other serial parameters.

Problem 2: crappy QA

I was too lazy to figure out how to get the Hardy installer not use its graphical splash screen, so that meant I needed to use the web-based graphical console redirection. Similarly to my experience with HP’s ILO console redirection, this is *painful* if you run GNU/Linux on your client. Eventually I came across this blog post which worked for me, with a few caveats: I had to start from a clear profile (i.e. mv ~/.mozilla ~/.moz-backup), and using the firefox-2 packages in Ubuntu just does not work, I really had to download the Mozilla binary (I used 2.0.18).

Why does this have to be so difficult? Is it really that hard to keep the firefox modules more up to date, Dell? If you find that is the case, perhaps you should consider outsourcing this task to the free software community. Release the sources for the firefox plugin under a free software license, and watch a community develop around them to keep them up to date. Of course, that will also require cooperation from the DRAC side of things. How about opening up the source for the DRAC controller? Doing that would allow the community to fix bugs more rapidly, and add functionality. Since we’ve proven that we can do a free software system bios and mp3 player firmware, DRAC controller firmware should not be a problem.

Besides, the network interface and netstat output on the DRAC5 looks eerily familiar:

$ racadm ifconfig
eth0      Link encap:Ethernet  HWaddr 00:22:19:98:AF:AF  
          inet addr:192.168.0.120  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:110880 errors:0 dropped:0 overruns:0 frame:0
          TX packets:74405 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:10413478 (9.9 MiB)  TX bytes:40059892 (38.2 MiB)
          Interrupt:27 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:1020 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1020 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:739308 (721.9 KiB)  TX bytes:739308 (721.9 KiB)

$ racadm netstat
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U         0 0          0 lo
0.0.0.0         192.168.0.1     0.0.0.0         UG        0 0          0 eth0
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 192.168.0.120:5900      192.168.1.1:3007        ESTABLISHED 
tcp        0      0 192.168.0.120:22        192.168.1.1:1257        ESTABLISHED 
tcp        0      0 192.168.0.120:5901      192.168.1.1:2091        ESTABLISHED 

The DRAC5 wouldn’t be running the Linux kernel now would it? A little poking at a firmware update image confirms that yes, it is – I see Linux kernel strings, Busybox strings, etc. In fact, they didn’t even try to hid this:

$ sh --help
BusyBox v1.00 (2008.08.22-17:37+0000) multi-call binary

No help available.

So, where is the source? A recent thread on the linux-poweredge list at Dell suggests you can ask for SKU 420-3178. I think I might try that.

Posted in Everything else | Leave a comment

page 56

“”

Sorry, that was the nearest book – page 56 is a blank page. Second nearest then.

“The variable’s name must be a valid identifier”

Rules:

* Grab the book nearest you. Right now.
* Turn to page 56.
* Find the fifth sentence
* Post that sentence along with these instructions
* Don’t dig for your favorite book, the coolest, the most intellectual. Use the CLOSEST.

(via John’s blog)

Posted in Personal | Leave a comment

we might actually get real telecom policy

Via the wired blog: Net Neutrality Advocates In Charge Of Obama Team Review of FCC.

The two people appointed are Susan Crawford and Kevin Werbach.

They ‘get it’ – they understand that the US is a broadband backwater, that the current telecom policies only work to fatten the bank accounts of the big telecom players, and that high speed internet access is just like water, sewage, and electricity: it’s a utility, and should be treated as such.

I have high hopes for the FCC under the new administration. If they are going to set up rules that will actually advance broadband penetration and foster competition, the US could finally see some real innovation in telecom land. Who knows – we might get cheap, fast and unencumbered broadband after all. This will of course take years, but things in the US can change quickly if there is (political) will…

Some concrete steps I would take (not in any particular order):

* Set up policies to encourage customer-owned last mile

* Split up the big telcos and cable companies in an infrastructure and a services company each. The infrastructure companies will roll out and maintain the physical local access networks, and charge ISPs for access to them – but there will be no preferential treatment for any of them, and pricing and service levels will be heavily regulated and monitored to avoid abuse. Basically, this is a return to the unbundling of the local loops as it used to exist in the US for copper telephone wire, before Verizon and co pushed the CLECs out of business by lobbying for bad laws and pricing them out of the market. Of course the same rules need to apply to the cable networks, and to the fiber networks that are being rolled out. Unbundling is very much alive in large parts of Europe, and it works well there.

* Make it easier to roll out infrastructure. There needs to be a much more uniform legal framework for access to utility poles and digging up road to install new networks. Right now most of these rules are set at the municipality level, and the legal patchwork that results is very difficult and expensive to navigate.

* Encourage the installation of utility pipes or tunnels which can have new wiring blown through them much more cheaply, without digging up the roads every time. Germany has been doing this for decades – why can’t we?

* Get back those $200 billion in subsidies and tax breaks that were handed out to the big telcos for rolling out a next generation network – which they never did. That money needs to come back, with interest. The federal government can certainly use it right now.

* Enforce network neutrality. It’s very simple – the network needs to be stupid, the endpoints smart. That’s how you foster innovation. And as to the “the pipes will clog – people won’t be able to watch tv on the internet – the sky will fall!” fallacy: there is a very simple solution to that problem. Fatter pipes. If we had real broadband here (100Mbit+ to the home) and properly designed networks (bring down overcommits to a more reasonable level), this would not be a problem. Would it cost money to roll these networks out and keep upgrading them? Sure. But equipment cost would come down quickly as purchase volume goes up and mass production kicks in, that’s basic market economics… And sane government policies can help here, too.

Posted in Broadband, Politics | Leave a comment

Dell’s GNU/Linux support leaves to be desired

So I have a poweredge 2800 with some hardware issues (voltage sensors on the riser card seeing things they should not…). The Dell support folks wanted me to run a DSET report with this tool


http://support.euro.dell.com/support/downloads/download.aspx?c=uk&l=en&s=gen&releaseid=R155882&formatcnt=2&libid=0&fileid=208066

Man – what a pile of crap. This thing is a self-extracting archive for rpm-based systems (redhat/suse). Fine – except that I run Debian on that machine. Anyway, you can run it with –extract to extract the contents:

./delldset_v1.4.0.8.bin --help


Command-line options for Dell System E-Support Tool (DSET)

Usage:  [options...]

Options:

-h,--help        : Display command-line usage help
-i,--install     : Install/upgrade the DSET application
-v,--version     : Display version information
--list           : Display contents of package (+)
--extract  : Extract files to specified path (+)

Using no options will display the interactive user
interface which will guide you through using DSET.

These contents are basically an rpm file and some support files:

-r-xr-xr-x  1 ward ward     4133 Apr  2  2007 Dell_License*
-r-xr-xr-x  1 ward ward     9909 May  8  2007 README*
-r-xr-xr-x  1 ward ward       58 May  8  2007 Version.txt*
-r-xr-xr-x  1 ward ward 15200341 May  9  2007 delldset-1.4.0-8.i386.rpm*
-rw-r--r--  1 ward ward 15187240 Oct 27 15:57 delldset_1.4.0-9_i386.deb
-r-xr-xr-x  1 ward ward    10271 Apr  2  2007 install.sh*
-r-xr-xr-x  1 ward ward      472 Apr  2  2007 sphelp.txt*
-r-xr-xr-x  1 ward ward       24 May  4  2005 test.sh*
-r-xr-xr-x  1 ward ward     9877 Apr  2  2007 utility.sh*

The rpm can be transformed into a .deb with alien:

$ alien delldset-1.4.0-8.i386.rpm 
Warning: Skipping conversion of scripts in package delldset: postinst postrm preinst prerm
Warning: Use the --scripts parameter to include the scripts.
delldset_1.4.0-9_i386.deb generated
$ alien delldset-1.4.0-8.i386.rpm --scripts
delldset_1.4.0-9_i386.deb generated

Which installs fine – but you really want to be using a kernel that has ipmi functionality compiled as modules; otherwise the package will not install properly until you add

hapi.allow.user.mode=yes

to /etc/omreg.cfg. In any case, you need to have ipmi as modules for the rest of this sequence of commands, so you might as well boot into a stock Debian kernel.

You’ll probably want to have installed Dell’s open manage before you run the DSET report. To do that, add

  deb ftp://ftp.sara.nl/pub/sara-omsa dell sara

to your /etc/apt/sources.list file and

  apt-get update
  apt-get install dellomsa

So, then you would be ready to run the dellsysteminfo command but WAIT – don’t do it yet. That script is dangerous. It does stuff like this, probing for the running services (from /opt/dell/dset/bin/dell-sysreport.sh):

#Dell: Extra OS stuff we want

echo "     Getting status of services ..." ${TEE2LOG}

#Dell: SuSE doesn't have a "service status" method so do it the manual way 
#catifexec "/sbin/service" "--status-all" 
 
rm -f $ROOT/service > /dev/null 2>&1 
 
SERVICEDIR=/etc/init.d 
 
#pushd ${SERVICEDIR} > /dev/null 
          for SERVICE in `ls -1 ${SERVICEDIR}` ; do 
 
            case "${SERVICE}" in 
#Skip over these "services" since they tend to hang some systems when requested for status 
              functions | halt | killall | single| linuxconf| kudzu | bgpd | boot | reboot | single | nscd | \ 
                  halt.* | rc | boot.* | *rpmorig | *rpmnew | *rpmsave | *~ | *.orig) 
      ;; 
              *) 
                if [ -x "${SERVICEDIR}/${SERVICE}" -a -f "${SERVICEDIR}/${SERVICE}" ]; then 
        echo "Service Name: ${SERVICE}" >> $ROOT/service 
                 /usr/bin/env -i LANG=$LANG PATH=$PATH TERM=$TERM "${SERVICEDIR}/${SERVICE}" status >> $ROOT/service 2>&1 
    echo >> $ROOT/service 
    echo "---------------------------------------------------------------------------" >> $ROOT/service 
               fi 
                ;; 
            esac 
          done 
#popd > /dev/null

That didn’t go over too well on my system – a bunch of things stopped working (like the ‘w’ command), samba started segfaulting, etc. It’s clearly calling some init scripts that don’t deal well with being asked for a status.

So I commented out that part of the script, which made things a lot happier – but of course services are now not mentioned in the report.

The other problem with the dset package is that it is riddled with calls to awk, sort, etc with absolute paths. Seriously, this is not 1995. You’ll need to fix tons of calls – just grep for ‘/bin/awk’, ‘/bin/grep’, ‘/bin/sort’ etc in /opt/dell/dset.

When all that’s done, you can generate the report

  dellsysteminfo

which will drop a zip file with a horribly long name (including the use of round brackets, sigh) in /root/ with the password ‘dell’. Awesome security guys. Why is there even a password if it’s so trivial?

The Dell tech I’m e-mailing with (who is very helpful btw, this post is not a criticism about Dell’s support people – only about their GNU/Linux tools) said I’d need to upgrade the system bios and the ‘ESM’, which is the ‘system firmware’. The former was easy with the help of this blog post. In a nutshell:

aptitude install libsmbios-bin libsmbios1 libsmbiosxml1
getSystemId

That will print out your ‘System ID’. Look up the latest system bios for that System ID at


http://linux.dell.com/repo/software/bios-hdrs/

and download it. Then

modprobe dell_rbu
dellBiosUpdate -u -f bios.hdr

and reboot, at which point your bios will be upgraded if all goes well. Note that if your cmos battery is dead, you can’t upgrade your bios anymore because all the command line tool does is load it into ram and flip a switch to tell the bios to upgrade itself on the next boot; without cmos battery either that switch or the bios in ram may not survive a reboot.

So that worked fine. The ESM update was a little trickier. I downloaded another .BIN file from


http://support.euro.dell.com/support/downloads/download.aspx?c=fr&l=fr&s=gen&releaseid=R147948&SystemID=PWE_PNT_2800&servicetag=8YGYQ1J&os=WNET&osl=fr&deviceid=5814&devlib=0&typecnt=0&vercnt=7&catid=-1&impid=-1&formatcnt=4&libid=29&fileid=196749

At least that thing didn’t try to do all sorts of rpm manipulations, it just runs a few scripts (again, you can verify with –extract).

The tricky thing here is that running the BIN file conflicts with the ipmi modules. Here’s what I did:

Boot into single user mode, and stop openmanage and ipmievd:

/etc/init.d/instsvcdrv stop
/etc/init.d/ipmievd stop

Then unload all ipmi modules:

rmmod ipmi_devintf
rmmod ipmi_si 
rmmod ipmi_msghandler

Then start screen, and start the .BIN file. It will look around, cause the ipmi modules to be loaded again and if all is well ask you if you really want to update your ESM, [Y/N].

At this point, break out of screen with ctrl-a d, and unload the ipmi modules again (I kid you not!):

rmmod ipmi_devintf
rmmod ipmi_si 
rmmod ipmi_msghandler

Now return to screen

screen -r

and press ‘Y’. Don’t worry, it’s going to load the ipmi modules yet again, and then starts printing dots on the screen as the upgrade progresses. It takes a while – at least 5 minutes on my system. So just wait until it is done.

Eventually the script said ‘update complete’. I read this trick with screen and the unloading of the ipmi modules somewhere on a dell wiki, but I can’t find the link now (sorry!) so I can’t give credit.

Anyway, and I was able to verify the upgrade with ipmitool (I upgraded to firmware revision 1.72):

# ipmitool -I open mc info
Device ID                 : 32
Device Revision           : 0
Firmware Revision         : 1.72
IPMI Version              : 1.5
Manufacturer ID           : 674
Manufacturer Name         : Unknown (0x2a2)
Product ID                : 0 (0x0000)
Device Available          : yes
Provides Device SDRs      : yes
Additional Device Support :
    Sensor Device
    SDR Repository Device
    SEL Device
    FRU Inventory Device
    IPMB Event Receiver
    Chassis Device
Aux Firmware Rev Info     :
    0x00
    0x00
    0x00
    0x00

So in the end I got almost everything to work (except for the dset output which is still partially missing). But seriously, this was way harder than it should be.

That DSET package is a total mess and needs some serious reworking.

The system bios upgrade procedure is elegant and simple – there Dell deserves credit.

The ESM upgrade… not so much. Seriously – what’s up with those ipmi modules getting in the way twice?

In a nutshell, I’d like Dell to hire some people to clean up the GNU/Linux tools they provide. But they have to be people who know about more than just rpm-based distros, and who are interested in truly cross-distribution solutions and want to work with all (popular) distributions. Basically, if Dell can get their tools into Debian, Fedora and OpenSuse, things will trickle down to almost every other popular flavor of GNU/Linux. There is a lot of work to do…

Posted in Free Software/Open Source, Hardware | 1 Comment

hotmail’s smtp welcome message

I came across this rather nonsensical message from one of Hotmail’s MX servers:

$ telnet mx1.hotmail.com 25
Trying 65.54.245.8...
Connected to mx1.hotmail.com.
Escape character is '^]'.
220 bay0-mc9-f11.bay0.hotmail.com Sending unsolicited
commercial or bulk e-mail to Microsoft's computer network 
is prohibited. Other restrictions are found at 
http://privacy.msn.com/Anti-spam/. Violations will result 
in use of equipment located in California and other states. 
Sat, 25 Oct 2008 09:26:41 -0700 

(newlines added to improve readability) No wonder there is so much spam on the internet. If you violate Microsoft’s antispam policy, that will result in the use of equipment located in California and other states. Oh dear. They are going to use equipment! In California! And in other states! The spammers must be really frightened now.

Posted in Completely clueless | 2 Comments

43% == 50% according to Metro

From today’s Metro, Boston edition:

metro math impaired

I’m thinking someone at the Metro needs a crash course in elementary math.

Posted in Completely clueless | Leave a comment

city of Monticello, MN can proceed with its fiber rollout

TDS Telecom lost its lawsuit against the city of Monticello, MN.

Some background: citizens of Monticello approved in a referendum a plan to roll out a city-owned fiber network to provide cheap and fast internet access.

Immediately after the referendum the city was sued by Bridgewater, the local incumbent telco. Bridgewater and its parent TDS telecom challenged the city’s right to fund a fiber network with municipal bonds, citing ‘concern about wasteful spending’.

That was of course a pretext – what they did not want was competition. Thankfully Bridgewater lost the suit today, but they may have stalled the city enough to ‘win’ in the end anyway. Right after it filed suit – not coincidentally – Bridgewater started a fiber rollout of its own in Monticello. While the city could not spend bond money to build out the network and thus had to stall the rollout to residential neighborhoods, Bridgewater has already laid 20 out of a planned 100 miles of fiber.

So – now the city-owned network will have to compete with the Bridgewater network – and the latter can subsidize its network with income from neighboring communities.

On the upside – Bridgewater needs to make money, unlike the city. Those fat profit margines need to come from somewhere. And if I were a resident of Monticello, I know I wouldn’t be willing to give them any money, not after pulling this kind of stunt. Also, as the Ars article says, this decision is a ‘big green light’ for other communities in Minnesota that want to roll out their own network.

This kind of telco behavior makes me angry. If the telcos spent half the money they waste on anti-competitive behavior and lawsuits on actually upgrading and building out modern networks, this country wouldn’t be the overpriced broadband backwater it is. I hope many more cities start rolling out their own fiber infrastructure. It’s time for more competition – that’s the only way we are going to get decent and affordable broadband in the US of A.

Posted in Broadband | Leave a comment

lessons learned from a colo migration

I moved a bunch of servers from one colocation facility to another last night. I had been preparing this move for a few weeks. While everything went well in the end, things took much longer than anticipated. Here are a few lessons learned.

1. preparation is *everything*

I spent a lot of time preparing for the move, on three levels: communication, hardware, and software.

Communicating the planned outage to people affected beforehand is obviously very important. Not only does it warn people of the planned outage, but it also helps identifying ways in which customers are affected that perhaps were not anticipated. Who knew that customer XYZ’s fetchmail script used a fixed mailserver IP instead of the proper hostname? Your customers will help identify potential problem areas before the migration, which will save everyone involved time and frustration.

On the hardware level, it helps to know what you are getting into. I prepared by making sure I was thoroughly familiar with the hardware I was going to deploy – especially the stuff that you don’t use that often when everything works as expected: PDUs, serial console server, etc.

As for software, this migration involved IP address renumbering, which is never fun. The physical machines all run a number of Xen instances that each run their own applications. That meant that IP addresses had to be modified all over the place. Meticulous inventorizing of what needs to change where before the migration pays off bigtime when the big move comes. I even went as far as updating config files with the new information before shutting down the machines at the old location. This worked very well, as it saved a lot of time and effort during the migration: I just switched on the machines and dealt with the (far fewer) files that could not be modified beforehand. Reducing the DNS time to live to 300 seconds well before the migration was essential, obviously.

2. pay attention to the little things

Details, details, details. I lost time because rack posts that needed moving back, as it turned out, were screwed down with Torx screws. Luckily I brought a lot of tools among which a set of allen wrenches – but this prevented me from using my battery operated drill/screwdriver. Speaking of which – there were quite a few screws that were recessed too far to be reachable, again requiring manual tightening. Note to self: bring extender bit for the drill next time, and see if I can get some hex/torx bits.

I also lost time because I got stuck in traffic driving to the old colo. While this was not such a big deal since everything was still up, it did shorten the amount of time I had to work with during the maintenance window. Lesson learned: time migrations and/or allot enough time for bad traffic.

3. prepare for the unexpected

Something will happen that you did not anticipate. In this case, a machine with Intel e1000 nics refused to properly autonegotiate with an HP Procurve switch, leading to all sorts of speed and duplex mismatch problems. This appears to be a bug in the e1000 firmware (yay!) and no matter what I tried with ethtool and setting fixed parameters on either side of the link, the nic would just not talk properly to the switch. Workaround: put a different switch in between the e1000 and the Procurve. Lesson learned: bring spare parts, even for things that you think won’t fail. Until yesterday I held Intel’s nics in very high esteem. Today – perhaps not as much. Bigger picture lesson: expect the unexpected, and try to prepare for all sorts of eventualities. This also means applying the freelance consulting rule to the timing of your maintenance window: estimate how much time you will need, and then do that times two or two and a half.

4. set aside time for the fallout

Aside from the autonegotiation problem, things went pretty well last night. The migration took about 4 hours longer than expected, but for the bulk of that time most services had already been restored. Still – my announced maintenance window was too short. It turned out not to be too big of a deal since I was working through a good part of the night, but there was some impact since I had to deal with customers in three timezones, in total 8 hours apart.

While everything took (much) longer than expected, there were only a couple smaller issues to deal with today. Still, you want to budget for some time after a migration to stabilize things. This is also a good time to document the new setup!

Posted in Sysadmin | Leave a comment

25 years of GNU

The FSF has put out a 5 minute anniversary video celebrating 25 years of GNU.

freedom-fry

It’s a pleasant, concise explanation of what free software is all about, why it’s important, and how the GNU project fits in to all this. Happy birthday GNU!

And if that’s not enough FSF-related news for you, you can read more about what it is we do on a day to day basis.

Posted in Free Software/Open Source | Leave a comment