In order to install Rockbox on an iPod, it needs to be formatted in FAT32, not HFS+. The relevant wiki page over at the Rockbox site suggest either connecting the iPod to an iTunes install on Windows, or using one of the bootsectors they have available for download from that page.
Those boot sectors assume your iPod has one of the factory disks installed. I’ve got an old 4th gen iPod that I converted to compact flash after its disk died. It happens to have an 8G CF card in there.
Since I don’t do Windows, I downloaded the 20G 4th gen bootsector, put that on the iPod, and used fdisk to change the size of the FAT32 partition. And that worked fine.