Debian fails to boot: libm.so.6 is missing in initrd
After migrating my two identical server systems (radix, vicari) from 686 to amd64 (both Debian squeeze) I encountered the following problem at the next kernel update: radix worked fine whereas vicari failed to boot:
boot/bin/sh: error while loading shared libraries: libm.so.6: cannot open shared object file: No such file or directory
The problem was obviously the initrd. Fortunately I have a serial console on this server (including BIOS access) but Murphy was with me: It took me a few hours to boot a rescue grml successfully over NFS with PXE.
I compared the initrd of both systems:
scp root@radix:/boot/initrd.img-2.6.32-5-openvz-amd64 radix scp vicari@radix:/boot/initrd.img-2.6.32-5-openvz-amd64 vicari cd radix; gzip -dc initrd.img-2.6.32-5-openvz-amd64 | cpio -id; cd.. cd vicari; gzip -dc initrd.img-2.6.32-5-openvz-amd64 | cpio -id; cd.. diff -ur radix vicari >diffs
The difference was indeed that libm.so.6 was found in lib for the (working) radix case and in lib64 in the (broken) vicari case. The difference on both systems was /etc/ld.so.config: radix has /lib as first entry whereas vicari had /lib64 as first entry. I modified /etc/ld.so.config to read something like this:
/lib include /etc/ld.so.conf.d/*.conf /lib64 /usr/lib64 /usr/X11R6/lib64
and rebuilt the initrd with
The next boot was fine ...
grub2 has a console: Set it to serial if it is not done yet:
serial --speed=57600 --unit=0 --word=8 --parity=no --stop=1 terminal_input console serial terminal_output console serial
I could boot the system first with the correct initrd provided by radix:
set root='(/dev/sda,msdos1)' search --no-floppy --fs-uuid --set ef1c09ac-8e3c-4c5d-80c9-313c334ab2ed linux /boot/vmlinuz-2.6.32-5-openvz-amd64 root=UUID=ef1c09ac-8e3c-4c5d-80c9-313c334ab2ed ro console=ttyS0,57600n8 initrd /boot/initrd.img-2.6.32-5-openvz-amd64.WORKS
For the booting from PXE I first had the problem that booting the kernel and TFTP worked but afterwards no other communication. It took me roughly two hours to find out why: The interface on the switch was configured to use tagged 802.1q frames! To switch them to an ordinary port, I used (also from serial console) with a SG-3424:
configure interface ethernet 24 switchport type access switchport pvid 2 end