[ENBD] diskless enbd-client/enbd on initrd

Thu Feb 23 03:25:38 MST 2006

Hi folks,

does anyone have experience with enbd-client on an initrd?
I'm trying to get a virtual machine running (under Xen 3.0.1, Kernel 
2.6.12.6) with an enbd-backed root device. It runs so far with 
nbd-client and nbd_root v0.1, but i experienced stability problems from 
time to time with it (leading to crashes of the VM due to disk problems).

I modified the nbd_root package to use enbd-client (static build of 
course), included insmod and enbd.ko (the mod seems to get loaded 
without any problems) and modified the init program, which triggers 
insmod, enbd-client etc.
Furthermore I altered the /dev structure of the initrd to provide 
/dev/nda and /dev/nda[1-4] and also added /var/run, /var/stat and 
/var/state/nbd (because i thought maybe enbd-client requires them to exist).

Here's the output of my tests:

-- snip --

Linux version 2.6.12.6-xenU (root at amd64) (gcc version 3.3.5 (Debian 
1:3.3.5-13)) #6 Wed Feb 22 17:33:21 CET 2006
BIOS-provided physical RAM map:
  Xen: 0000000000000000 - 0000000018200000 (usable)
0MB HIGHMEM available.
394MB LOWMEM available.
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line: 
ip=172.16.20.3:1.2.3.4::255.255.255.224:nbdtest:eth0:dhcp root=/dev/ram0 
4 initrd=initrd enbdroot=172.16.20.1:1111 init=/linuxrc
Initializing CPU#0
PID hash table entries: 2048 (order: 11, 32768 bytes)
Xen reported: 2000.066 MHz processor.
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
vmalloc area: d9800000-fb7fe000, maxmem 34000000
Memory: 386816k/403456k available (1491k kernel code, 8184k reserved, 
462k data, 108k init, 0k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Mount-cache hash table entries: 512
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 512K (64 bytes/line)
CPU: AMD Athlon(tm) 64 Processor 3000+ stepping 08
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... disabled
checking if image is initramfs...it isn't (bad gzip magic numbers); 
looks like an initrd
Freeing initrd memory: 1520k freed
Grant table initialized
NET: Registered protocol family 16
xen_mem: Initialising balloon driver.
SCSI subsystem initialized
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
netfront: Initialising virtual ethernet driver.
NET: Registered protocol family 2
IP: routing cache hash table of 4096 buckets, 32Kbytes
TCP established hash table entries: 16384 (order: 5, 131072 bytes)
TCP bind hash table entries: 16384 (order: 4, 65536 bytes)
TCP: Hash tables configured (established 16384 bind 16384)
NET: Registered protocol family 1
NET: Registered protocol family 17
IP-Config: Complete:
       device=eth0, addr=172.16.20.3, mask=255.255.255.224, 
gw=255.255.255.255,
      host=nbdtest, domain=, nis-domain=(none),
      bootserver=1.2.3.4, rootserver=1.2.3.4, rootpath=
RAMDISK: romfs filesystem found at block 0
RAMDISK: Loading 1520KiB [1 disk] into ram disk... done.
VFS: Mounted root (romfs filesystem) readonly.
Freeing unused kernel memory: 108k freed
enbd_root version 0.2
root: 172.16.20.1:1111
Loading enbd module...
exec: Success
Starting enbd-client...
enbd-client   572: client says target 0 is 172.16.20.1:1111
mount: No such device or address
pivot_root: Device or resource busy
Failed to start init :(
Kernel panic - not syncing: Attempted to kill init!

-- snip --

and here's what happens inside linuxrc.c (the init program); this is the 
relevant part of main()
usually the ip/port parameters are inserted via a kernel-param...I 
hardcoded it for testing purposes (and yes, the enbd-server on the other 
side is definetly running)

-- snip --
   if ((pid = fork())) {
     waitpid(pid,NULL,0);
   } else {
     fprintf(stderr, "Loading enbd module...\n");
     system("/bin/insmod /modules/enbd.ko");
     perror("exec");

     fprintf(stderr, "Starting enbd-client...\n");
     execl("/bin/enbd-client", "enbd-client", "172.16.20.1:1111", "-n", 
"1", "-i", "xenoo", "/dev/nda", NULL);
     perror("exec");
   }

   if (mount("/dev/nda", "/mnt", "ext3", MS_RDONLY, "")) {
     perror("mount");
   }
   if (stat("/mnt/initrd", &statbuf)) {
     fprintf(stderr, "The directory /initrd does not exist on the root 
filesystem.\nPlease create it and try again.\n");
     return 1;
   }
   if (pivot_root("/mnt","/mnt/initrd")) {
     perror("pivot_root");
   }
   chdir("/");
   execl("/sbin/init", "init", NULL);
   fprintf(stderr, "Failed to start init :(\n");

-- snip --

I really hope anyone is able to give me some hints on that or has 
already managed to setup a similar environment.

-- 
Mit freundlichen Gruessen / with kind regards
  Rudolph Bott

-------------------------------------------------------------------
Megabit Informationstechnik GmbH  Karstr.25  41068 Moenchengladbach
Tel: 02161/308980   mailto:info at megabit.net       ftp://megabit.net
Fax: 02161/3089818  mailto:support at megabit.net   http://megabit.net
-------------------------------------------------------------------