[ENBD] [gmane.network.aoe.aoetools.general] enbd and 4TB

[Peter T. Breuer]

"Also sprach Florian Frank:"
> could enbd handle devices with 4TB size?

In principle, yes, but of course I have no way of checking!

> enbd-server displays the
> correct size:

enbd-server 1035 /dev/sda -i storage1b -b 512
...
enbd-server  4200: server (-1) sent size 3999887523840 ok

OK.



> but if i start enbd-client it displays:

> enbd-client storage1b:1035 -i storage1b -n 2 -m -b 512 /dev/ndb

> enbd-client 23395: client (-1) got size 3999887523840

Well, the right size was handed over in the negotiation.


> enbd-client 23395: client (-1) set device size 1800864268288

Interesting. Is that half? A bit less than half. Very strange. The code
goes:

   sscanf(buf, "%Lu\n", &size64) ....
   self->size = size64;
   MSG ("client (%d) got size %Lu\n", self->i, size64);

and the size field is 64 bit:

   u64         size;     /* PTB size of device in bytes */

so I would guess tha right size data is safely lodged in the client.
Let's move on ...

   sectors = size64 >> 9;
   // PTB rounding to multiple of 512
   size64 = ((u64) sectors) << 9;
   err = self->ioctl (self, ENBD_SET_SECTORS,
                          (char *) (unsigned long) sectors);
   MSG ("client (%d) set device size %Lu\n", self->i,
            (unsigned long long) size64);

so the problem appears to be the "sectors" count.

    unsigned sectors;

Yerrs .. that's 32 bit. The ENBD_SET_SECTORS call expects a 32 bit
argument. It counts in 512B sectors, which is 9 bits, so it can get to
41 bits in size, which is, uh, 2TB.


Hmmm. Houston, we seem to have a 2TB limit there!



> "Support for Large Block Devices" is aktiv on both systems, kernel
> 2.6.15 and enbd 2.4.32.

Well, I don't know what that means. It might mean that the internal
size count for a device is in 4KB blocks insteatead of 512B sectors, or
it might mean that they use a 64 bit count, still in sectors, and that
there is a new call to set a large block device size which takes some
kind of new unit or else is 64 bit.

Harumph. I suspect the easist thing to do is to invent a 
ENBD_SET_SECTORS64 ioctl to accept 64 bits for starters, then worry
about what to do in the kernel driver later.

In the enbd_base.c code, you should see

   case ENBD_SET_SECTORS:
        err = enbd_set_size (lo, ((__u64) arg) << 9);
        return err;

We will add another ioctl in like this:

   case ENBD_SET_SECTORS64:
        do {
            u64 longlongval;

            if (get_user (longlongval, (__u64 *)arg))
                 return -EFAULT;
            err = enbd_set_size (lo, longlongval);
            return err;
        } while(0);
        return -EINVAL;

and add a new ioctl definition in enbd.h

    #define ENBD_SET_SECTORS64    _IOWR(0xab, 0x30, __u64)

Then I think you can modify the setdevicesize() routine in
enbd-client.c as follows:

     - unsigned sectors;
     + u64 sectors64;

     ...

     - if (self->ioctl (self, BLKGETSIZE, (char *) &sectors) >= 0
     + if (self->ioctl (self, BLKGETSIZE64, (char *) &sectors64) >= 0
     -  && (int) sectors >= 0) {
     +  && (s64) sectors64 >= 0) {
     -  if (size64 != ((u64) sectors) << 9) {
     +  if (size64 != sectors64 << 9) {
            PERR ("client (%d) Warning! kernel says size is %Lu != %Lu\n",
     -            self->i, (unsigned long long) sectors,
     +            self->i, sectors64,
                  (unsigned long long) size64);
 
     ...

     - sectors = size64 >> 9;
     + sectors64 = size64 >> 9;
       // PTB rounding to multiple of 512
     - size64 = ((u64) sectors) << 9;
     + size64 = sectors64 << 9;
     - err = self->ioctl (self, ENBD_SET_SECTORS,
     + err = self->ioctl (self, ENBD_SET_SECTORS64,
     -                 (char *) (unsigned long) sectors);
     +                 (char *) &sectors64);


Let me know if that works or if anything else is required.
   

Peter