[ENBD] 2GB limit drives me crazy... :(

[Peter T. Breuer]

"A month of sundays ago Kai Chen wrote:"
> 
> >>[root@u575 /dev]# ls -al sdo3
> >>brw-rw----    1 root     disk       8, 227 Apr 16  1999 sdo3
> >
> >Do you really have all those disks ? o is something like the fifteenth!
> 
> OK, here is how my system is set up.  u575 is an nbd server that has an 
> internal SCSI drive and connected to an external Fibre Channel box that has 
> ten 36GB drives.  I set up 14 logical volumes over a RAID 5 array in the FC 
> box.  All of these logical volumes show up as SCSI devices in the system.

Logical volumes! You are running over LVM! Which kernel?

How can they show up as scsi devices? LV's show up as /dev/<VG>/<LV>.
(on 2.4.0) like this:

  nbd:/usr/oboe/ptb% ll /dev/users/
  total 0
  crw-r-----    1 root     109,   0 Mar  9 22:43 group
  brw-rw----    1 root      58,   1 Mar  9 22:43 lab
  brw-rw----    1 root      58,   0 Mar  9 22:43 lm

and if you want to know the sizes, yes, one of these is over 2GB

 nbd:/usr/oboe/ptb% cat /proc/lvm/VGs/users/group
 name:         users
 size:         6287360
 access:       3
 status:       5
 number:       0
 LV max:       256
 LV current:   2
 LV open:      2
 PV max:       256
 PV current:   4
 PV active:    4
 PE size:      4096
 PE total:     1535
 PE allocated: 1535
 uuid:         04FvrRskk20ff2mwKo0gMQj7gT2aDOt8

etc. etc.

> So /dev/sdo is my 14-th logical volume in my FC box.

This is simply not possible. 8 is the major for scsi devices. The major
for LVM is 58. From devices.txt in 2.4.0:

 58 char        Hayes ESP serial card - alternate devices
                  0 = /dev/cup0         Callout device for ttyP0
                  1 = /dev/cup1         Callout device for ttyP1
                    ...
    block       Reserved for logical volume manager

You are saying that it used to present itself as scsi in 2.2.18?

> My glibc is of version 2.1.3-15, so I guess it's OK, too.  I just don't 
> understand what's wrong with my system....  It's driving me crazy!!!  :(

There is something very basically wrong if scsi and lvm majors are
confused, no?

To be clear, this is the 2.4.0 kernel I am talking about.

> >I just am choosing conservative options for you, to see if it makes a
> >difference.
> 
> Well your effort paid off for me.  I'm very appreciative.

That's very interesting. It's clear indication of

1) either a mistake of mine (if merge_requests=0 makes a difference)

or

2) a buffer layer deadlock (if sync_intvl=1 makes a difference).

I suspect it's the latter. But please try and slow down your copies!
If it's (2) then you are filling the whole kernel up with dirty
buffers that want to go to the driver. When the driver gets them
it calls the client daemon to transfer them to the server and
free them up, and bang .. the client can't find any memory that is not
already claimed. Deadlock. I simply forced a sync every 4MB or so
of transfers this way, to keep the kernels buffer usage low. But I
need some way of either protecting some memory for the client's use
or a way of detecting that VFS is full and we need to sync the kernel
now before disaster happens.

If mounting the file system -o sync did not do the same job for you,
then that is an error in the file system code.


> >Are you running to localhost?
> 
> No.  Other than "make test" I didn't run anything to "localhost."  I use 
> another separate machine as the nbd-client.
> 
> All of my machines were installed with RedHat Linux with Kernel 2.2.14-5.0 
> originally.  Then I downloaded Kernel 2.2.18 from kernel.org and changed the 

OK .. you are using 2.2.18. But 2.2.18 does not have LVM, surely!
At least my copy does not (I have 18pre18). There is nothing in the
config file containing *LVM* and there is no Documentation/*{lvm,LVM}*.

So if you have LVM, it must be a patch.

Effectively, you ARE running to localhost. You are talking to the lvm
layers, which are on localhost.

> /usr/src/linux symbolic link to point to the new 2.2.18 linux directory.  I 
> did copy version*.h from the old 2.2.14-5.0 directory to the 2.2.18 one as 
> it doesn't have those files.  I guess they are probably a RedHat thing.  I 

If you copied version.h then you already have a huge problem. The
version numbers will be all wrong. The version.h is _generated_ by a
kernel compile. Mine says:

 nbd:/usr/local/src% cat linux-2.2.18-pre18/include/linux/version.h
#define UTS_RELEASE "2.2.18pre18-SMP"
#define LINUX_VERSION_CODE 131602
#define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))


> don't think those version control header files could cause the problem I'm 
> having now...

They certainly could, and worse. If you have the wrong kernel version
number then all kinds of tests inside the kernel code compile will go wrong.

But the real problem appears to me that  you have redhat code, not real
kernel code. There is no way a real kernel code compile could survive
the implant of a redhat "version.h" file, because I believe those are
awk files, and C code does not like to have awk code included in it!
At least the version.h that I have seen from redhat was awk code ..
written by someone who knows entirely too much awk!

> Oh, I guess I don't need to repeat what many people are saying, but I think 
> you have a wonderful product here.  Do you know if anyone is using it in a 
> SAN type of environment?  Combined with GB Ethernet, it could become an 
> open-source, more cost-efficient alternative to compete directly with FC or 
> iSCSI in the market.

I have received contacts from companies who appear to be talking exactly
those kinds of buzzwords. They too seem to want the device to present
itself as scsi (this would require a patch to the scsi driver, but not
much more).  If there is much more pressure of that kind, I will take
the code in that direction.

Peter