[ENBD] More on 2GB Limits... Huh...

[Kai Chen]

Hi! Peter,

>Logical volumes! You are running over LVM! Which kernel?
>
>How can they show up as scsi devices? LV's show up as /dev/<VG>/<LV>.
>(on 2.4.0) like this:

Well I'm NOT running LVM with NBD yet, although I do plan to do so very 
soon.

The logical volumes are created by my FC controller, which is hardware 
based.  The computer talks to the controller through a QLogic FC card over 
FC wires.  These volumes are logical, because they are created over a RAID 5 
array and do not correspond directly to physical devices.

My kernel is 2.2.18, and I haven't upgraded it to 2.4 because I'm running 
AFS, which supports up to 2.2.18 only at this point.

>>My glibc is of version 2.1.3-15, so I guess it's OK, too.  I just don't 
>>understand what's wrong with my system....  It's driving me crazy!!!  :(
>
>There is something very basically wrong if scsi and lvm majors are
>confused, no?
>
>To be clear, this is the 2.4.0 kernel I am talking about.

Fortunately that's not the case here.  :)  I guess I didn't make myself very 
clear last time.  Hope the above explanation helps.

>1) either a mistake of mine (if merge_requests=0 makes a difference)
>
>or
>
>2) a buffer layer deadlock (if sync_intvl=1 makes a difference).
>
>I suspect it's the latter. But please try and slow down your copies!
>If it's (2) then you are filling the whole kernel up with dirty
>buffers that want to go to the driver. When the driver gets them
>it calls the client daemon to transfer them to the server and
>free them up, and bang .. the client can't find any memory that is not
>already claimed. Deadlock. I simply forced a sync every 4MB or so
>of transfers this way, to keep the kernels buffer usage low. But I
>need some way of either protecting some memory for the client's use
>or a way of detecting that VFS is full and we need to sync the kernel
>now before disaster happens.
>
>If mounting the file system -o sync did not do the same job for you,
>then that is an error in the file system code.

OK.  It's the combination of "merge_requests=0" and "mount -o sync" together 
that did the trick.  Missing one of them will again create deadlock.  
Together they seem to work fine.

>OK .. you are using 2.2.18. But 2.2.18 does not have LVM, surely!
>At least my copy does not (I have 18pre18). There is nothing in the
>config file containing *LVM* and there is no Documentation/*{lvm,LVM}*.
>
>So if you have LVM, it must be a patch.

Yes.  I do have LVM, and it is a patch for 2.2.18.  I just haven't run it 
with NBD yet.

>Effectively, you ARE running to localhost. You are talking to the lvm
>layers, which are on localhost.

I see.  Because I plan to do that soon, please tell me how running LVM makes 
a difference to the 2GB limit...

>>If you copied version.h then you already have a huge problem. The
>>version numbers will be all wrong. The version.h is _generated_ by a
>>kernel compile. Mine says:
>
>nbd:/usr/local/src% cat linux-2.2.18-pre18/include/linux/version.h
>#define UTS_RELEASE "2.2.18pre18-SMP"
>#define LINUX_VERSION_CODE 131602
>#define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c))

I'll fix that and try again.  Hope that's the problem which caused the 2GB 
limit...

>>Oh, I guess I don't need to repeat what many people are saying, but I 
>>think you have a wonderful product here.  Do you know if anyone is using 
>>it in a SAN type of environment?  Combined with GB Ethernet, it could 
>>become an open-source, more cost-efficient alternative to compete directly 
>>with FC or iSCSI in the market.
>
>I have received contacts from companies who appear to be talking exactly
>those kinds of buzzwords. They too seem to want the device to present
>itself as scsi (this would require a patch to the scsi driver, but not
>much more).  If there is much more pressure of that kind, I will take
>the code in that direction.

That's very interesting!!!  But I think we have much to talk about if people 
are trying to make you present them as SCSI devices...  :)  I believe what 
you're doing now, namely transferring only blocks over TCP and then present 
the device as a simple block device on the client side is the right thing to 
do for SAN.  I think iSCSI is valuable but not suitable for SAN if you're 
using it only to transfer data (if you're using it to control SCSI devices 
on the network then it is a different matter).  It's too thick for data 
transmission!!!  Blocks are the right level of abstraction.  FC is a block 
level protocol, but it's very expensive (an EMC box can cost $1M...).  I 
think NBD is a better alternative to FC than iSCSI...

Thanks.

Kai

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com