[ENBD] multiple write clients

[Bryan Bayerdorffer]

Any success yet with ENBD and openMosix?  I'm about to attempt the same thing,
that is exporting each node's local disk to all nodes via ENBD.

I suppose a node that exports a given device needs to access it strictly via
its own external interface to avoid deadlocks and/or multiple write entry
points?  Does the attached quote from Peter suggest that there is some more
direct method for accessing the local device in a coherent way?

What about pooling the exported devices under LVM (on each node!), assuming the
volumes are configured identically on each node (see second quote)?


Peter T. Breuer wrote:
> On localhost, there is no need to use the network and the client/server
> apparatus at all. I think the nbd-client.c code can initialize a
> "fileserver" instead of a "netserver" as the stub that it talks to.
> 
> It will require some coding, but i can work with you to set it up.
> The thing to change may be as little as this line at the end of
> nbd-client.c.
> 
>   
>       init_netserver(&self->server, self->i, self->size, &self->stream,
>                  0 != (self->flags&NBD_CLIENT_SERVER_RO), NULL,
>                  0 != (self->flags&NBD_CLIENT_ASYNC));
> 
> I think it should be changed to  (from fileserver.c)
> 
>   init_fileserver(struct nbd_stub_server *self, short i,
>        s64 size, void *file, short ro, void *alternate):
> 
> with appropriate arguments.  Maybe
> 
>   init_fileserver(&self->server, 0,
>        size, pointertonbdfile, 0 != (self->flags&NBD_CLIENT_SERVER_RO),
>        NULL);
> 
> 
> ??  and then you only need the pointertonbdfile. I think that's an "nbd file
> object", as described in the file.c file. It'll have to be made first.
> It's probably OK to steal the bit of code that initilaizes this struct
> from nbd-server.c.
> 
>     
>      struct nbd_file * file = & self->file;
>      init_file(file, nfile, mode, blksize,
>                             names, flags & F_READONLY);
> 
> where nfile = 1, mode = NBD_LINEAR_MODE, names is an array containing
> just the name of the file you want, and you can guess the rest.
> 
> Get back to me.
> 
> Peter


"" wrote:
> In a message dated: Mon, 16 Sep 2002 11:53:00 +0200
> chacron1 said:
> 
> >Hi,
> >
> >I would like to know if it is possible to use LVM on top of enbd devices?
> 
> Yeah, it should work.  I've done it a couple of times, but mostly 
> just in an experimental capacity.  I've never trusted live data to 
> it, or done any formal testing.
> 
> However, I'm planning on doing this quite shortly (within the next 
> week or so) using 5 enbd servers and 1 enbd client, then using RAID5 
> and LVM (or EVMS) to manage the disk space.  I call this a RAIN model
> (Redundant Array of In[dependant,expensive] Nodes) where you can 
> actually create RAID5 stripes accross nodes and use LVM or EVMS to 
> mirror those RAID5 stripes!
> 
> I posted quite a detailed message to the LVM mailing list a few weeks 
> back which outlined my plan if you care to search the archives for it.
> -- 
> 
> Seeya,
> Paul


Tad Kollar wrote:
> I'm very interested in this as well. I tried Sistina's GFS 4.0 almost two 
> years ago with high hopes but did not get great results with speed and 
> stability. Version 5.1 is probably much improved but I haven't tried it, nor 
> openGFS, yet. Their approach is to have the fs take care of locking through a 
> protocol called DMEP. However, few, if any, devices support DMEP, so they made 
> an IP version which required an additional host. They also have a gnbd. There 
> are several papers at www.sistina.com.

> Tad

> Huntsinger, Reid wrote:
> > I'd be interested to test. I would be especially interested in using it
> > together with openMosix (currently requires 2.4.19 kernel). I suppose this
> > might make DFSA ("direct filesystem access", openMosix's way to allow a
> > migrated process to do filesystem access from remote cluster nodes) and
> > openMFS (a sort of cache-coherent NFS which works with DFSA) unnecessary? Or
> > perhaps I've misunderstood.
> > 
> > As some interest has been expressed in ENBD on the openMosix list, I'm
> > forwarding the message there. Hope that's OK.
> > 
> > I'm pretty ignorant about the issues involved. Any comments on how this
> > might compare with things like openGFS, IBM's GPFS, Lustre, or PVFS? That
> > is, other ways to share a disk? 
> > 
> > Reid Huntsinger