[ENBD] Doubled requests

[Peter T. Breuer]

"Also sprach Arne Wiebalck:"
> On Tue, 13 Jan 2004, Peter T. Breuer wrote:
> > > it is the same bh, shouldn't it be a different one?
> > The bh's can be reused, indeed, most likely are, in the normal course
> > of events.
> 
> Yes, bhs are being reused, but here we have two requests arriving at the
> device in a very short period of time. I can't imagine that the bh has
> already been released and put back for reuse. I wouldn't wonder if the bh
> struct is reused several requests later, but for two adjacent reqs?

It's possible - first released will go at the head of the free list
(surely it's only a singly linked list).  First reused will come off the
head of the free list.

Need more data!

> > Can you please also detail the size of the requests and other details
> > about them? Umm, what information? Maybe the full struct layout, or as
> > much of it as you can easily see.
> 
> The size is 1K, since I see "data wholes" of 1K, but I can dump the full
> bh struct and post it.

OK - so I understand that your device is being written to by the enbd
server? You should be able to confirm the writes in the server itself -
the writes will go through the fileserver.c frontend object 
to the file.c backend object. Its methods are:

   int (*write)(struct nbd_file *self, char * buf, unsigned len, u64 off);
   int (*read )(struct nbd_file *self, char * buf, unsigned len, u64 off);
   int (*sync )(struct nbd_file *self);
   int (*reopen)(struct nbd_file *self);
   int (*open)(struct nbd_file *self);
   int (*close)(struct nbd_file *self);
   s64 (*seek)(struct nbd_file *self, s64 off, int whence);

   int lives; // =0                /* number of reopens */
   int (*ioctl)(struct nbd_file *self, int ioctl, char *arg);
   short pid;
   int (*lock)(struct nbd_file *self, u64 from, s64 len, int type);
   int (*unlock)(struct nbd_file *self, u64 from, s64 len);

and they are set by the init_file() call.

I would try and confirm your reading by instrumenting the write method.
You probably can take the instrumentation from your driver, no?

> > > some more twins:
> > >
> > >     Jan 13 17:47:54 e005 kernel: cr: Suspicious block number: 11777 (last
> > >     11777) de6748c0 (de6748c0)
> >
> > Each request should contain many BH's, in general (i.e., many buffers).
> > Is there any more information you can get on the contents of the
> > request?
> 
> Since my device does not deal with reqs, but only uses a make_req_function,
> I only see single BHs.

OK. You run make_request(bh).


> > Not necessarily. That they are not different may well be a function of
> > whatever mechanism it is.
> 
> One thought was that they are somewhow returned from my devices to the
> kernel with an error and then a retry occurs. For the kernel there would
> be no reason to allocate a new BH for that.
> 
> However,
> 
> i)  My device _should_ notify me about an error scenario.
> 
> ii) It does not happen without enbd, i.e. if I use the device directly.

This may be a timing effect. Write requests are normally "plugged"
(blocked) by the kernel and aggregated into bigger requests, then
released all at once to the driver.  This normally results in
consecutive writes to the same place overwriting not the target device,
but only the buffers held in the kernel. Then the resultant buffers are
released.

So it is hard to imagine how any two consecutive writes to the same
place can ever appear in a driver.

BUT - if the two writes arrive in the kernel either side of a kernel
release ("unplug") event, then they would be seen in the driver. This
would be a rare event. It would arise whenever one always wrote
everything twice - only a few of the doubles would actually hit the
driver instead of the kernel buffer.

You say 1/100000. That is about 1/100 squared. I would estimate the
chances of two consecutive writes to the same place falling either
side of a kernel unplug as about 1/100, since there will be a max of
128 or so requests stored before release.

> I'll do some testing on a UP machine, but I got little hope that it will

You can try with just one enbd channel. If this is a enbd driver
problem arising from an unlocked window in the code, the frequency
woud be about proportional to n-1, where n is the number of daemons.

That is, once one daemon decides to place its window of opportunity,
each of the other n-1 daemons get a chance to hit that window. Grr ..
and each other. I suppose that would be (n-1)n/2, then. I.e. for each
time slot, the probabilty that some two daemons chose that time slot
to lay their window of opportunity in is proportional to the number
of pairs of daemons there are. 

You should be able to vary n easily and see how the observed frequency
changes.

> solve the issue, since I am currently running only 1 daemon anyway.

Oh - well then it cannot be a collision issue in the driver :-).
And if the driver does not say that it has done a rollback, it has not
repeated a reuest attempt.

Peter