[ENBD] stability issues

[Peter T. Breuer]

"A month of sundays ago Arne Wiebalck wrote:"
> > between the entries in /proc/sys/vm/bdflush help?  The first entry marks
> > the onset of buffer transfers to disk in terms of % memory pressure.
> > The 7th entry marks the passing to synchronous buffer treatment, which
> > is bad.  I don't see why those shouldn't be 0% and 100% for nbd, but
> > 25% and 95% is probably very safe.
> 
> no. I tried that, but it did not help.

Strange, the test you did below suggests that VM is implicated. What
numbers did you use? And is this SMP or UP?

> > BTW, if you run the test with -y, then what happens is that every
> > write is followed by a sync on the device. This makes it impossible for
> > the kernel to store more than one write at a time for the device. It
> > makes imagining a bug even harder!
> 
> if the kernel stores only one request, why is memory usage increasing
> over time? I think I saw this on my machines.

It shouldn't. But memory usage is very hard to measure. There are stats
that you can look at if you press something like alt-shift-numlock
(I forget exactly), but they requre a lot of interpretation.

Some kind of _real_ memory leak would kill a machine over time. But as
I said, it's hard to imagine how it can come about from nbd, which does
the same thing again and again. What could leak? nbd doesn't do 
kmallocs. Maybe it could lose buffers irretrievably? A fault in a 
rarely used error path may do that.

> > If it doesn't occur with one channel and tcp/ip, and it does
> > with one channel and sci, then it's dubious.
> 
> no, vice versa. sci and one channel _seems_ more stable than tcp/ip and
> one channel. maybe this is due to the fact that sci internally does not
> need any buffers aside from segments set up in the beginning.

If there is a leak, then you should be able to detect it statistically.
That is to say, the memory "lost" should be proportional to the use of
the device. Reading or writing? I saw VM lose buffer heads on _read_.
That is to say, when the nbd driver tried to write incoming data to a
read request, it found the request didn't contain enough buffers
for the data.

> > and then perform the nbd-test suite against /dev/raw1? This is
> > a character device, but it understands read and write. You may have to
> > adapt nbd-test slightly - I'll help (or do it).
> 
> yes, I did this with tcp/ip and one channel over the whole weekend.
> although the raw manpage said I should not use dd, it worked fine.
> I have written to the nbd permanently with dd for more than 48 hours.
> so that seems "stable" :-)

Can you repeat doing reads as well? Does nbd-test now work if you shift
alignment of the buffers to 512B?

> what can we conclude from that test? can we say that the vm system is the
> source for the problems? or is it stable just because we throttled the
> input rate?

Well, the throttling should be only short-term. VM buffering speeds
up reads and writes instantaneously, but does nothing over the long
term (except shorten latency). And buffering still happens at the
server end. (One could connect the server to a raw device too to remove
buffering there :-).

I don't know. But I feel that removing buffering is what is wanted.

> maybe I should try reading also ...
> maybe I or you or (the both of us) should adapt nbd-test for raw
> devices...

Yes, I'll do it. According to you it just needs aligned buffers,
weird as it sounds!

Peter