[ENBD] Re: Some Questions

[Peter T. Breuer]

And while I am testing .. I tried to locate a limit on bandwidth by
removing the network card from the equation. Indeed, I seem to see a
bandwidth limitation of about 17MB/s operation to loopback nbd. But then
I see the same figures to my local disk, without nbd!

barney:/mnt% df .
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/nda                  7920        20      7492   0% /mnt
barney:/mnt% sudo ~/bonnie -s 7
File './Bonnie.650', size: 7340032
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start ém...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
            7  7331 91.0 29130 61.0 17174 74.3  8390 90.1 52295 51.1 7400.4 96.2

This is a P2 300MHz under kernel 2.2.15 with nbd 2.4.15 running 4KB
blksize with four client daemons... It would be faster if I striped the
resource 4-ways. But it's clear that reads all went to the kernel
buffers. Nearly none went through to the device. So the bonnie read
resulyts are pure kernel vfs.

Device a:       Open
[a] State:      initialized, verify, rw, enabled, last error 0
[a] Queued:     +0 curr reqs/+0 real reqs/+42 max reqs
[a] Buffersize: 86016   (sectors=168)
[a] Blocksize:  4096    (log=12)
[a] Size:       8192KB
[a] Blocks:     2048
[a] Sockets:    4       (+)     (+)     (+)     (*)
[a] Requested:  1175    (285)   (222)   (343)   (325)   8R/1167W
[a] Despatched: 1175    (285)   (222)   (343)   (325)   8R/1167W
[a] Errored:    0       (0)     (0)     (0)     (0)
[a] Pending:    0R/0W+0R/0W
[a] Kthreads:   0       (0 waiting/0 running/1 max)
[a] Cthreads:   4       (+)     (+)     (+)     (+)
[a] Cpids:      4       (641)   (642)   (643)   (644)
Device b-p:     Closed


And to my local disk (well, really to the disk buffers, at this size):

barney:/usr/oboe/ptb/lang/c/nbd/nbd-2.4.15% sudo ~/bonnie -s 7
Password:
File './Bonnie.672', size: 7340032
Writing with putc()...done
Rewriting...done
Writing intelligently...done
Reading with getc()...done
Reading intelligently...done
Seeker 1...Seeker 2...Seeker 3...start ém...done...done...done...
              -------Sequential Output-------- ---Sequential Input-- --Random--
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks---
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
            7  7293 91.6 49739 97.1 19868 88.7  8574 92.1 74242 51.8 7368.1 97.6
barney:/usr/oboe/ptb/lang/c/nbd/nbd-2.4.15% 

To the naked eye, it looks like a kernel or VFS limit on read at about
50MB/s. Tell me if you deduce anything more from those figures. I don't
see 17MB/s except on "rewrite". What is that in the bonnie code?

Peter