[ENBD] Extreme CPU usage by enbd-client?

[Peter T. Breuer]

"Also sprach Bas van Schaik:"
> Quote (Denis Bonnenfant):
> > Perfs are pretty good. Without any tweaking,  36 MB/s during
> > reconstruction over GbE link, and bonnie between 12-20 MB/s Writes / 20-50 MB/s
> >  reads. It's a little bit less for writes thant with direct access to disk, and
> > slightly more for reads.
> 
> My eyes dropped out after reading this: 35MB/s during reconstruction?!

I suppose he might have been using FR1. But higher rates are perfectly
possible.

> Currently, I'm running four enbd-servers (somewhat older hardware like
> 800MHz, but I don't think that should cause problems?) with the
> following options:
> > enbd-server 1100 /dev/hda5 -i iss01-hda5 -b 4096
> 
> And the clients:
> > enbd-client iss01 1100 -n 2 -e -b 4096 /dev/nda
> 
> On a P4 2.8GHz, I get about 5MB/s (!!) during initial reconstruction of
> my raid5 array (without any fastraid patches),

Raid5 writes several times for each block (you might be lucky if a
complete stripe is written at once).

but that machine was
> instable (some hardware problems). I've replaced the client machine with
> a P4 2.6GHz and now I get about 3.5MB/s during initial reconstruction.
> The initial reconstruction hasn't finished yet, so I can't do other

Are you aggregating requests? What does /proc/nbdinfo show for the
request sizes? Over a Gfast ethernet you want to aggregate requests
if you can. Echo something to nbdinfo (I forget what) or echo something
to a proc/sys area ...

   % cat /proc/sys/dev/enbd/devices/a/merge_requests
   0

Set that to some-large-number.


> I've tried multiple block sizes, multiple combinations of servers (to

Well, larger blocksizes should show something!

> test if one of the servers is the bottleneck), but the difference
> between a P4 2.8GHz client and a P4 2.6GHz client makes me think the
> problem lies within the client.

??


> One thing which attracts my attention, is the CPU usage on the client.

Look at nbdinfo for data.


> FYI, the top output during the reconstruction:
> >   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> >  3072 root      15   0  1932  988 1500 S 10.0  0.2   1:07.01 enbd-client

Not terribly high, but you seem to have a lot of them!

> >  3076 root      15   0  1932  988 1500 S  8.7  0.2   1:07.55 enbd-client
> >  3067 root      15   0  1932  988 1500 S  8.3  0.2   1:00.62 enbd-client
> >  3071 root      15   0  1932  988 1500 S  7.5  0.2   1:00.07 enbd-client
> >  3080 root      15   0  1932  988 1500 S  7.5  0.2   0:57.85 enbd-client
> >  3077 root      15   0  1932  988 1500 S  5.8  0.2   0:58.34 enbd-client
> >  3081 root      15   0  1932  992 1500 S  5.8  0.2   0:50.01 enbd-client
> >  3082 root      15   0  1932  992 1500 S  5.8  0.2   0:48.92 enbd-client
> (CPU usage is normal when I/O is idle)

Peter