[Linux-ha-dev] Cluster Test Harness
lmb at suse.de
Thu Jun 3 04:44:33 MDT 2004
Lars (Ellenberg, not me ;) has written a CTH for drbd (which it tests
quite well so far). However, it struck me the design is quite elegant,
and that it is very applicable to testing heartbeat too (in particular,
the N-node version).
The basic concept is that it models the components of the cluster - the
network links, the nodes, the disks, but also potentially the various
software layers themselves, and the dependencies between them. (This
sounds complicated, but in practice it is not, for it's normally just a
(Right now, it deals with physical components, but I can easily see it
extended to go in a do a kill -9 -1 to simulate a software crash on a
node etc too, this is trivial.)
Then it goes in an hits a component on the head randomly, and will
eventually heal it before poking the next one.
During all of this time, it can run various tests against the cluster
from the outside. (Right now, as it's just doing drbd testing, it's a IO
benchmark on top of drbd, but it could easily be something which
verifies that the cluster is still exporting the services to the
clients after having responded to the failure, and also measure the time
needed for this etc).
It can also be scripted (instead of random runs) to reproduce a specific
failure scenario for regression testing.
I _really_ like this approach, it's a great blackbox model. We shoot at
it and see whether it still lives.
I think this approach is somewhat more powerful than the current CTS
one; most tests could likely be migrated into the CTH model. Not all of
them, maybe; it's possible that CTS has slightly different goals,
exactly because it has some more in-depth knowledge about heartbeat.
Lars, do you agree with this? Alan? Andrew, is this something you could
see useful for testing the CRM?
Lars Marowsky-Brée <lmb at suse.de>
High Availability & Clustering \ ever tried. ever failed. no matter.
SUSE Labs | try again. fail again. fail better.
Research & Development, SUSE LINUX AG \ -- Samuel Beckett
More information about the Linux-HA-Dev