[Linux-ha-dev] Re: [Linux-ha-cvs] riloe plugin commit

[Lars Marowsky-Bree]

On 2004-10-02T01:46:48,
   Chris Babcock <cbabcock at luthresearch.com> said:

> If the IO path to the disks used to run stonith (or other fencing) causes
> blocking, then you are probably SCREWED because your node taking over
> primary service is blocking on a routine operation and most likely can't
> support real services anyway.

Well, there's three points which can be raised in response here; I know
them well because the above is the argument I usually voice ;-)

First, even if it's not blocked for good and will succeed, the need to
swap may cause a noticeable delay on the order of several hundred ms or
up to seconds if on a bad day. This affects the cluster recovery time
and can make the difference between 6 or 7 nines...

Second, the node asked to perform the fencing operation may not actually
be the one which is later asked to start any of the services (relevant
in a >2 node cluster).

Third, the load burst which caused the STONITH delay may actually have
been related to the need to fence the node; ie, buffers filling up
because the cluster file system can't coordinate the access to the disk
until the node has been fenced for good etc.

These are all corner-cases, and for most if not all scenarios, executing
an external script is far good enough. But preserving the ability to
lock specialized stonith subsystems into memory is also valuable.

(However, these then better should meet these high requirements, or else
the whole point is moot.)

> Idea...  Perhaps a "watchdog" like system could be used to monitor
> stonith/fencing operations and take action on stonith/fence failure or
> blockage?  ( Perhaps something like retry, retry, reboot local node... )

The STONITH subsystem already monitors the fencing success; right now it
will eternally retry, and the new smart fencing daemon subsystem (see
the Wiki) will have the ability to even retry the fence operation from
another node.

> It seems to me that there is some significant value to having a
> stonith/fencing interface that can be used without programming the
> modules in C.

There's absolutely no doubt about this one. The external module already
goes quite a bit into that direction, and because it's one of my pet
peeves ;), I'll enhance it somewhat more.

But there's also value in being able to have it locked into memory.


Sincerely,
    Lars Marowsky-Brée <lmb at suse.de>

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX AG - A Novell company