[Linux-ha-dev] "Hot Failover" status/plans

[ben.carlyle at wrsa.com.au]

linux-ha-dev-bounces at lists.linux-ha.org wrote on 04/08/2007 09:22:59 PM:
> ben.carlyle at wrsa.com.au wrote:
> > I'm currently reviewing technology in the high availability area for
> > applicability to replace an existing proprietary system.
...
> > The brief of the existing software is to provide n-way redundancy for
> > applications/resources that need to be highly available, with hot
> > failover. The existing system takes a fairly simple approach. It
> > requires prior configuration of exactly where all resources will be
> > running. It runs all replicas all the time, and tells one replica to 
be
> > the controller while others are told not to be the controller.
> I'm not sure which "existing software" you're talking about.  This
> doesn't sound like Linux-HA 2.x...

Sorry... this is the existing proprietary system being used in-house. I am 
performing a semi-regular review as to which (if any) of our properietary 
systems could be replaced with open technology. Linux HA has been at the 
top of the list of candidates for replacing this particular system for 
some time.

> > Is there an established mechanism or approach to providing hot 
failover
> > using existing LinuxHA technology?
> > If not, what do the plans towards this goal look like? For example, is
> > there a clear idea as to how resources managed in this way are told
> > about decision of control?
> Yes.  Look at the master/slave resources.  Otherwise called multi-state
> resources.

Thanks.. I think it will take me some time to digest, but this sounds 
plausible. If I understand correctly, multistate resources allow me to:
* Specify which nodes in the system are legal for the resource to run on
* Specify how many instances of a particular resource will be running 
system-wide
* Specify that there will be at most one instance per node
* Specify the maximum number of instances in state 1 (as compared to state 
2)
In particular, that I can say:
* Run at least three replicas across these six nodes
* Make sure one them is in state 1 (master), and the others are in state 2 
(slave)
I have to write a resource agent (a shell script, I think) that is able to 
tell linux-ha whether the particular instance is in 
active/inactive/master/slave/etc state at any time. linux-ha will call the 
script with promote or demote commands to request transitions between 
state 1 and state 2.

This all sounds like it might be roughly in-line with the functionality 
that is a candidate for replacement. I think I would have to work through 
some of the error cases, and understand better how failures of individual 
instances are detected and dealt with.

> > Is it possible to set up some sort of "mega cluster" with Linux HA? By
> > this I mean that a resource is shared across multiple geographical
> > locations, each with a mini-cluster. Each mini-cluster would elect a
> > leader, which exchanges heartbeat messages with the leader of the 
other
> > cluster. Within each mini-cluster, heartbeats can be exchanged n-ways.
...
> For example, for synchronous replication, typically ping times between
> sites need to be on the order of a millisecond.  Long distance
> geographic replication with asynchronsous replication relaxes this to a
> few tens of milliseconds, and bandwidth on the order of 10s to hundreds
> of megabytes/second.  Heartbeat's needs are for latency on the order of
> hundreds or thousands of milliseconds, and bandwidth on the order of a
> half a megabit peak.

My case is not as extreme as this. Data replication is asynchronous, "as 
recent as possible". I am mostly dealing with soft real-time, rather than 
strict transactional processing. My needs for data replication are on a 
similar scale to heartbeat's needs.

> It's my belief that what we have here will work well in typical DR
> situations.  The special support we have for DR applications consists of
> three things:
>    ability to distinguish which site a node is in
>    ability to give each node a different weight in quorum voting
>    quorum daemon - to act as a long-distance quorum tiebreaker
> Now, if you want to split an application across sites, in an
> active-active way, the application itself has to be prepared to manage
> that situation.  Very few are prepared to do that.

Mine are, which I can understand is a fairly unique situation. The 
super-cluster thing is not something implemented in my current technology, 
rather something I have been dreaming up for a while to try and minimise 
"base network load". I have been working on a variety of protocol areas to 
try and keep this base load down as a percentage of overall communication. 
It is a nice-to-have for me at the moment.

Benjamin.