How do I set up a resource as "secondary only"?
Wallwork, Nathan
Wallwork, Nathan" <nwallwo@pnm.com
Tue, 15 Oct 2002 12:00:00 -0600 (MDT)
On Tue, 15 Oct 2002, Lars Marowsky-Bree wrote:
LMB> This is not currently supported cleanly.
That's what I was assuming. I figured I'd bring it up and
see if it's worth working on, or worth developing either a
less ugly solution or a generally accepted ugly solution.
LMB> As far as ugly hacks go, this is probably the cleanest:
LMB>
LMB> - Write a resource script to touch a "no slap" file somewhere in the
LMB> filesystem where it keeps track of the status of the slapd as this one was
LMB> started. Have it remove this file when stopped and restore the slapd (ie,
LMB> maybe start it) state written inside it.
LMB>
LMB> - Add a hook to the slapd resource script to make it not start if the file is
LMB> present, but instead write a "please start me" into it.
LMB>
LMB> Maybe you can even unconditionally "negate" the action; ie, if slurpd is
LMB> started, stop slapd and vice-versa, that would get rid of the house-keeping in
LMB> the state file. Remember to also negate the monitor status.
LMB>
LMB> In this case, I'd call the resource type simply "negate", and you could use it
LMB> like such:
LMB>
LMB> node1 192.168.1.1 negate::slapd slurpd
LMB> node2 192.168.2.2 slapd
Running slapd is more important than running slurpd. I'd _need_
to run slapd on the primary node, and I'd _like_ to run slurpd
on the secondary node if there is a secondary node, so I think it
would look more like this:
node1 negate::slurpd 192.168.1.1 slapd
node2 slurpd 192.168.2.2
This still doesn't look right. It is okay at startup, it is okay
when node1 fails over to node2, because slurpd is already running,
but when node2 then fails back to node1, node1 will attempt to run
both slurpd and negate::slurpd. If the slurpd resource looses that
race condition, the slurpd process would be running on the same
node as slapd.
Also, while the ordering of resource startup is predictable, the
timing of the startup/shutdown of a particular resource isn't
strictly reliable. Ideally, I'd like to be able to guarantee
that slurpd has stopped before slapd is started.
By its nature (as a wrapper for another script) negate would
take longer to run than the script it wraps, or a similar
script. Suppose 'slapd start', 'slapd stop', 'slurpd start'
and 'slurpd stop' all take the same amount of time on an
unloaded system. If 'negate::slurpd start' calls 'slurpd stop',
it will take longer to run than 'slapd start'. This would
imply that slapd is likely to start before slurpd stops, and
that's what we want to avoid.
On Tue, 15 Oct 2002, Ragnar Kj?rstad wrote:
RK> I think the way it _should_ be implemented is a resource-manager that
RK> not only knows about dependancies but also conflicts.
RK> Then you could define that you wanted to run both slapd and slurpd, but
RK> they conflict so the RM would have to run them on different servers.
RK> This problem is not specific to ldap - I think DRDB and clustered
RK> sql-servers have the exact same issue.
This would also need some way to say "Slapd is more important than slurpd,
so if there is only one node, run slapd and not slurpd.".
RK> Anyway, I guess that's not what you wanted to hear.
Yeah, but it's what I expected.
RK> I guess you could hack it with a "reverse-script" - that is a resource
RK> script that started the resource when you asked it to stop and stopped
RK> when you asked it to start? It's messy, but it might work.
Two votes (of two) for that solution.
Sounds like that may be the ugly hack that wins, if we can iron out
the logic.
Suppose we have node1 and node2, and resources A and B. We want A to
run on the primary node. We'd like B to run on the secondary node
if there is a secondary node.
Lars suggest something like this:
node1 A negate::B
node2 B
This doesn't work so well:
At startup (assuming node1 starts first), node1 has A, node2 has B,
and everything looks good.
node1 now holds two resources (A and negate::B).
node2 now holds one resource (B).
node1 enters standby.
node1 runs 'A stop' and 'negate::B stop'
node1 stops A, which is good.
node1 starts B, which is good.
node2 runs 'A start' and 'negate::B start'
node1 starts A, which is good.
node1 stops B, which is good.
node2 now holds all three resources (A, negate::B, and B).
node2 enters standby.
node1 runs 'A start', 'B start', and 'negate::B start'
node1 starts A, which is good.
node1 has a race condition between
'B start' and 'negate::B start'
'negate::B start' probably looses
the race, so node1 probably
starts B then stops B.
node2 runs 'A stop', 'B stop, and 'negate::B stop'
node1 stops A, which is good.
node2 has a race condition between
'B stop' and 'negate::B stop'
'negate::B stop' probably looses
the race, so node2 probably
stops B then starts B.
The race conditions suggest we set things up incorrectly,
that B and negate::B shouldn't both be resources, but
how else to get B started on node2 in the initial state?
We could have a script called 'standby' instead of 'negate',
with logic like this:
if 'start':
if 'heartbeat is starting up as standby node':
start resource
if 'heartbeat is starting as primary node OR
heartbeat is entering acquiring resources as it
becomes the primary mode':
stop resource
if 'stop':
if 'heartbeat is shutting down':
stop resource
if 'heartbeat is starting up as the secondary node OR
heartbeat is entering standby mode':
start resource
but I don't know how I'd code the second-level if statements
for this logic, because heartbeat has overloaded the meanings
of 'start' and 'stop'.
Perhaps that's the key to finding a good solution.
If we could remove that overloading, we could still set up scripts
so that case statements fall through and exhibit the same behavior
they have now, but this would allow more sophisticated scripts to
differentiate between 'shutdown' and 'release', and
differentiate between 'startup', and 'acquire'.
I think we could make this work if we had heartbeat call
'startup', and 'acquire' instead of just 'start' and call
'shutdown' and 'release' instead of just 'stop'.
Then the typical resource script would look like this:
startup)
acquire)
start)
# start resources
;;
shutdown)
release)
stop)
# stop resources
;;
A standby resource script (with release and acquire reversed)
would look like this:
startup)
release)
start)
# start resources
# [...]
;;
shutdown)
acquire)
stop)
# stop resources
# [...]
;;
status)
# Inverted status reporting....
# [...]
;;
Suppose we'd changed heartbeat and set up such a standby script.
Then we could set up haresources with
node1 A
node2 standby::B
We'd want this behaviour (and get it, I think):
At startup, node1 gets A, node2 gets B.
node1 now holds one resource (A).
node2 now holds one resource (standby::B).
node1 enters standby:
node1 runs 'A release' and 'standby::B release'
node1 stops A and starts B.
node2 runs 'A acquire' and 'standby::B acquire'
node2 starts A and stops B.
node2 now holds all resources (A and standby::B).
node2 enters standby.
node1 runs 'A acquire' and 'standby::B acquire'
node1 starts A and stops B.
node2 runs 'A release' and 'standby::B release'
node1 stops A and starts B.
This doesn't address the timing issues (wanting the stop to
occur before the start), but I could get around that by adding
a 'sleep 2' or something in front of the commands that start
A and B, if I need to.
Would this make sense?
Would it work?