[Linux-ha-dev] serializing actions of resource scripts

[Sebastian Reitenbach]

Hi,

I try to add a bit memory management monitoring of services within the Xen 
resource, as I am usually interested in the services that the Xen resource 
provides and not whether Xen runs or not. I already got a lot of valuable 
feedback from Dejan, the state can be tracked here:
http://developerbugs.linux-foundation.org/show_bug.cgi?id=1778

The problems right now are: 
1) how to handle paused domU's
2) how to handle memory management

The first is a bit work, but easy, before every action check in which 
condition the domU is, e.g. running/stopped/paused/... and then handle that 
situation accordingly.

The second one is a bit trickier.
Right now I have the Xen script working this way on startup/stop of the 
domU:
1. check how many memory is available
2. subtract the reserved memory for the dom0
3. check how many domains will after that action running, divide the result 
from step two with that number
4. set xm mem-set the memory for all virtual machines

This is a very basic way to give all virtual machines more or less the same 
amount of memory. Nevertheless, in case more than one Xen resource is 
starting or stopping, this behavior is prone to race conditions, and I 
already have seen it failing. 
My workaround was to create orders, with symmetrical=no and score=0 for all 
Xen resources, so that only one can start at a time.
Dejan suggested to add locking to the Xen resource script, but I fear that 
this will lead to new errors, e.g. assume the default-action-timeout=30s
and you have 4 Xen resources, and all four will start up at the same time, 
then the first, will aquire the lock, the rest is waiting. Maybe everything 
will work for the second Xen resource too. But I assume then the startup of 
3 and 4 will fail, because the default-action-timeout was hit.

Is it possible serialize actions of a given type of resource?
e.g. the Xen resource could be marked as serialization needed, that means, 
in case there are multiple Xen resources in a cluster, actions to these 
resources are not allowed to happen at the same time?
So that not the Xen resource script would be responsible for locking and 
allowing/disallowing actions to itself, but the CRMD is responsible. 
This could be made more fine-grained, in case you say, it is not allowed to 
have action for resource Xen at the same time on the same node, but it is 
allowed to have multiple actions to the Xen resource in the cluster.
Then the LRMD could be made responsible to serialize the actions.

Does that makes sense to anyone, or maybe is that already possible?

kind regards
Sebastian