[Linux-HA] Adding third node turns all resources unmanaged

[Andrew Beekhof]

On Mon, Jul 28, 2008 at 08:56, Gerard Petersen <gerard at gp-net.nl> wrote:
> Dear Andrew,
>
> Thanx for your response.
>
> I see two options/conclusions on which I would like your feedback:
>
> - Enable stonith so the attempt to start the resources on the third node,
> shall be 'naturally' disabled and therefore moved back to the first two
> nodes by the cluster software.
>
> - Install Xen (and drbd) on the third node, so the cluster software get's a
> change to initialise some commands and get a proper answer to see that the
> resources don't belong here.

I think you missed the most preferable option... fix the RA to return
OCF_NOT_INSTALLED in such cases and send us a patch :-)

>
>
> Kind regards,
>
> Gerard.
>
> Andrew Beekhof wrote:
>>
>> On Thu, Jul 24, 2008 at 16:29, Gerard Petersen <gerard at gp-net.nl> wrote:
>>
>>>
>>> Hi all,
>>>
>>> I'm trying to add a third node to a two node working cluster
>>> withresources
>>> in the form of mirrored Xen (and underlying drbd) virtual servers. The
>>> two
>>> node setup works great and as expected. (On failure, the drbd mirrors
>>> switch master/slave roles, XenU's migrate automatically, etc). The goal
>>> is
>>> to manually spread master slave combinations of the XenU's over the three
>>> pysical nodes.
>>>
>>> The third node is already added to heartbeat config, and in standby mode.
>>> We have contraints in place (full log and config will follow), that work
>>> with the +INF, 'zero' and -INF values, respectively as Master location,
>>> Slave location and  'Never' location constraints.
>>>
>>> When we take the third node online, where the current XenU's according to
>>> the constraints are not allowed, the resources somehow all are moved to
>>> the third node, where no xen or drbd is present yet. It seems some of the
>>> constraints are completely ignored. We have tried this, among other
>>> things, with the symmetric_cluster value True and False, but no luck.
>>>
>>> Furthermore the log shows that the resources become 'to active', and
>>> after
>>> that they become unmanaged.
>>>
>>
>> When a new node joins the cluster, we check to see if its running any
>> of the cluster resources.
>> These checks occur regardless of any location constraints (precisely
>> so that we can enforce them for you).
>>
>> What can happen however, is that these checks may fail.
>> Sometimes they fail because the service was unexpectedly found to be
>> active on the node.
>> Sometimes its because the resource agent (or the software it tries to
>> talk to) isnt installed.
>>
>> in your case, it seems the RA is misbehaving and incorrectly telling
>> the cluster that the resources are active
>> eg.
>>             <lrm_rsc_op id="server128_monitor_0" operation="monitor"
>> crm-debug-origin="build_active_RAs"
>> transition_key="15:10:c195d63f-e91f-4162-8454-f6dde2c71ef1"
>> transition_magic="0:0;15:10:c195d63f-e91f-4162-8454-f6dde2c71ef1"
>> call_id="6" crm_feature_set="2.0" rc_code="0" op_status="0"
>> interval="0" op_digest="78122685b830dcb8197c65561be6d6a5"/>
>>
>> rc_code="0" being the relevant piece of information
>>
>> The cluster then thinks that the service is active on more than one
>> node and tries to recover.
>> But the RA then compounds the initial problem by failing to stop the
>> service:
>>
>>             <lrm_rsc_op id="server128_stop_0" operation="stop"
>> crm-debug-origin="build_active_RAs"
>> transition_key="25:11:c195d63f-e91f-4162-8454-f6dde2c71ef1"
>> transition_magic="0:1;25:11:c195d63f-e91f-4162-8454-f6dde2c71ef1"
>> call_id="12" crm_feature_set="2.0" rc_code="1" op_status="0"
>> interval="0" op_digest="78122685b830dcb8197c65561be6d6a5"/>
>>
>> again, rc_code="1" being the part indicating failure.
>>
>> at which point the cluster can do nothing (since stonith is disabled)
>>
>>
>>>
>>> Some notes to clearify the setup (and make the log more readable):
>>>
>>> We run heartbeat version 2.1.3-5~bpo40+1 from debian backports. At the
>>> time of testing, one node was still on 2.1.3-2~bpo40+1.
>>>
>>> Fysical nodes:
>>> server010 (still to be added)
>>> server011
>>> server012
>>>
>>> Virtual servers (the resources):
>>> server128 - server133
>>>
>>> All resources have contraints allowing a primary role on server011 and
>>> secondary role on server012 (or viceversa). And are not allowed on
>>> server010.
>>>
>>> # Attached files are:
>>>
>>> - cleancib.xml
>>> The one we started of with.
>>>
>>> - fullcib.xml
>>> The most recent full dump (with counters etc. added by the cluster
>>> software itself).
>>>
>>> - syslog.clusterlog.080722.full(.tgz)
>>> A cleaned up syslog wherein, with different values for symmetric_cluster,
>>> the trail can be followed how all resources became to active, and end up
>>> unmanaged on server010
>>>
>>> - syslog.clusterlog.080722.part(.tgz)
>>> A stripped version of the previous one with only one trail, hopefully
>>> isolation enough information, for easier analyses.
>>>
>>> It looks like the behaviour deviates from what the docs describe in
>>> relation to the symmetric_cluster directive, or it's just a very ugly
>>> typo
>>> somewhere .. :-)
>>>
>>> I sincerely hope somebody can pinpoint the weakspot.
>>>
>>> Thanx a lot!!
>>>
>>>
>>> Kind regards,
>>>
>>> Gerard.
>>>
>>>
>>>
>>> --
>>> ~
>>> ~
>>> :wq!
>>> _______________________________________________
>>> Linux-HA mailing list
>>> Linux-HA at lists.linux-ha.org
>>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>>> See also: http://linux-ha.org/ReportingProblems
>>>
>>>
>>
>> _______________________________________________
>> Linux-HA mailing list
>> Linux-HA at lists.linux-ha.org
>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>> See also: http://linux-ha.org/ReportingProblems
>>
>>
>
>
> --
>>>>
>>>> urls
>
> {'fun':  'www.zonderbroodje.nl',  'tech':  'www.gp-net.nl'}
>
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems
>