[Linux-ha-dev] no failover after stonith
Peter Kruse
pk at q-leap.com
Tue Nov 8 05:59:46 MST 2005
Hello,
failover is not initiated after a stonith operation.
I forced a monitor error and put a "return 1" in
a stop operation to force a stonith. That succeeded.
But the resource group is not started on the second node.
attached are the syslog of the second node, output
of "cibadmin -Q" and here is the output of "crm_mon -1":
============
Last updated: Tue Nov 8 13:55:43 2005
Current DC: ha-test-2 (14267cb7-ea21-4a91-a1af-b4466782ec43)
2 Nodes configured.
2 Resources configured.
============
Node: ha-test-1 (026c6f3c-65d5-40ae-855f-234c1a3c6fcc): OFFLINE
Node: ha-test-2 (14267cb7-ea21-4a91-a1af-b4466782ec43): online
Clone Set: fence1
fence1:apc1:0 (stonith:apcmastersnmp): ha-test-2
fence1:apc1:1 (stonith:apcmastersnmp): NOT ACTIVE
Resource Group: rg1
rg1:drbd1 (q-leap::ocf:DRBD): ha-test-1
rg1:fpbs1 (q-leap::ocf:filesystem): NOT ACTIVE
rg1:IPaddr1 (q-leap::ocf:IP_address): NOT ACTIVE
rg1:IPaddr3 (q-leap::ocf:IP_address): NOT ACTIVE
rg1:pbs1 (q-leap::ocf:openpbs): NOT ACTIVE
This is with current CVS, it worked with a CVS Version
one week ago.
Peter
-------------- next part --------------
<cib_fragment>
<cib generated="true" admin_epoch="0" have_quorum="true" num_peers="2" origin="ha-test-2" last_written="Tue Nov 8 13:49:09 2005" ccm_transition="3" cib_feature_revision="1.2" crm_feature_set="1.0.3" debug_source="finalize_join" dc_uuid="14267cb7-ea21-4a91-a1af-b4466782ec43" epoch="4" num_updates="97">
<configuration>
<crm_config>
<nvpair id="transition_idle_timeout" name="transition_idle_timeout" value="60s"/>
<nvpair id="symmetric_cluster" name="symmetric_cluster" value="true"/>
<nvpair id="no_quorum_policy" name="no_quorum_policy" value="ignore"/>
<nvpair id="stonith_enabled" name="stonith_enabled" value="true"/>
<nvpair id="suppress_cib_writes" name="suppress_cib_writes" value="false"/>
<nvpair id="default_resource_stickiness" name="default_resource_stickiness" value="INFINITY"/>
</crm_config>
<nodes>
<node uname="ha-test-1" type="member" id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
<nvpair id="ha-test-1-bond0" name="ha-test-1-bond0" value="up"/>
<nvpair id="ha-test-1-eth1" name="ha-test-1-eth1" value="up"/>
<nvpair id="ha-test-2-bond0" name="ha-test-2-bond0" value="up"/>
<nvpair id="ha-test-2-eth1" name="ha-test-2-eth1" value="down"/>
<nvpair id="drbd1-RestartCount" name="drbd1-RestartCount" value="ha-test-1-0"/>
<nvpair id="fpbs1-RestartCount" name="fpbs1-RestartCount" value="ha-test-1-0"/>
<nvpair id="IPaddr1-RestartCount" name="IPaddr1-RestartCount" value="ha-test-1-0"/>
<nvpair name="IPaddr3-RestartCount" id="IPaddr3-RestartCount" value="ha-test-1-0"/>
<nvpair id="pbs1-RestartCount" name="pbs1-RestartCount" value="ha-test-1-0"/>
</node>
<node id="14267cb7-ea21-4a91-a1af-b4466782ec43" uname="ha-test-2" type="member"/>
</nodes>
<resources>
<clone id="fence1">
<instance_attributes>
<attributes>
<nvpair id="max_fencing_1" name="clone_max" value="2"/>
<nvpair id="max_node_fencing_1" name="clone_node_max" value="1"/>
</attributes>
</instance_attributes>
<primitive id="apc1" class="stonith" type="apcmastersnmp">
<operations>
<op id="apc1-monitor" name="monitor" interval="20s" timeout="40s" prereq="nothing"/>
<op id="apc1-start" name="start" timeout="20s" prereq="nothing"/>
</operations>
<instance_attributes>
<attributes>
<nvpair id="apc1-ipaddr" name="ipaddr" value="apc"/>
<nvpair id="apc1-port" name="port" value="161"/>
<nvpair id="apc1-community" name="community" value="private"/>
</attributes>
</instance_attributes>
</primitive>
</clone>
<group resource_stickiness="INFINITY" id="rg1" is_managed="true">
<primitive id="drbd1" class="ocf" type="DRBD" on_stopfail="fence" provider="q-leap">
<instance_attributes>
<attributes>
<nvpair id="drbd1_device" name="Device" value="drbd1"/>
<nvpair id="drbd1_restart" name="RestartCount" value="0"/>
<nvpair id="drbd1_maxrestart" name="MaxRestartCount" value="1"/>
</attributes>
</instance_attributes>
<operations>
<op id="drbd1_monitor" name="monitor" interval="30s" timeout="30s"/>
<op id="drbd1_start" name="start" timeout="30s"/>
<op id="drbd1_stop" on_fail="fence" name="stop" timeout="30s"/>
</operations>
</primitive>
<primitive id="fpbs1" class="ocf" type="filesystem" on_stopfail="fence" provider="q-leap">
<instance_attributes>
<attributes>
<nvpair id="fpbs1_device" name="Device" value="/dev/drbd/1"/>
<nvpair id="fpbs1_dir" name="MountPoint" value="/c/pbs"/>
<nvpair id="fpbs1_fstype" name="FSType" value="reiserfs"/>
<nvpair id="fpbs1_opt" name="mount_options" value="rw"/>
<nvpair id="fpbs1_restart" name="RestartCount" value="0"/>
<nvpair id="fpbs1_maxrestart" name="MaxRestartCount" value="1"/>
</attributes>
</instance_attributes>
<operations>
<op id="fpbs1_monitor" name="monitor" interval="30s" timeout="10s"/>
<op id="fpbs1_start" name="start" timeout="10s"/>
<op id="fpbs1_stop" on_fail="fence" name="stop" timeout="10s"/>
</operations>
</primitive>
<primitive class="ocf" id="IPaddr1" on_stopfail="fence" provider="q-leap" type="IP_address">
<instance_attributes>
<attributes>
<nvpair id="IPaddr1_ip" name="ip" value="192.168.53.169"/>
<nvpair id="IPaddr1_nic" name="nic" value="bond0"/>
<nvpair id="IPaddr1_restart" name="RestartCount" value="0"/>
<nvpair id="IPaddr1_maxrestart" name="MaxRestartCount" value="1"/>
</attributes>
</instance_attributes>
<operations>
<op id="IPaddr1_monitor" name="monitor" interval="30s" timeout="400s"/>
<op id="IPaddr1_start" name="start" timeout="10s"/>
<op id="IPaddr1_stop" on_fail="fence" name="stop" timeout="10s"/>
</operations>
</primitive>
<primitive class="ocf" id="IPaddr3" on_stopfail="fence" provider="q-leap" type="IP_address">
<instance_attributes>
<attributes>
<nvpair id="IPaddr3_ip" name="ip" value="192.168.254.3"/>
<nvpair id="IPaddr3_nic" name="nic" value="eth1"/>
<nvpair id="IPaddr3_restart" name="RestartCount" value="0"/>
<nvpair id="IPaddr3_maxrestart" name="MaxRestartCount" value="1"/>
<nvpair id="IPaddr3_PointToPoint" name="PointToPoint" value="true"/>
</attributes>
</instance_attributes>
<operations>
<op id="IPaddr3_monitor" name="monitor" interval="30s" timeout="400s"/>
<op id="IPaddr3_start" name="start" timeout="10s"/>
<op id="IPaddr3_stop" on_fail="fence" name="stop" timeout="10s"/>
</operations>
</primitive>
<primitive id="pbs1" class="ocf" type="openpbs" on_stopfail="fence" provider="q-leap">
<instance_attributes>
<attributes>
<nvpair id="pbs1_sched" name="Scheduler" value="maui"/>
<nvpair id="pbs1_restart" name="RestartCount" value="0"/>
<nvpair id="pbs1_maxrestart" name="MaxRestartCount" value="1"/>
</attributes>
</instance_attributes>
<operations>
<op id="pbs1_monitor" name="monitor" interval="30s" timeout="10s"/>
<op id="pbs1_start" name="start" timeout="10s"/>
<op id="pbs1_stop" on_fail="fence" name="stop" timeout="10s"/>
</operations>
</primitive>
</group>
</resources>
<constraints/>
</configuration>
<status>
<node_state uname="ha-test-1" crmd="online" in_ccm="false" ha="active" expected="member" origin="ghash_update_cib_node" join="down" id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
<lrm>
<lrm_resources>
<lrm_resource type="apcmastersnmp" class="stonith" id="fence1:apc1:0">
<lrm_rsc_op operation="start" op_status="0" call_id="8" rc_code="0" transition_key="1:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;1:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" id="fence1:apc1:0_start_0" origin="build_active_RAs">
<parameters clone="0" clone_max="2" community="private" timeout="20000" ipaddr="apc" crm_feature_set="1.0.3" port="161" clone_node_max="1"/>
</lrm_rsc_op>
<lrm_rsc_op operation="monitor" transition_key="1:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;1:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" call_id="9" rc_code="0" op_status="0" id="fence1:apc1:0_monitor_20000" origin="build_active_RAs">
<parameters clone="0" clone_max="2" community="private" timeout="40000" ipaddr="apc" crm_feature_set="1.0.3" interval="20000" port="161" clone_node_max="1"/>
</lrm_rsc_op>
<lrm_rsc_op id="fence1:apc1:0_stop_0" operation="stop" origin="do_update_resource" transition_key="11:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;11:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="37" rc_code="0" op_status="0"/>
</lrm_resource>
<lrm_resource type="openpbs" class="ocf" provider="q-leap" rsc_state="starting" id="rg1:pbs1">
<lrm_rsc_op operation="monitor" transition_key="0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="7" rc_code="7" id="rg1:pbs1_monitor_0" origin="build_active_RAs" transition_magic="4:7;0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" op_status="4">
<parameters is_managed="false" on_stopfail="fence" resource_stickiness="INFINITY" te-target-rc="7" Scheduler="maui" MaxRestartCount="1" lrm-is-probe="true" crm_feature_set="1.0.3" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op operation="start" rsc_state="running" transition_key="8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:pbs1_start_0" op_status="0" call_id="25" rc_code="0" origin="do_update_resource" transition_magic="0:0;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218">
<parameters is_managed="true" on_stopfail="fence" resource_stickiness="INFINITY" timeout="10000" Scheduler="maui" MaxRestartCount="1" crm_feature_set="1.0.3" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op id="rg1:pbs1_monitor_30000" operation="monitor" origin="do_update_resource" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" call_id="19" rc_code="0" op_status="0">
<parameters is_managed="true" on_stopfail="fence" resource_stickiness="INFINITY" timeout="10000" Scheduler="maui" MaxRestartCount="1" crm_feature_set="1.0.3" interval="30000" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op operation="stop" origin="do_update_resource" rsc_state="stopped" rc_code="0" op_status="0" id="rg1:pbs1_stop_0" transition_key="9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="26"/>
</lrm_resource>
<lrm_resource type="filesystem" class="ocf" provider="q-leap" rsc_state="starting" id="rg1:fpbs1">
<lrm_rsc_op operation="monitor" transition_key="0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="4" rc_code="7" id="rg1:fpbs1_monitor_0" origin="build_active_RAs" transition_magic="4:7;0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" op_status="4">
<parameters is_managed="false" on_stopfail="fence" resource_stickiness="INFINITY" mount_options="rw" te-target-rc="7" MaxRestartCount="1" lrm-is-probe="true" Device="/dev/drbd/1" crm_feature_set="1.0.3" MountPoint="/c/pbs" FSType="reiserfs" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op operation="start" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:fpbs1_start_0" op_status="0" call_id="12" rc_code="0" origin="do_update_resource" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running">
<parameters is_managed="true" on_stopfail="fence" resource_stickiness="INFINITY" mount_options="rw" timeout="10000" MaxRestartCount="1" Device="/dev/drbd/1" crm_feature_set="1.0.3" MountPoint="/c/pbs" FSType="reiserfs" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op id="rg1:fpbs1_monitor_30000" operation="monitor" origin="do_update_resource" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" call_id="13" rc_code="0" op_status="0">
<parameters is_managed="true" on_stopfail="fence" resource_stickiness="INFINITY" mount_options="rw" timeout="10000" MaxRestartCount="1" Device="/dev/drbd/1" crm_feature_set="1.0.3" interval="30000" MountPoint="/c/pbs" FSType="reiserfs" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op id="rg1:fpbs1_stop_0" operation="stop" origin="do_update_resource" transition_key="10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="33" rc_code="0" op_status="0"/>
</lrm_resource>
<lrm_resource type="DRBD" class="ocf" provider="q-leap" rsc_state="starting" id="rg1:drbd1">
<lrm_rsc_op operation="monitor" transition_key="0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="3" rc_code="7" id="rg1:drbd1_monitor_0" origin="build_active_RAs" transition_magic="4:7;0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" op_status="4">
<parameters is_managed="false" on_stopfail="fence" resource_stickiness="INFINITY" te-target-rc="7" MaxRestartCount="1" lrm-is-probe="true" Device="drbd1" crm_feature_set="1.0.3" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op operation="start" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:drbd1_start_0" op_status="0" call_id="10" rc_code="0" origin="do_update_resource" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running">
<parameters is_managed="true" on_stopfail="fence" resource_stickiness="INFINITY" timeout="30000" MaxRestartCount="1" Device="drbd1" crm_feature_set="1.0.3" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op id="rg1:drbd1_monitor_30000" operation="monitor" origin="do_update_resource" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" call_id="11" rc_code="0" op_status="0">
<parameters is_managed="true" on_stopfail="fence" resource_stickiness="INFINITY" timeout="30000" MaxRestartCount="1" Device="drbd1" crm_feature_set="1.0.3" interval="30000" RestartCount="0"/>
</lrm_rsc_op>
<lrm_rsc_op id="rg1:drbd1_stop_0" operation="stop" origin="do_update_resource" transition_key="10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:1;10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stop_failed" call_id="35" rc_code="1" op_status="4"/>
</lrm_resource>
<lrm_resource type="IP_address" class="ocf" provider="q-leap" rsc_state="starting" id="rg1:IPaddr3">
<lrm_rsc_op operation="monitor" transition_key="0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="6" rc_code="7" id="rg1:IPaddr3_monitor_0" origin="build_active_RAs" transition_magic="4:7;0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" op_status="4">
<parameters is_managed="false" ip="192.168.254.3" PointToPoint="true" on_stopfail="fence" resource_stickiness="INFINITY" te-target-rc="7" MaxRestartCount="1" lrm-is-probe="true" crm_feature_set="1.0.3" RestartCount="0" nic="eth1"/>
</lrm_rsc_op>
<lrm_rsc_op operation="start" transition_key="9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:IPaddr3_start_0" op_status="4" call_id="27" rc_code="1" origin="do_update_resource" transition_magic="4:1;9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="start_failed">
<parameters is_managed="true" ip="192.168.254.3" PointToPoint="true" on_stopfail="fence" resource_stickiness="INFINITY" timeout="10000" MaxRestartCount="1" crm_feature_set="1.0.3" RestartCount="0" nic="eth1"/>
</lrm_rsc_op>
<lrm_rsc_op operation="monitor" origin="do_update_resource" transition_key="8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="24" id="rg1:IPaddr3_monitor_30000" transition_magic="4:7;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" rc_code="7" op_status="4">
<parameters is_managed="true" ip="192.168.254.3" PointToPoint="true" on_stopfail="fence" resource_stickiness="INFINITY" timeout="400000" MaxRestartCount="1" crm_feature_set="1.0.3" interval="30000" RestartCount="0" nic="eth1"/>
</lrm_rsc_op>
<lrm_rsc_op id="rg1:IPaddr3_stop_0" operation="stop" origin="do_update_resource" transition_key="10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="29" rc_code="0" op_status="0"/>
</lrm_resource>
<lrm_resource type="IP_address" class="ocf" provider="q-leap" rsc_state="starting" id="rg1:IPaddr1">
<lrm_rsc_op operation="monitor" transition_key="0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="5" rc_code="7" id="rg1:IPaddr1_monitor_0" origin="build_active_RAs" transition_magic="4:7;0:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" op_status="4">
<parameters is_managed="false" ip="192.168.53.169" on_stopfail="fence" resource_stickiness="INFINITY" te-target-rc="7" MaxRestartCount="1" lrm-is-probe="true" crm_feature_set="1.0.3" RestartCount="0" nic="bond0"/>
</lrm_rsc_op>
<lrm_rsc_op operation="start" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:IPaddr1_start_0" op_status="0" call_id="14" rc_code="0" origin="do_update_resource" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running">
<parameters is_managed="true" ip="192.168.53.169" on_stopfail="fence" resource_stickiness="INFINITY" timeout="10000" MaxRestartCount="1" crm_feature_set="1.0.3" RestartCount="0" nic="bond0"/>
</lrm_rsc_op>
<lrm_rsc_op id="rg1:IPaddr1_monitor_30000" operation="monitor" origin="do_update_resource" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" call_id="15" rc_code="0" op_status="0">
<parameters is_managed="true" ip="192.168.53.169" on_stopfail="fence" resource_stickiness="INFINITY" timeout="400000" MaxRestartCount="1" crm_feature_set="1.0.3" interval="30000" RestartCount="0" nic="bond0"/>
</lrm_rsc_op>
<lrm_rsc_op id="rg1:IPaddr1_stop_0" operation="stop" origin="do_update_resource" transition_key="10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="31" rc_code="0" op_status="0"/>
</lrm_resource>
</lrm_resources>
</lrm>
<transient_attributes id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
<instance_attributes id="crmd-transient--026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
<attributes>
<nvpair id="lrm-probe-026c6f3c-65d5-40ae-855f-234c1a3c6fcc" name="probe_complete" value="true"/>
</attributes>
</instance_attributes>
</transient_attributes>
</node_state>
<node_state uname="ha-test-2" in_ccm="true" crmd="online" origin="do_lrm_query" join="member" ha="active" expected="member" id="14267cb7-ea21-4a91-a1af-b4466782ec43">
<transient_attributes id="14267cb7-ea21-4a91-a1af-b4466782ec43">
<instance_attributes id="crmd-transient--14267cb7-ea21-4a91-a1af-b4466782ec43">
<attributes>
<nvpair id="lrm-probe-14267cb7-ea21-4a91-a1af-b4466782ec43" name="probe_complete" value="true"/>
</attributes>
</instance_attributes>
</transient_attributes>
<lrm>
<lrm_resources>
<lrm_resource id="rg1:pbs1" type="openpbs" class="ocf" provider="q-leap">
<lrm_rsc_op id="rg1:pbs1_monitor_0" operation="monitor" origin="build_active_RAs" transition_key="4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:7;4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" call_id="7" rc_code="7" op_status="4">
<parameters is_managed="false" on_stopfail="fence" resource_stickiness="INFINITY" te-target-rc="7" Scheduler="maui" MaxRestartCount="1" lrm-is-probe="true" crm_feature_set="1.0.3" RestartCount="0"/>
</lrm_rsc_op>
</lrm_resource>
<lrm_resource id="rg1:fpbs1" type="filesystem" class="ocf" provider="q-leap">
<lrm_rsc_op id="rg1:fpbs1_monitor_0" operation="monitor" origin="build_active_RAs" transition_key="4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:7;4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" call_id="4" rc_code="7" op_status="4">
<parameters is_managed="false" on_stopfail="fence" resource_stickiness="INFINITY" mount_options="rw" te-target-rc="7" MaxRestartCount="1" lrm-is-probe="true" Device="/dev/drbd/1" crm_feature_set="1.0.3" MountPoint="/c/pbs" FSType="reiserfs" RestartCount="0"/>
</lrm_rsc_op>
</lrm_resource>
<lrm_resource id="fence1:apc1:0" type="apcmastersnmp" class="stonith" rsc_state="starting">
<lrm_rsc_op id="fence1:apc1:0_monitor_0" operation="monitor" origin="build_active_RAs" transition_key="4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:7;4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" call_id="2" rc_code="7" op_status="4">
<parameters clone="0" clone_max="2" community="private" ipaddr="apc" te-target-rc="7" lrm-is-probe="true" crm_feature_set="1.0.3" port="161" clone_node_max="1"/>
</lrm_rsc_op>
<lrm_rsc_op id="fence1:apc1:0_start_0" operation="start" op_status="0" call_id="11" rc_code="0" origin="do_update_resource" transition_key="0:fd3ac6d8-2e07-4b4c-ae26-7923eaf5c964" transition_magic="0:0;0:fd3ac6d8-2e07-4b4c-ae26-7923eaf5c964" rsc_state="running">
<parameters clone="0" clone_max="2" community="private" timeout="20000" ipaddr="apc" crm_feature_set="1.0.3" port="161" clone_node_max="1"/>
</lrm_rsc_op>
<lrm_rsc_op id="fence1:apc1:0_monitor_20000" operation="monitor" origin="do_update_resource" transition_key="0:fd3ac6d8-2e07-4b4c-ae26-7923eaf5c964" transition_magic="0:0;0:fd3ac6d8-2e07-4b4c-ae26-7923eaf5c964" rsc_state="running" call_id="12" rc_code="0" op_status="0">
<parameters clone="0" clone_max="2" community="private" timeout="40000" ipaddr="apc" crm_feature_set="1.0.3" interval="20000" port="161" clone_node_max="1"/>
</lrm_rsc_op>
</lrm_resource>
<lrm_resource id="fence1:apc1:1" type="apcmastersnmp" class="stonith">
<lrm_rsc_op id="fence1:apc1:1_start_0" operation="start" origin="build_active_RAs" transition_key="5:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;5:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" call_id="9" rc_code="0" op_status="0">
<parameters clone="1" clone_max="2" community="private" timeout="20000" ipaddr="apc" crm_feature_set="1.0.3" port="161" clone_node_max="1"/>
</lrm_rsc_op>
<lrm_rsc_op id="fence1:apc1:1_monitor_20000" operation="monitor" origin="build_active_RAs" transition_key="6:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:7;6:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" call_id="10" rc_code="7" op_status="4">
<parameters clone="1" clone_max="2" community="private" timeout="40000" ipaddr="apc" crm_feature_set="1.0.3" interval="20000" port="161" clone_node_max="1"/>
</lrm_rsc_op>
</lrm_resource>
<lrm_resource id="rg1:drbd1" type="DRBD" class="ocf" provider="q-leap">
<lrm_rsc_op id="rg1:drbd1_monitor_0" operation="monitor" origin="build_active_RAs" transition_key="4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:7;4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" call_id="3" rc_code="7" op_status="4">
<parameters is_managed="false" on_stopfail="fence" resource_stickiness="INFINITY" te-target-rc="7" MaxRestartCount="1" lrm-is-probe="true" Device="drbd1" crm_feature_set="1.0.3" RestartCount="0"/>
</lrm_rsc_op>
</lrm_resource>
<lrm_resource id="rg1:IPaddr1" type="IP_address" class="ocf" provider="q-leap">
<lrm_rsc_op id="rg1:IPaddr1_monitor_0" operation="monitor" origin="build_active_RAs" transition_key="4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:7;4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" call_id="5" rc_code="7" op_status="4">
<parameters is_managed="false" ip="192.168.53.169" on_stopfail="fence" resource_stickiness="INFINITY" te-target-rc="7" MaxRestartCount="1" lrm-is-probe="true" crm_feature_set="1.0.3" RestartCount="0" nic="bond0"/>
</lrm_rsc_op>
</lrm_resource>
<lrm_resource id="rg1:IPaddr3" type="IP_address" class="ocf" provider="q-leap">
<lrm_rsc_op id="rg1:IPaddr3_monitor_0" operation="monitor" origin="build_active_RAs" transition_key="4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:7;4:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" call_id="6" rc_code="7" op_status="4">
<parameters is_managed="false" ip="192.168.254.3" PointToPoint="true" on_stopfail="fence" resource_stickiness="INFINITY" te-target-rc="7" MaxRestartCount="1" lrm-is-probe="true" crm_feature_set="1.0.3" RestartCount="0" nic="eth1"/>
</lrm_rsc_op>
</lrm_resource>
</lrm_resources>
</lrm>
</node_state>
</status>
</cib>
</cib_fragment>
-------------- next part --------------
Nov 8 13:45:07 ha-test-2 syslogd 1.4.1#17: restart.
Nov 8 13:45:07 ha-test-2 stonithd: [1329]: debug: SO_PEERCRED returned [1916, (0:0)]
Nov 8 13:45:07 ha-test-2 stonithd: [1329]: debug: Verifying authentication: cred.uid=0 cred.gid=0
Nov 8 13:45:07 ha-test-2 lrmd: [1916]: debug: Will send the stonith RA operation to stonithd: apcmastersnmp monitor
Nov 8 13:45:07 ha-test-2 stonithd: [1329]: debug: Verifying authentication: uidptr=0x805c2f8 gidptr=0x0
Nov 8 13:45:07 ha-test-2 lrmd: [1916]: debug: Waiting until the final result returned.
Nov 8 13:45:07 ha-test-2 lrmd: [1916]: debug: Will call stonithd_receive_ops_result.
Nov 8 13:45:07 ha-test-2 lrmd: [1916]: debug: setting exit code=0
Nov 8 13:45:07 ha-test-2 lrmd: [1916]: debug: stonithRA orignal exit code=0
Nov 8 13:45:27 ha-test-2 stonithd: [1329]: debug: SO_PEERCRED returned [1934, (0:0)]
Nov 8 13:45:27 ha-test-2 stonithd: [1329]: debug: Verifying authentication: cred.uid=0 cred.gid=0
Nov 8 13:45:27 ha-test-2 stonithd: [1329]: debug: Verifying authentication: uidptr=0x805c2f8 gidptr=0x0
Nov 8 13:45:27 ha-test-2 lrmd: [1934]: debug: Will send the stonith RA operation to stonithd: apcmastersnmp monitor
Nov 8 13:45:27 ha-test-2 lrmd: [1934]: debug: Waiting until the final result returned.
Nov 8 13:45:27 ha-test-2 lrmd: [1934]: debug: Will call stonithd_receive_ops_result.
Nov 8 13:45:27 ha-test-2 lrmd: [1934]: debug: setting exit code=0
Nov 8 13:45:27 ha-test-2 lrmd: [1934]: debug: stonithRA orignal exit code=0
Nov 8 13:45:47 ha-test-2 stonithd: [1329]: debug: SO_PEERCRED returned [1952, (0:0)]
Nov 8 13:45:47 ha-test-2 stonithd: [1329]: debug: Verifying authentication: cred.uid=0 cred.gid=0
Nov 8 13:45:47 ha-test-2 stonithd: [1329]: debug: Verifying authentication: uidptr=0x805c2f8 gidptr=0x0
Nov 8 13:45:47 ha-test-2 lrmd: [1952]: debug: Will send the stonith RA operation to stonithd: apcmastersnmp monitor
Nov 8 13:45:47 ha-test-2 lrmd: [1952]: debug: Waiting until the final result returned.
Nov 8 13:45:47 ha-test-2 lrmd: [1952]: debug: Will call stonithd_receive_ops_result.
Nov 8 13:45:47 ha-test-2 lrmd: [1952]: debug: setting exit code=0
Nov 8 13:45:47 ha-test-2 lrmd: [1952]: debug: stonithRA orignal exit code=0
Nov 8 13:46:01 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (171) from ha-test-1
Nov 8 13:46:01 ha-test-2 cib: [1328]: WARN: mask(notify.c:do_cib_notify): Inhibiting notify.
Nov 8 13:46:01 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (172) from ha-test-1
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.61
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.62
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="61">
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" rc_code="0" op_status="0" id="rg1:IPaddr3_monitor_30000"/>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="62">
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op transition_magic="4:7;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" rc_code="7" op_status="4" id="rg1:IPaddr3_monitor_30000"/>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:01 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:02 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (174) from ha-test-1
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.62
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.63
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="62">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op op_status="0" call_id="16" rc_code="0" origin="do_update_resource" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:IPaddr3_start_0"/>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="63">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op op_status="-1" call_id="-1" rc_code="-1" origin="cib_action_update" transition_key="8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="-1:-1;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:IPaddr3_start_0"/>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:02 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (175) from ha-test-1
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.63
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.64
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="63">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op op_status="-1" call_id="-1" rc_code="-1" origin="cib_action_update" transition_magic="-1:-1;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:IPaddr3_start_0"/>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="64">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op op_status="0" call_id="22" rc_code="0" origin="do_update_resource" transition_magic="0:0;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:IPaddr3_start_0"/>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:02 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (176) from ha-test-1
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.64
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.65
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="64"/>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="65">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:pbs1">
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op id="rg1:pbs1_stop_0" operation="stop" origin="do_update_resource" transition_key="8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="21" rc_code="0" op_status="0"/>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:02 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:03 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (178) from ha-test-1
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.65
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.66
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="65">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:pbs1">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op op_status="0" call_id="18" rc_code="0" origin="do_update_resource" transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:pbs1_start_0"/>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="66">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:pbs1">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op op_status="-1" call_id="-1" rc_code="-1" origin="cib_action_update" transition_key="8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="-1:-1;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:pbs1_start_0"/>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:03 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (179) from ha-test-1
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.66
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.67
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="66">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op transition_key="7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:7;7:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" call_id="17" rc_code="7" op_status="4" id="rg1:IPaddr3_monitor_30000"/>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="67">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op transition_key="8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" call_id="24" rc_code="0" op_status="0" id="rg1:IPaddr3_monitor_30000"/>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:03 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:05 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (17c) from ha-test-1
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.67
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.68
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="67">
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:pbs1">
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op op_status="-1" call_id="-1" rc_code="-1" origin="cib_action_update" transition_magic="-1:-1;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:pbs1_start_0"/>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="68">
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:pbs1">
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op op_status="0" call_id="25" rc_code="0" origin="do_update_resource" transition_magic="0:0;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:pbs1_start_0"/>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:05 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:07 ha-test-2 stonithd: [1329]: debug: SO_PEERCRED returned [1970, (0:0)]
Nov 8 13:46:07 ha-test-2 stonithd: [1329]: debug: Verifying authentication: cred.uid=0 cred.gid=0
Nov 8 13:46:07 ha-test-2 lrmd: [1970]: debug: Will send the stonith RA operation to stonithd: apcmastersnmp monitor
Nov 8 13:46:07 ha-test-2 stonithd: [1329]: debug: Verifying authentication: uidptr=0x805c2f8 gidptr=0x0
Nov 8 13:46:07 ha-test-2 lrmd: [1970]: debug: Waiting until the final result returned.
Nov 8 13:46:07 ha-test-2 lrmd: [1970]: debug: Will call stonithd_receive_ops_result.
Nov 8 13:46:07 ha-test-2 lrmd: [1970]: debug: setting exit code=0
Nov 8 13:46:07 ha-test-2 lrmd: [1970]: debug: stonithRA orignal exit code=0
Nov 8 13:46:27 ha-test-2 stonithd: [1329]: debug: SO_PEERCRED returned [1991, (0:0)]
Nov 8 13:46:27 ha-test-2 stonithd: [1329]: debug: Verifying authentication: cred.uid=0 cred.gid=0
Nov 8 13:46:27 ha-test-2 stonithd: [1329]: debug: Verifying authentication: uidptr=0x805c2f8 gidptr=0x0
Nov 8 13:46:27 ha-test-2 lrmd: [1991]: debug: Will send the stonith RA operation to stonithd: apcmastersnmp monitor
Nov 8 13:46:27 ha-test-2 lrmd: [1991]: debug: Waiting until the final result returned.
Nov 8 13:46:27 ha-test-2 lrmd: [1991]: debug: Will call stonithd_receive_ops_result.
Nov 8 13:46:27 ha-test-2 lrmd: [1991]: debug: setting exit code=0
Nov 8 13:46:27 ha-test-2 lrmd: [1991]: debug: stonithRA orignal exit code=0
Nov 8 13:46:33 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (199) from ha-test-1
Nov 8 13:46:33 ha-test-2 cib: [1328]: WARN: mask(notify.c:do_cib_notify): Inhibiting notify.
Nov 8 13:46:33 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (19a) from ha-test-1
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.69
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.70
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="69">
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op transition_magic="0:0;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" rc_code="0" op_status="0" id="rg1:IPaddr3_monitor_30000"/>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="70">
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op transition_magic="4:7;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="monitor_failed" rc_code="7" op_status="4" id="rg1:IPaddr3_monitor_30000"/>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:33 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:33 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (19b) from ha-test-1
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.70
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.71
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="70">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op op_status="0" call_id="22" rc_code="0" origin="do_update_resource" transition_key="8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:IPaddr3_start_0"/>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="71">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op op_status="-1" call_id="-1" rc_code="-1" origin="cib_action_update" transition_key="9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="-1:-1;9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" id="rg1:IPaddr3_start_0"/>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:34 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (19d) from ha-test-1
Nov 8 13:46:34 ha-test-2 cib: [1328]: WARN: mask(notify.c:do_cib_notify): Inhibiting notify.
Nov 8 13:46:34 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (19e) from ha-test-1
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.72
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.73
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="72">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op op_status="-1" call_id="-1" rc_code="-1" origin="cib_action_update" transition_magic="-1:-1;9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="running" id="rg1:IPaddr3_start_0"/>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="73">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op op_status="4" call_id="27" rc_code="1" origin="do_update_resource" transition_magic="4:1;9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="start_failed" id="rg1:IPaddr3_start_0"/>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:34 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (19f) from ha-test-1
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.73
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.74
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="73">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_resource id="rg1:pbs1">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <lrm_rsc_op transition_key="8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;8:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="21" id="rg1:pbs1_stop_0"/>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resource>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </node_state>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - </cib>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="74">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:pbs1">
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op transition_key="9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;9:95faea5b-47d9-4ab1-89bf-ecb5293bf218" call_id="26" id="rg1:pbs1_stop_0"/>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:34 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:35 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (1a1) from ha-test-1
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.74
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.75
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="74"/>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="75">
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr3">
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op id="rg1:IPaddr3_stop_0" operation="stop" origin="do_update_resource" transition_key="10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="29" rc_code="0" op_status="0"/>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:35 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:35 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (1a2) from ha-test-1
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.75
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.76
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="75"/>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="76">
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:IPaddr1">
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op id="rg1:IPaddr1_stop_0" operation="stop" origin="do_update_resource" transition_key="10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="31" rc_code="0" op_status="0"/>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:36 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:40 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (1a8) from ha-test-1
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.76
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.77
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="76"/>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="77">
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:fpbs1">
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op id="rg1:fpbs1_stop_0" operation="stop" origin="do_update_resource" transition_key="10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="33" rc_code="0" op_status="0"/>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:40 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (1a9) from ha-test-1
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.77
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.78
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="77"/>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="78">
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="rg1:drbd1">
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op id="rg1:drbd1_stop_0" operation="stop" origin="do_update_resource" transition_key="10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="4:1;10:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stop_failed" call_id="35" rc_code="1" op_status="4"/>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:40 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:41 ha-test-2 stonithd: [1329]: info: stonith_operate_locally::2144: sending fencing op (1) for ha-test-1 to device apcmastersnmp (rsc_id=fence1:apc1:1, pid=2009)
Nov 8 13:46:41 ha-test-2 stonithd: [1329]: info: Node ha-test-2 try to help node ha-test-1 to fence node ha-test-1.
Nov 8 13:46:41 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_peer_callback): Processing cib_apply_diff msg (1ac) from ha-test-1
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cibmon.c:cibmon_diff): [cib_diff_notify] cib_apply_diff confirmed
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: --- 0.2.78
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): Diff: +++ 0.2.79
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): - <cib num_updates="78"/>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <cib num_updates="79">
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <status>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <node_state id="026c6f3c-65d5-40ae-855f-234c1a3c6fcc">
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resources>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_resource id="fence1:apc1:0">
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + <lrm_rsc_op id="fence1:apc1:0_stop_0" operation="stop" origin="do_update_resource" transition_key="11:95faea5b-47d9-4ab1-89bf-ecb5293bf218" transition_magic="0:0;11:95faea5b-47d9-4ab1-89bf-ecb5293bf218" rsc_state="stopped" call_id="37" rc_code="0" op_status="0"/>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resource>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm_resources>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </lrm>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </node_state>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </status>
Nov 8 13:46:41 ha-test-2 cibmon: [1326]: info: mask(cib_apply_diff): + </cib>
Nov 8 13:46:42 ha-test-2 kernel: e1000: eth1: e1000_watchdog: NIC Link is Down
Nov 8 13:46:46 ha-test-2 kernel: drbd0: PingAck did not arrive in time.
Nov 8 13:46:46 ha-test-2 kernel: drbd0: drbd0_asender [24606]: cstate Connected --> NetworkFailure
Nov 8 13:46:46 ha-test-2 kernel: drbd0: asender terminated
Nov 8 13:46:46 ha-test-2 kernel: drbd0: drbd0_receiver [1703]: cstate NetworkFailure --> BrokenPipe
Nov 8 13:46:46 ha-test-2 kernel: drbd0: short read expecting header on sock: r=-512
Nov 8 13:46:47 ha-test-2 kernel: drbd0: worker terminated
Nov 8 13:46:47 ha-test-2 kernel: drbd2: PingAck did not arrive in time.
Nov 8 13:46:47 ha-test-2 kernel: drbd0: drbd0_receiver [1703]: cstate BrokenPipe --> Unconnected
Nov 8 13:46:47 ha-test-2 kernel: drbd0: Connection lost.
Nov 8 13:46:47 ha-test-2 kernel: drbd0: drbd0_receiver [1703]: cstate Unconnected --> WFConnection
Nov 8 13:46:47 ha-test-2 kernel: drbd2: drbd2_asender [24607]: cstate Connected --> NetworkFailure
Nov 8 13:46:47 ha-test-2 kernel: drbd3: PingAck did not arrive in time.
Nov 8 13:46:47 ha-test-2 kernel: drbd3: drbd3_asender [24608]: cstate Connected --> NetworkFailure
Nov 8 13:46:47 ha-test-2 kernel: drbd3: asender terminated
Nov 8 13:46:47 ha-test-2 kernel: drbd3: drbd3_receiver [1727]: cstate NetworkFailure --> BrokenPipe
Nov 8 13:46:47 ha-test-2 kernel: drbd3: short read expecting header on sock: r=-512
Nov 8 13:46:47 ha-test-2 kernel: drbd3: worker terminated
Nov 8 13:46:47 ha-test-2 kernel: drbd3: drbd3_receiver [1727]: cstate BrokenPipe --> Unconnected
Nov 8 13:46:47 ha-test-2 kernel: drbd3: Connection lost.
Nov 8 13:46:47 ha-test-2 kernel: drbd3: drbd3_receiver [1727]: cstate Unconnected --> WFConnection
Nov 8 13:46:47 ha-test-2 kernel: drbd2: asender terminated
Nov 8 13:46:47 ha-test-2 kernel: drbd2: drbd2_receiver [1719]: cstate NetworkFailure --> BrokenPipe
Nov 8 13:46:47 ha-test-2 kernel: drbd2: short read expecting header on sock: r=-512
Nov 8 13:46:47 ha-test-2 kernel: drbd2: worker terminated
Nov 8 13:46:47 ha-test-2 kernel: drbd4: PingAck did not arrive in time.
Nov 8 13:46:47 ha-test-2 kernel: drbd4: drbd4_asender [24609]: cstate Connected --> NetworkFailure
Nov 8 13:46:47 ha-test-2 kernel: drbd4: asender terminated
Nov 8 13:46:47 ha-test-2 kernel: drbd4: drbd4_receiver [1735]: cstate NetworkFailure --> BrokenPipe
Nov 8 13:46:47 ha-test-2 kernel: drbd4: short read expecting header on sock: r=-512
Nov 8 13:46:47 ha-test-2 kernel: drbd4: worker terminated
Nov 8 13:46:47 ha-test-2 kernel: drbd4: drbd4_receiver [1735]: cstate BrokenPipe --> Unconnected
Nov 8 13:46:47 ha-test-2 kernel: drbd4: Connection lost.
Nov 8 13:46:47 ha-test-2 kernel: drbd4: drbd4_receiver [1735]: cstate Unconnected --> WFConnection
Nov 8 13:46:47 ha-test-2 kernel: drbd2: drbd2_receiver [1719]: cstate BrokenPipe --> Unconnected
Nov 8 13:46:47 ha-test-2 kernel: drbd2: Connection lost.
Nov 8 13:46:47 ha-test-2 kernel: drbd2: drbd2_receiver [1719]: cstate Unconnected --> WFConnection
Nov 8 13:46:47 ha-test-2 stonithd: [1329]: debug: SO_PEERCRED returned [2014, (0:0)]
Nov 8 13:46:47 ha-test-2 stonithd: [1329]: debug: Verifying authentication: cred.uid=0 cred.gid=0
Nov 8 13:46:47 ha-test-2 lrmd: [2014]: debug: Will send the stonith RA operation to stonithd: apcmastersnmp monitor
Nov 8 13:46:47 ha-test-2 stonithd: [1329]: debug: Verifying authentication: uidptr=0x805c2f8 gidptr=0x0
Nov 8 13:46:47 ha-test-2 lrmd: [2014]: debug: Waiting until the final result returned.
Nov 8 13:46:47 ha-test-2 lrmd: [2014]: debug: Will call stonithd_receive_ops_result.
Nov 8 13:46:47 ha-test-2 lrmd: [2014]: debug: setting exit code=0
Nov 8 13:46:47 ha-test-2 lrmd: [2014]: debug: stonithRA orignal exit code=0
Nov 8 13:46:48 ha-test-2 ifmonitord[1342]: setting interface status of eth1 to down
Nov 8 13:46:48 ha-test-2 crmadmin: [2036]: info: mask(cib_native.c:cib_native_signon): Connection to CIB successful
Nov 8 13:46:48 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_query from client 9881f909-e509-463c-81ca-0aa36736a22/cib_rw
Nov 8 13:46:48 ha-test-2 crmadmin: [2036]: info: mask(cib_native.c:cib_native_signoff): Signing out of the CIB Service
Nov 8 13:46:48 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x8110220) 9881f909-e509-463c-81ca-0aa36736a22/<null>
Nov 8 13:46:48 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x818ef58) b7ce1756-e269-4e9e-9b88-9b3ed5e536c/crmadmin
Nov 8 13:46:48 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client 2ecac33f-feda-480d-8e54-32d73b5dc79/cib_rw
Nov 8 13:46:49 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x818ef58) 2ecac33f-feda-480d-8e54-32d73b5dc79/<null>
Nov 8 13:46:49 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x81e4e88) 4d0951b3-6f9c-41f7-bb50-a476f8aa554/crm_attribute
Nov 8 13:46:50 ha-test-2 kernel: drbd1: PingAck did not arrive in time.
Nov 8 13:46:50 ha-test-2 kernel: drbd1: drbd1_asender [1874]: cstate Connected --> NetworkFailure
Nov 8 13:46:50 ha-test-2 kernel: drbd1: asender terminated
Nov 8 13:46:50 ha-test-2 kernel: drbd1: drbd1_receiver [25419]: cstate NetworkFailure --> BrokenPipe
Nov 8 13:46:50 ha-test-2 kernel: drbd1: short read expecting header on sock: r=-512
Nov 8 13:46:50 ha-test-2 kernel: drbd1: worker terminated
Nov 8 13:46:50 ha-test-2 kernel: drbd1: drbd1_receiver [25419]: cstate BrokenPipe --> Unconnected
Nov 8 13:46:50 ha-test-2 kernel: drbd1: Connection lost.
Nov 8 13:46:50 ha-test-2 kernel: drbd1: drbd1_receiver [25419]: cstate Unconnected --> WFConnection
Nov 8 13:46:50 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client 55734054-4811-4eda-b8cf-232a1069bd2/cib_rw
Nov 8 13:46:51 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x81e4e88) 55734054-4811-4eda-b8cf-232a1069bd2/<null>
Nov 8 13:46:51 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x81e4d28) 6663aef3-ce33-4625-8b87-3ac83ed0793/crm_attribute
Nov 8 13:46:52 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client 72eb28d0-4c1d-4268-ac96-551ee3a4f65/cib_rw
Nov 8 13:46:53 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x81e4d28) 72eb28d0-4c1d-4268-ac96-551ee3a4f65/<null>
Nov 8 13:46:53 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x81db208) ac2e66d9-b86c-413a-a0a2-8194bff8937/crm_attribute
Nov 8 13:46:54 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client 020469f0-d010-4c65-b0a1-09857ee04b7/cib_rw
Nov 8 13:46:55 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x81db208) 020469f0-d010-4c65-b0a1-09857ee04b7/<null>
Nov 8 13:46:55 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x818b9f8) 3ac150f5-fcf1-4451-a1c8-1b4e63c6efa/crm_attribute
Nov 8 13:46:56 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client 6eb576a4-4754-4458-80e0-8c40de67342/cib_rw
Nov 8 13:46:57 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x818b9f8) 6eb576a4-4754-4458-80e0-8c40de67342/<null>
Nov 8 13:46:57 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x81d90a8) dbba65a1-3913-4fcf-9abf-37ba33e3e2b/crm_attribute
Nov 8 13:46:58 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client 79580307-6c11-4a93-af2a-61d99dec0eb/cib_rw
Nov 8 13:46:59 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x81d90a8) 79580307-6c11-4a93-af2a-61d99dec0eb/<null>
Nov 8 13:46:59 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x80c4b88) bcb46b94-5024-47b5-be08-931882456fe/crm_attribute
Nov 8 13:47:00 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client c1cdd330-dfab-40e1-b9ea-5f3b12e1251/cib_rw
Nov 8 13:47:01 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x80c4b88) c1cdd330-dfab-40e1-b9ea-5f3b12e1251/<null>
Nov 8 13:47:01 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x806fd80) 8c5d15d2-03f6-44d7-92ad-85a113346be/crm_attribute
Nov 8 13:47:02 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client 7df97999-a734-4fa5-a569-24cedb045b9/cib_rw
Nov 8 13:47:03 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x806fd80) 7df97999-a734-4fa5-a569-24cedb045b9/<null>
Nov 8 13:47:03 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x81df780) 3ab58f65-b8ae-4552-bbe0-98f2624cd02/crm_attribute
Nov 8 13:47:04 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client f12b7393-8c84-45a6-9769-d555d247100/cib_rw
Nov 8 13:47:05 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_rw channel disconnect from client (0x81df780) f12b7393-8c84-45a6-9769-d555d247100/<null>
Nov 8 13:47:05 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_process_disconnect): Cleaning up after cib_callback channel disconnect from client (0x80c4768) 107652b8-634e-44a1-b3e3-0b9b6699513/crm_attribute
Nov 8 13:47:06 ha-test-2 cib: [1328]: info: mask(callbacks.c:cib_common_callback): Operation cib_update from client 7fcf91d4-b105-4731-98ac-06e129d289d/cib_rw
More information about the Linux-HA-Dev
mailing list