[Linux-ha-dev] hb report - troubles on 4 node cluster

[Dejan Muhamedagic]

Hi Andreas,

On Sun, Feb 10, 2008 at 09:38:45PM +0100, Andreas Mather1 wrote:
> ***********************
> Warning: Your file, report_1.tar.gz, contains more than 32 files after decompression and cannot be scanned.
> ***********************
> 
> 
> 
> 
> Hi all,
> 
> Please find attached a hb_report for a problem I experienced when
> implementing heartbeat.
> 
> The environment:
> It's an asymmetric 4 node cluster, running heartbeat 2.1.3. All nodes share
> a couple of filesystems, all GPFS formatted. Services inlcude WebSphere
> (modified RA), DB2 (modified RA), vsftpd (Xinetd), samba, nfs, MCS (self
> written RA), IHS and are put in 4 groups (filesvc, mcs, was, db). Dejan is
> also familiar with the setup.
> OS: SLES 9.3 (x86_64)
> hearbeat: build via ./ConfigureMe package
> 
> 
> The Problem:
> In general, everything works fine (crm_standby works for every node, etc.),
> but, when I simulate a power loss of one node (via IBM RSA)*, a cluster
> split occurs when this node rejoins. Suddenly, on every node, crm_mon shows
> the node it is running on as 'online' while reporting the other nodes as
> 'OFFLINE'. After 1 - 2 min. the cluster is fully operational again (all
> nodes found themself again), but it seems as every resource gets restarted.
> 
> Please let me know, if I can provide further information.

>From the log on rbxw02:

Feb 10 19:10:16 rbxw02 heartbeat: [22769]: info: Link rbxd01:eth0 up.
Feb 10 19:10:16 rbxw02 heartbeat: [22769]: info: Link rbxw01:eth0 up.
Feb 10 19:10:32 rbxw02 heartbeat: [22769]: info: Link rbxd02:eth2 dead.
Feb 10 19:13:16 rbxw02 heartbeat: [22769]: info: Link rbxd01:eth0 dead.
Feb 10 19:13:16 rbxw02 heartbeat: [22769]: info: Link rbxd01:eth2 dead.
Feb 10 19:13:17 rbxw02 heartbeat: [22769]: info: Link rbxw01:eth0 dead.
Feb 10 19:13:17 rbxw02 heartbeat: [22769]: info: Link rbxw01:eth2 dead.
Feb 10 19:15:06 rbxw02 heartbeat: [22769]: CRIT: Cluster node rbxw01 returning after partition.
Feb 10 19:15:06 rbxw02 heartbeat: [22769]: info: Link rbxw01:eth2 up.
Feb 10 19:15:06 rbxw02 heartbeat: [22769]: info: Link rbxd02:eth2 up.
Feb 10 19:15:07 rbxw02 heartbeat: [22769]: CRIT: Cluster node rbxd01 returning after partition.
Feb 10 19:15:07 rbxw02 heartbeat: [22769]: info: Link rbxd01:eth2 up.

Strange timestamps. Which node went down? And when? Also,
rbxd02:eth0 was not reported as down and rbxw01:eth0 rbxd01:eth0
not as up: probably at some point rbxw02:eth0 went down. It would
be interesting to see logs from the other nodes. Don't know why
hb_report didn't pack them.

Two extra nodes went DC around 19:13 for about two minutes, which
means that there were three partitions: w02,d02 and w01 and d01.
Note that none of them had quorum.

Looks like a network problem, but an awkward one. Don't know how
it got disrupted this much. Perhaps you could try with unicast:
replace each mcast directive with four ucast directives.

Cheers,

Dejan

> Thanks,
> 
> Andreas
> 
> 
> * Sorry, I forgot to test what happens, when I just stop and start
> heartbeat on that node - would be useful too, I think... :(
> 
> 
> 
> 
> (See attached file: report_1.tar.gz)
> 
> Mit freundlichen Gr??en / Best regards
> 
> Andreas MATHER
> ESLT - Enterprise Services for Linux Technologies
> 
> IBM Austria, Obere Donaustrasse 95, 1020 Vienna
> Phone : +43-1-21145/4799
> Fax: +43-1-21145/8888
> e-mail: andreas.mather at at.ibm.com
> 
> IBM ?sterreich Internationale B?romaschinen Gesellschaft m.b.H.
> Sitz: Wien
> Firmenbuchgericht: Handelsgericht Wien, FN 80000y


> _______________________________________________________
> Linux-HA-Dev: Linux-HA-Dev at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/