DRBD/Pacemaker failover issue

DRBD
1

Hello

There is two-node active/standby DRBD/Pacemaker cluster configured with qdevice.
There are two resource groups (g-nfs, g-iscsi) configured to always run on the same node.

Could you please help me understand the next behavior - during reboot active node memverge, ha-iscsi was successfully promoted to standby memverge2, while ha-nfs was failed due to timed out. This blocked start resources on standby memverge2 node.

Here is below more description behaviour, logs and configs.

Cluster with quorum, qdevice temporarily unavailable. Two nodes up and running, resource groups are running on memverge.

Perform os update (dnf update) on memverge, reboot. Boot is blocked due to the LUK2 encryption password requirement.

Cluster without quorum, all resource groups are stopped.

Qdevice back online, cluster with quorum again, but resource groups are still stopped with next status,

Cluster Summary:
  * Stack: corosync (Pacemaker is running)
  * Current DC: memverge2 (28) (version 3.0.0-5.1.el10_0-48413c8) - partition with quorum
  * Last updated: Sat Oct 18 09:12:43 2025 on memverge2
  * Last change:  Sat Oct 18 09:06:03 2025 by root via root on memverge2
  * 2 nodes configured
  * 22 resource instances configured

Node List:
  * Node memverge (27): OFFLINE
  * Node memverge2 (28): online, feature set 3.20.1

Full List of Resources:
  * ipmi-fence-memverge2        (stonith:fence_ipmilan):         Stopped
  * ipmi-fence-memverge (stonith:fence_ipmilan):         Started memverge2
  * Clone Set: ha-nfs-clone [ha-nfs] (promotable):
    * ha-nfs    (ocf:linbit:drbd):       Unpromoted memverge2
    * ha-nfs    (ocf:linbit:drbd):       Stopped
  * Resource Group: g-nfs:
    * pb_nfs    (ocf:heartbeat:portblock):       Stopped
    * ip0_nfs   (ocf:heartbeat:IPaddr2):         Stopped
    * fs_nfs_internal_info_HA   (ocf:heartbeat:Filesystem):      Stopped
    * fs_nfsshare_exports_HA    (ocf:heartbeat:Filesystem):      Stopped
    * nfsserver (ocf:heartbeat:nfsserver):       Stopped
    * expfs_nfsshare_exports_HA (ocf:heartbeat:exportfs):        Stopped
    * samba_service     (systemd:smb):   Stopped
    * fs_sambashare_exports_HA  (ocf:heartbeat:Filesystem):      Stopped
    * punb_nfs  (ocf:heartbeat:portblock):       Stopped
  * Clone Set: ha-iscsi-clone [ha-iscsi] (promotable):
    * ha-iscsi  (ocf:linbit:drbd):       Promoted memverge2
    * ha-iscsi  (ocf:linbit:drbd):       Stopped
  * Resource Group: g-iscsi:
    * pb_iscsi  (ocf:heartbeat:portblock):       Stopped
    * ip0_iscsi (ocf:heartbeat:IPaddr2):         Stopped
    * ip1_iscsi (ocf:heartbeat:IPaddr2):         Stopped
    * iscsi_target      (ocf:heartbeat:iSCSITarget):     Stopped
    * iscsi_lun_drbd3   (ocf:heartbeat:iSCSILogicalUnit):        Stopped
    * iscsi_lun_drbd4   (ocf:heartbeat:iSCSILogicalUnit):        Stopped
    * punb_iscsi        (ocf:heartbeat:portblock):       Stopped


Failed Resource Actions:
  * ha-nfs_promote_0 on memverge2 'Error occurred' (1): call=133, status='Timed out', exitreason='Resource agent did not complete within 1m30s', last-rc-change='Sat Oct 18 09:06:04 2025', queued=0ms, exec=90738ms



[root@memverge2 ~]# cat /var/log/messages|grep -i ha-nfs_promote
Oct 18 09:06:04 memverge2 pacemaker-controld[2939]: notice: Initiating promote operation ha-nfs_promote_0 locally on memverge2
Oct 18 09:07:35 memverge2 pacemaker-controld[2939]: notice: Transition 803 action 7 (ha-nfs_promote_0 on memverge2): expected 'OK' but got 'Error occurred'
[root@memverge2 ~]#
[root@memverge2 ~]# cat /var/log/messages|grep -i ha-iscsi_promote
Oct 18 09:06:03 memverge2 pacemaker-controld[2939]: notice: Initiating promote operation ha-iscsi_promote_0 locally on memverge2
Oct 18 09:11:48 memverge2 pacemaker-controld[2939]: notice: Initiating promote operation ha-iscsi_promote_0 locally on memverge2

[root@memverge2 ~]# cat /var/log/pacemaker/pacemaker.log|grep -i ha-nfs_promote
Oct 18 09:06:04.547 memverge2 pacemaker-controld  [2939] (execute_rsc_action)   notice: Initiating promote operation ha-nfs_promote_0 locally on memverge2 | action 7
Oct 18 09:06:04.547 memverge2 pacemaker-controld  [2939] (do_lrm_rsc_op)        notice: Requesting local execution of promote operation for ha-nfs on memverge2 | transition_key=7:803:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309 op_key=ha-nfs_promote_0
Oct 18 09:06:04.547 memverge2 pacemaker-based     [2934] (cib_perform_op)       info: +  /cib/status/node_state[@id='28']/lrm[@id='28']/lrm_resources/lrm_resource[@id='ha-nfs']/lrm_rsc_op[@id='ha-nfs_last_0']:  @operation_key=ha-nfs_promote_0, @operation=promote, @crm-debug-origin=controld_update_resource_history, @transition-key=7:803:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309, @transition-magic=-1:193;7:803:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1760767564, @exec-time=0
Oct 18 09:07:35.285 memverge2 pacemaker-execd     [2936] (async_action_complete)        info: Resource agent ha-nfs_promote_0[3534142] timed out after 1m30s
Oct 18 09:07:35.285 memverge2 pacemaker-controld  [2939] (log_executor_event)   error: Result of promote operation for ha-nfs on memverge2: Timed out after 1m30s (Resource agent did not complete within 1m30s) | graph action confirmed; call=133 key=ha-nfs_promote_0
Oct 18 09:07:35.286 memverge2 pacemaker-based     [2934] (cib_perform_op)       info: ++ /cib/status/node_state[@id='28']/lrm[@id='28']/lrm_resources/lrm_resource[@id='ha-nfs']:  <lrm_rsc_op id="ha-nfs_last_failure_0" operation_key="ha-nfs_promote_0" operation="promote" crm-debug-origin="controld_update_resource_history" crm_feature_set="3.20.1" transition-key="7:803:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309" transition-magic="2:1;7:803:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309" exit-reason="Resource agent did not complete within 1m30s" on_node="memverge2" call-id="133" rc-code="1" op-status="2" interval="0" last-rc-change="1760767564" exec-time="90738" queue-time="0" op-digest="876800cf6419330beb37dc5fd73e30fb"/>
Oct 18 09:07:35.286 memverge2 pacemaker-controld  [2939] (abort_transition_graph)       info: Transition 803 aborted by operation ha-nfs_promote_0 'modify' on memverge2: Event failed | magic=2:1;7:803:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309 cib=0.4055.28 source=process_graph_event:559 complete=false
Oct 18 09:07:35.286 memverge2 pacemaker-controld  [2939] (process_graph_event)  notice: Transition 803 action 7 (ha-nfs_promote_0 on memverge2): expected 'OK' but got 'Error occurred' | target-rc=0 rc=1 call-id=133
[root@memverge2 ~]#

[root@memverge2 ~]# cat /var/log/pacemaker/pacemaker.log|grep -i ha-iscsi_promote
Oct 18 09:06:03.193 memverge2 pacemaker-controld  [2939] (execute_rsc_action)   notice: Initiating promote operation ha-iscsi_promote_0 locally on memverge2 | action 71
Oct 18 09:06:03.193 memverge2 pacemaker-controld  [2939] (do_lrm_rsc_op)        notice: Requesting local execution of promote operation for ha-iscsi on memverge2 | transition_key=71:801:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309 op_key=ha-iscsi_promote_0
Oct 18 09:06:03.194 memverge2 pacemaker-based     [2934] (cib_perform_op)       info: +  /cib/status/node_state[@id='28']/lrm[@id='28']/lrm_resources/lrm_resource[@id='ha-iscsi']/lrm_rsc_op[@id='ha-iscsi_last_0']:  @operation_key=ha-iscsi_promote_0, @operation=promote, @crm-debug-origin=controld_update_resource_history, @transition-key=71:801:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309, @transition-magic=-1:193;71:801:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1760767563, @exec-time=0
Oct 18 09:06:03.382 memverge2 pacemaker-controld  [2939] (log_executor_event)   notice: Result of promote operation for ha-iscsi on memverge2: OK | graph action confirmed; call=125 key=ha-iscsi_promote_0 rc=0
Oct 18 09:06:03.384 memverge2 pacemaker-controld  [2939] (process_graph_event)  info: Transition 801 action 71 (ha-iscsi_promote_0 on memverge2) confirmed: OK | rc=0 call-id=125
Oct 18 09:11:48.123 memverge2 pacemaker-controld  [2939] (execute_rsc_action)   notice: Initiating promote operation ha-iscsi_promote_0 locally on memverge2 | action 37
Oct 18 09:11:48.123 memverge2 pacemaker-controld  [2939] (do_lrm_rsc_op)        notice: Requesting local execution of promote operation for ha-iscsi on memverge2 | transition_key=37:807:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309 op_key=ha-iscsi_promote_0
Oct 18 09:11:48.124 memverge2 pacemaker-based     [2934] (cib_perform_op)       info: +  /cib/status/node_state[@id='28']/lrm[@id='28']/lrm_resources/lrm_resource[@id='ha-iscsi']/lrm_rsc_op[@id='ha-iscsi_last_0']:  @operation_key=ha-iscsi_promote_0, @operation=promote, @transition-key=37:807:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309, @transition-magic=-1:193;37:807:0:b9d3697a-e33b-4b53-bb46-35e2e80d0309, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1760767908, @exec-time=0
Oct 18 09:11:48.154 memverge2 pacemaker-controld  [2939] (log_executor_event)   notice: Result of promote operation for ha-iscsi on memverge2: OK | graph action confirmed; call=156 key=ha-iscsi_promote_0 rc=0
Oct 18 09:11:48.155 memverge2 pacemaker-controld  [2939] (process_graph_event)  info: Transition 807 action 37 (ha-iscsi_promote_0 on memverge2) confirmed: OK | rc=0 call-id=156
[root@memverge2 ~]#
[root@memverge2 ~]# drbdadm status
ha-iscsi role:Primary
  volume:3 disk:UpToDate open:no
  volume:4 disk:UpToDate open:no
  memverge connection:Connecting

ha-nfs role:Secondary
  volume:1 disk:Consistent open:no
  volume:2 disk:Consistent open:no
  volume:5 disk:Consistent open:no
  memverge connection:Connecting

[root@memverge2 ~]#

Location Constraints:
  resource 'ipmi-fence-memverge' avoids node 'memverge' with score INFINITY (id: location-ipmi-fence-memverge-memverge--INFINITY)
  resource 'ipmi-fence-memverge2' avoids node 'memverge2' with score INFINITY (id: location-ipmi-fence-memverge2-memverge2--INFINITY)
  resource 'ha-iscsi-clone' (id: drbd-fence-by-handler-ha-iscsi-ha-iscsi-clone)
    Rules:
      Rule: role=Promoted score=-INFINITY (id: drbd-fence-by-handler-ha-iscsi-rule-ha-iscsi-clone)
        Expression: #uname ne memverge2 (id: drbd-fence-by-handler-ha-iscsi-expr-28-ha-iscsi-clone)
Colocation Constraints:
  Started resource 'g-nfs' with Promoted resource 'ha-nfs-clone' (id: colocation-g-nfs-ha-nfs-clone-INFINITY)
    score=INFINITY
  Started resource 'g-iscsi' with Promoted resource 'ha-iscsi-clone' (id: colocation-g-iscsi-ha-iscsi-clone-INFINITY)
    score=INFINITY
  Started resource 'g-nfs' with Started resource 'g-iscsi' (id: colocation-g-nfs-g-iscsi-INFINITY)
    score=INFINITY
Order Constraints:
  promote resource 'ha-nfs-clone' then start resource 'g-nfs' (id: order-ha-nfs-clone-g-nfs-mandatory)
  promote resource 'ha-iscsi-clone' then start resource 'g-iscsi' (id: order-ha-iscsi-clone-g-iscsi-mandatory)
  start resource 'g-nfs' then start resource 'g-iscsi' (id: order-g-nfs-g-iscsi-mandatory)

Anton

2 
[root@memverge2 ~]# drbdadm status
ha-iscsi role:Primary
  volume:3 disk:UpToDate open:no
  volume:4 disk:UpToDate open:no
  memverge connection:Connecting

ha-nfs role:Secondary
  volume:1 disk:Consistent open:no
  volume:2 disk:Consistent open:no
  volume:5 disk:Consistent open:no
  memverge connection:Connecting

Your DRBD resources are not connected. I don’t see the status of the resources from memverge but I suspect it’s likely in StandAlone You’ve probably split-brained the resources at some point in all your testing (DRBD 9.0 en - LINBIT).

1 Like
3

Hhmm… I just cross-checked it again. There are no split-brained resources at the beginning of the test. The same behaviour as week before.

Cluster with quorum, qdevice temporarily unavailable. Two nodes up and running, resource groups are running on memverge.

[root@memverge ~]# drbdadm status
ha-iscsi role:Primary
  volume:3 disk:UpToDate open:yes
  volume:4 disk:UpToDate open:yes
  memverge2 role:Secondary
    volume:3 peer-disk:UpToDate
    volume:4 peer-disk:UpToDate

ha-nfs role:Primary
  volume:1 disk:UpToDate open:yes
  volume:2 disk:UpToDate open:yes
  volume:5 disk:UpToDate open:yes
  memverge2 role:Secondary
    volume:1 peer-disk:UpToDate
    volume:2 peer-disk:UpToDate
    volume:5 peer-disk:UpToDate

[root@memverge ~]#

Perform os update (dnf update, kernel 6.12.0-55.39.1 → 6.12.0-55.40.1) on memverge, reboot. Boot is blocked due to the LUK2 encryption password requirement.

Cluster without quorum, all resource groups are stopped.

Failed Resource Actions:
  * ha-nfs_promote_0 on memverge2 'Error occurred' (1): call=113, status='Timed out', exitreason='Resource agent did not complete within 1m30s', last-rc-change='Sun Oct 26 09:45:01 2025', queued=0ms, exec=90152ms

[root@memverge2 ~]# drbdadm status
# No currently configured DRBD found.
[root@memverge2 ~]#

Qdevice back online, cluster with quorum again, but resource groups are still stopped with next status.

[root@memverge2 ~]# drbdadm status
ha-iscsi role:Primary
  volume:3 disk:UpToDate open:no
  volume:4 disk:UpToDate open:no
  memverge connection:Connecting

ha-nfs role:Secondary
  volume:1 disk:Consistent open:no
  volume:2 disk:Consistent open:no
  volume:5 disk:Consistent open:no
  memverge connection:Connecting

[root@memverge2 ~]#

Anton

4

I assume the qdevices were configured within Pacemaker, correct?
Are we using any kind of fencing or quorum within DRBD? If so, try disabling and re-running the test. If that changes the test behavior then go ahead and share the DRBD configuration.

5

I assume the qdevices were configured within Pacemaker, correct?

Yes, correct.
I followed instructions described in link - https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/pdf/configuring_and_managing_high_availability_clusters/Red_Hat_Enterprise_Linux-10-Configuring_and_managing_high_availability_clusters-en-US.pdf, started at page 151.

Are we using any kind of fencing or quorum within DRBD?

No. For fencing or quorum I use fence_ipmilan and qdevice host.

Here is below drbd conf for ha-iscsi and ha-nfs,

[root@memverge drbd.d]# cat ha-iscsi.res
resource ha-iscsi {

options {
      auto-promote no;
#      quorum majority;
#      on-no-quorum io-error;
#      on-no-data-accessible suspend-io;
#      on-suspended-primary-outdated force-secondary;
        }

handlers {
      fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
      unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh";
    }

disk {
        c-plan-ahead 0;
        resync-rate 32M;
        al-extents 6007;
       no-disk-flushes;
       no-md-flushes;
       no-disk-barrier;
       no-disk-drain;
     }

  volume 3 {
    device      /dev/drbd3;
    disk        /dev/mapper/object_block_nfs_vg-ha_block_exports_lv_with_vdo_1x8;
    meta-disk   internal;
  }
  volume 4 {
    device      /dev/drbd4;
    disk        /dev/mapper/object_block_nfs_vg-ha_block_exports_lv_without_vdo;
    meta-disk   internal;
  }

  on memverge {
#    address   10.72.14.152:7901;
    node-id   27;
  }
  on memverge2 {
#    address   10.72.14.154:7901;
    node-id   28;
  }

#  on qs {
#    volume 29 {
#      disk    none;
#    }
#    volume 30 {
#      disk    none;
#    }
#    address   10.72.14.156:7900;
#    node-id   29;
#  }

#  connection-mesh {
#    hosts memverge memverge2;
#                  }

net
    {
#        load-balance-paths      yes;
        transport tcp;
        protocol  C;
        sndbuf-size 10M;
        rcvbuf-size 10M;
        max-buffers 80K;
        max-epoch-size 20000;
        timeout 90;
        ping-timeout 10;
        ping-int 15;
        connect-int 15;
        fencing resource-and-stonith;
    }
connection
    {
        path
        {
            host memverge address 192.168.0.6:7901;
            host memverge2 address 192.168.0.8:7901;
        }
        path
        {
            host memverge address 1.1.1.6:7901;
            host memverge2 address 1.1.1.8:7901;
        }
net
    {
#        load-balance-paths      yes;
        transport tcp;
        protocol  C;
        sndbuf-size 10M;
        rcvbuf-size 10M;
        max-buffers 80K;
        max-epoch-size 20000;
        timeout 90;
        ping-timeout 10;
        ping-int 15;
        connect-int 15;
        fencing resource-and-stonith;
    }
    }

}
[root@memverge drbd.d]#
[root@memverge drbd.d]#
[root@memverge drbd.d]#
[root@memverge drbd.d]# cat ha-nfs.res
resource ha-nfs {

options {
      auto-promote no;
#      quorum majority;
#      on-no-quorum io-error;
#      on-no-data-accessible suspend-io;
#      on-suspended-primary-outdated force-secondary;
        }

handlers {
      fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
      unfence-peer "/usr/lib/drbd/crm-unfence-peer.9.sh";
    }

disk {
        c-plan-ahead 0;
        resync-rate 32M;
        al-extents 6007;
     }

  volume 1 {
    device      /dev/drbd1;
    disk        /dev/mapper/object_block_nfs_vg-ha_nfs_exports_lv_with_vdo_1x8;
    meta-disk   internal;
  }
  volume 2 {
    device      /dev/drbd2;
    disk        /dev/mapper/object_block_nfs_vg-ha_nfs_internal_lv_without_vdo;
    meta-disk   internal;
  }
  volume 5 {
    device      /dev/drbd5;
    disk        /dev/mapper/object_block_nfs_vg-ha_samba_exports_lv_with_vdo_1x8;
    meta-disk   internal;
  }

  on memverge {
#    address   10.72.14.152:7900;
    node-id   27;
  }
  on memverge2 {
#    address   10.72.14.154:7900;
    node-id   28;
  }

#  on qs {
#    volume 29 {
#      disk    none;
#    }
#    volume 30 {
#      disk    none;
#    }
#    address   10.72.14.156:7900;
#    node-id   29;
#  }

#  connection-mesh {
#    hosts memverge memverge2;
#                  }

net
    {
#        load-balance-paths      yes;
        transport tcp;
        protocol  C;
        sndbuf-size 10M;
        rcvbuf-size 10M;
        max-buffers 80K;
        max-epoch-size 20000;
        timeout 90;
        ping-timeout 10;
        ping-int 15;
        connect-int 15;
        fencing resource-and-stonith;
    }
#               }
connection
    {
        path
        {
            host memverge address 192.168.0.6:7900;
            host memverge2 address 192.168.0.8:7900;
        }
        path
        {
            host memverge address 1.1.1.6:7900;
            host memverge2 address 1.1.1.8:7900;
        }
net
    {
#        load-balance-paths      yes;
        transport tcp;
        protocol  C;
        sndbuf-size 10M;
        rcvbuf-size 10M;
        max-buffers 80K;
        max-epoch-size 20000;
        timeout 90;
        ping-timeout 10;
        ping-int 15;
        connect-int 15;
        fencing resource-and-stonith;
    }
    }

}
[root@memverge drbd.d]#
[root@memverge drbd.d]#

Anton

6

Ah hah! Just as I suspected.

So, you do have fencing enabled. Note the fencing resource-and-stonith; in your DRBD configuration. In that case everything is working as I would expect.
With fencing configured within DRBD, a lone DRBD node will come up with a disk state of Consistent. In this state you cannot promote this DRBD device to primary without using --force. This is per design and to prevent a lone DRBD from going primary after a reboot and causing a split-brain. Once the node can again reach its peers and compare the data the node will fix the disk states assuming no split-brain already occurred. Essentially, it’s just a solution or safeguard to the “I started up isolated from the rest of the nodes, I have no way to guarantee my copy of the data is the right one” scenario.

Please try commenting this line out of the configuration and re-running your test again. I suspect things will behave as you expect them to in that case. At that point, you need decide which behavior you prefer. Less risk of split-brains or the ability for a single node to boot and run without admin intervention (i.e. --force)? Personally, with proper fencing within Pacemaker and a q-device, I would feel “safe enough”, even without fencing within DRBD.

7

Please try commenting this line out of the configuration and re-running your test again. I suspect things will behave as you expect them to in that case. At that point, you need decide which behavior you prefer. Less risk of split-brains or the ability for a single node to boot and run without admin intervention (i.e. --force)? Personally, with proper fencing within Pacemaker and a q-device, I would feel “safe enough”, even without fencing within DRBD.

To repeat the same test, I would wait for the nearest available kernel update. Hope it will happen in the next few days.

I have started thinking in the same way, when yesterday I opened drbd conf files.
But there is something that very confused me and I can’t explain that.

Please note, that when qdevice comes back online, cluster with quorum again, but resource groups are still stopped with next status.

[root@memverge2 ~]# drbdadm status
ha-iscsi role:Primary
  volume:3 disk:UpToDate open:no
  volume:4 disk:UpToDate open:no
  memverge connection:Connecting

ha-nfs role:Secondary
  volume:1 disk:Consistent open:no
  volume:2 disk:Consistent open:no
  volume:5 disk:Consistent open:no
  memverge connection:Connecting

There are two drbd resources in different states. Why ha-iscsi in Primary/UpTpDate state, while ha-nfs in Secondary/Consistent state ? I suspect that two drbd resources must have the same states even with the current configuration, both resources in Primary/UpTpDate or both in Secondary/Consistent state, but not in different states.

Or am I wrong ?

Anton

8

Please note, that when qdevice comes back online, cluster with quorum again, but resource groups are still stopped with next status.

I think it’s important to understand that DRBD is configured with only two nodes and has no concept of quorum at all here. Pacemaker does, sure, and it is pacemaker which manages and starts the resource groups. I strongly believe it is DRBD which is preventing Pacemaker from starting the resource-groups in this case though.

While yes, 90% of the time the two different DRBD resources are going to be in the same states. They are still very much two completely independent DRBD resources. They each have their own sets of data and DRBD meta-data. It is not impossible for them to be in two different states like you are observing.

The Consistent disk sate you observe on the ha-nfs is likely because that resource restarted due to Pacemaker trying to recover from the ha-nfs_promote_0 on memverge2 'Error occurred' (1): call=133, status='Timed out' error.

One thing I did notice with the DRBD configuration is the duplicate net {} stanzas in the DRBD configuration. I would expect this to return with a parser error. I am not sure how you this works in your environment, but I would advise removing/commenting one of them.

9

There were no parser errors, because drbdadm adjust all worked without errors. Anyway, I just commented duplicated net {} for two resources.

I also commented fencing resource-and-stonith for two resources and repeat the entire original test without kernel update (as not yet available). All looks good, and I will repeat the test with kernel update as was original test.

But now I’m afraid what behaviour will be, in case of using default fencing dont-care and all DRBD replication links failed, or simply physically unplugged in my test case.

Anton

10

I would go back to this,

Let’s take a look how two drbd resources (ha-nfs and ha-iscsi) were stopping after press “reboot“ on active node (memverge)

[root@memverge2 ~]# dmesg|grep -i nfs
[11945.692871] drbd ha-nfs: Preparing remote state change 1151807023: 27->all role( Secondary )
[11945.718559] drbd ha-nfs memverge: Committing remote state change 1151807023 (primary_nodes=0)
[11945.720185] drbd ha-nfs memverge: peer( Primary -> Secondary ) [remote]
[11945.720367] drbd ha-nfs/1 drbd1: Enabling local AL-updates
[11945.720540] drbd ha-nfs/2 drbd2: Enabling local AL-updates
[11945.720699] drbd ha-nfs/5 drbd5: Enabling local AL-updates
[11945.815936] drbd ha-nfs: Preparing remote state change 2644890067: 27->28 conn( Disconnecting )
[11945.839571] drbd ha-nfs memverge: Committing remote state change 2644890067 (primary_nodes=0)
[11945.839728] drbd ha-nfs memverge: conn( Connected -> TearDown ) peer( Secondary -> Unknown ) [remote]
[11945.839882] drbd ha-nfs/1 drbd1 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [remote]
[11945.840038] drbd ha-nfs/2 drbd2 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [remote]
[11945.840191] drbd ha-nfs/5 drbd5 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [remote]
[11945.864473] drbd ha-nfs memverge: Terminating sender thread
[11945.864667] drbd ha-nfs memverge: Starting sender thread (peer-node-id 27)
[11945.905457] drbd ha-nfs memverge: Connection closed
[11945.905928] drbd ha-nfs memverge: helper command: /sbin/drbdadm disconnected
[11945.929568] drbd ha-nfs memverge: helper command: /sbin/drbdadm disconnected exit code 0
[11945.929724] drbd ha-nfs memverge: conn( TearDown -> Unconnected ) [disconnected]
[11945.929875] drbd ha-nfs memverge: Restarting receiver thread
[11945.930026] drbd ha-nfs memverge: conn( Unconnected -> Connecting ) [connecting]
[11946.118837] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11950.214309] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11950.214543] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11950.245975] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11950.312169] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11950.312452] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11951.335128] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11951.403729] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11951.404040] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11952.425090] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11952.490702] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11952.490963] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11953.513436] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11953.578894] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11953.579208] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11954.602210] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11954.667935] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11954.668564] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11955.690887] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11955.757956] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11955.758186] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11956.779958] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11956.845738] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11956.845973] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11957.867893] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11957.932575] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11957.933067] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11958.954818] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11959.019821] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11959.020014] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11960.043215] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11960.109320] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11960.109531] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11961.131227] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11961.197065] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11961.197427] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11962.219397] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11962.285195] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11962.285798] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11963.307399] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11963.374326] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11963.374911] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11964.397305] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11964.461927] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11964.462120] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11965.483814] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11965.549311] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11965.549852] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11966.570154] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11966.638816] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11966.639028] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11967.661144] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11967.725313] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11967.725516] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11968.748080] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11968.813623] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11968.813825] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11969.836659] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11969.898934] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11969.899137] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11970.920212] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11970.983965] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11970.984190] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11972.007393] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11972.072176] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11972.072797] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11973.097451] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11973.163489] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11973.163780] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11974.185313] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11974.249688] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11974.250224] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11975.272358] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11975.336697] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11975.336898] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11976.359854] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11976.425424] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11976.425637] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11977.448104] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11977.513455] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11977.513928] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11978.535192] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11978.600524] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11978.601133] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11979.623523] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11979.686387] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11979.687015] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11980.708559] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11980.774903] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11980.775478] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11981.797531] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11981.860280] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11981.860909] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11982.883528] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11982.959271] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11982.959779] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11983.981551] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11984.046377] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11984.046820] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11985.067828] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11985.134036] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11985.134239] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11986.155705] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11986.221669] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11986.221867] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11987.243784] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11987.309182] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11987.309698] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11988.331702] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11988.397019] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11988.397641] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11989.419236] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11989.486457] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11989.487022] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11990.509847] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11990.575532] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11990.575744] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11991.597377] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11991.662479] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11991.662916] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11992.686179] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11992.749526] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11992.749735] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11993.771886] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11993.834561] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11993.834742] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11994.856725] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11994.923087] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11994.923678] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11995.945053] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11996.008342] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11996.008660] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11997.030343] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11997.095205] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11997.095726] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11998.118614] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11998.183842] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11998.184053] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[11999.206766] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[11999.271250] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[11999.271448] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12000.292870] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12000.357920] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12000.358154] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12001.379652] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12001.446079] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12001.446319] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12002.467689] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12002.533173] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12002.533386] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12003.555796] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12003.621287] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12003.621533] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12004.643408] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12004.709495] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12004.709692] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12005.732163] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12005.799539] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12005.799783] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12006.822560] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12006.890941] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12006.891380] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12007.912844] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12007.975630] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12007.975976] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12008.997898] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12009.061835] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12009.062075] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12010.082989] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12010.149380] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12010.149565] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12011.171457] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12011.235141] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12011.235331] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12012.256734] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12012.321188] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12012.321551] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12013.343655] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12013.408214] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12013.408403] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12014.430871] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12014.496844] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12014.497334] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12015.519619] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12015.586065] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12015.586641] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12016.607937] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12016.673387] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12016.673910] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12017.696869] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12017.763342] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12017.763834] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12018.784818] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12018.853565] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12018.853778] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12019.875998] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12019.942664] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12019.943195] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12020.965817] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12021.032766] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12021.032966] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12022.055679] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12022.121118] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12022.121310] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12023.143782] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12023.210424] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12023.210888] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12024.232445] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12024.297544] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12024.297722] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12025.320349] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12025.386974] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12025.387131] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12026.410354] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12026.475698] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12026.475937] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12027.498128] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12027.564755] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12027.564969] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12028.586909] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12028.652516] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12028.652766] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12029.675211] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12029.740931] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12029.741498] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12030.763658] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12030.827759] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12030.827964] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12031.849487] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12031.913015] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12031.913611] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12032.936484] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12033.002559] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12033.003150] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12034.025048] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12034.090604] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12034.090774] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12035.113004] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer
[12035.179801] drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 1 (0x100)
[12035.180378] drbd ha-nfs memverge: fence-peer helper broken, returned 1
[12036.144895] drbd ha-nfs memverge: conn( Connecting -> Disconnecting ) [down]
[12036.168147] drbd ha-nfs memverge: Terminating sender thread
[12036.168324] drbd ha-nfs memverge: Starting sender thread (peer-node-id 27)
[12036.298782] drbd ha-nfs memverge: Connection closed
[12036.298982] drbd ha-nfs memverge: helper command: /sbin/drbdadm disconnected
[12036.326641] drbd ha-nfs memverge: helper command: /sbin/drbdadm disconnected exit code 0
[12036.326825] drbd ha-nfs memverge: conn( Disconnecting -> StandAlone ) [disconnected]
[12036.326982] drbd ha-nfs memverge: Terminating receiver thread
[12036.327161] drbd ha-nfs memverge: Terminating sender thread
[12036.354129] drbd ha-nfs/1 drbd1: disk( UpToDate -> Detaching ) [down]
[12036.354825] drbd ha-nfs/1 drbd1: disk( Detaching -> Diskless ) [go-diskless]
[12036.355328] drbd ha-nfs/1 drbd1: drbd_bm_resize called with capacity == 0
[12036.372992] drbd ha-nfs/2 drbd2: disk( UpToDate -> Detaching ) [down]
[12036.373164] drbd ha-nfs/2 drbd2: disk( Detaching -> Diskless ) [go-diskless]
[12036.373777] drbd ha-nfs/2 drbd2: drbd_bm_resize called with capacity == 0
[12036.390916] drbd ha-nfs/5 drbd5: disk( UpToDate -> Detaching ) [down]
[12036.391506] drbd ha-nfs/5 drbd5: disk( Detaching -> Diskless ) [go-diskless]
[12036.392005] drbd ha-nfs/5 drbd5: drbd_bm_resize called with capacity == 0
[12036.436897] drbd /unregistered/ha-nfs: Terminating worker thread
[root@memverge2 ~]#
[root@memverge2 ~]#
[root@memverge2 ~]#
[root@memverge2 ~]#
[root@memverge2 ~]# dmesg|grep -i iscsi
[11944.196868] drbd ha-iscsi: Preparing remote state change 2936933537: 27->all role( Secondary )
[11944.221363] drbd ha-iscsi memverge: Committing remote state change 2936933537 (primary_nodes=0)
[11944.222351] drbd ha-iscsi memverge: peer( Primary -> Secondary ) [remote]
[11944.222572] drbd ha-iscsi/3 drbd3: Enabling local AL-updates
[11944.222781] drbd ha-iscsi/4 drbd4: Enabling local AL-updates
[11944.300839] drbd ha-iscsi: Preparing remote state change 3062745336: 27->28 conn( Disconnecting )
[11944.327344] drbd ha-iscsi memverge: Committing remote state change 3062745336 (primary_nodes=0)
[11944.327546] drbd ha-iscsi memverge: conn( Connected -> TearDown ) peer( Secondary -> Unknown ) [remote]
[11944.327738] drbd ha-iscsi/3 drbd3 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [remote]
[11944.327931] drbd ha-iscsi/4 drbd4 memverge: pdsk( UpToDate -> DUnknown ) repl( Established -> Off ) [remote]
[11944.355559] drbd ha-iscsi memverge: Terminating sender thread
[11944.355792] drbd ha-iscsi memverge: Starting sender thread (peer-node-id 27)
[11944.391922] drbd ha-iscsi memverge: Connection closed
[11944.392116] drbd ha-iscsi memverge: helper command: /sbin/drbdadm disconnected
[11944.418296] drbd ha-iscsi memverge: helper command: /sbin/drbdadm disconnected exit code 0
[11944.418484] drbd ha-iscsi memverge: conn( TearDown -> Unconnected ) [disconnected]
[11944.418665] drbd ha-iscsi memverge: Restarting receiver thread
[11944.418841] drbd ha-iscsi memverge: conn( Unconnected -> Connecting ) [connecting]
[11944.752924] drbd ha-iscsi memverge: helper command: /sbin/drbdadm fence-peer
[11944.909059] drbd ha-iscsi memverge: helper command: /sbin/drbdadm fence-peer exit code 4 (0x400)
[11944.909394] drbd ha-iscsi memverge: fence-peer helper returned 4 (peer was fenced)
[11944.909666] drbd ha-iscsi/3 drbd3 memverge: pdsk( DUnknown -> Outdated ) [primary]
[11944.909885] drbd ha-iscsi/4 drbd4 memverge: pdsk( DUnknown -> Outdated ) [primary]
[11944.910106] drbd ha-iscsi: Preparing cluster-wide state change 724255080: 28->all role( Primary )
[11944.910311] drbd ha-iscsi: Committing cluster-wide state change 724255080 (1ms)
[11944.910500] drbd ha-iscsi: role( Secondary -> Primary ) [primary]
[12036.079654] drbd ha-iscsi: Preparing cluster-wide state change 3642009903: 28->all role( Secondary )
[12036.079843] drbd ha-iscsi: Committing cluster-wide state change 3642009903 (0ms)
[12036.080029] drbd ha-iscsi: role( Primary -> Secondary ) [secondary]
[12036.176450] drbd ha-iscsi memverge: conn( Connecting -> Disconnecting ) [down]
[12036.176787] drbd ha-iscsi memverge: Terminating sender thread
[12036.177062] drbd ha-iscsi memverge: Starting sender thread (peer-node-id 27)
[12036.196634] drbd ha-iscsi memverge: Connection closed
[12036.196810] drbd ha-iscsi memverge: helper command: /sbin/drbdadm disconnected
[12036.198023] drbd ha-iscsi memverge: helper command: /sbin/drbdadm disconnected exit code 0
[12036.198187] drbd ha-iscsi memverge: conn( Disconnecting -> StandAlone ) [disconnected]
[12036.198348] drbd ha-iscsi memverge: Terminating receiver thread
[12036.198522] drbd ha-iscsi memverge: Terminating sender thread
[12036.240919] drbd ha-iscsi/3 drbd3: disk( UpToDate -> Detaching ) [down]
[12036.241660] drbd ha-iscsi/3 drbd3: disk( Detaching -> Diskless ) [go-diskless]
[12036.242373] drbd ha-iscsi/3 drbd3: drbd_bm_resize called with capacity == 0
[12036.267039] drbd ha-iscsi/4 drbd4: disk( UpToDate -> Detaching ) [down]
[12036.267695] drbd ha-iscsi/4 drbd4: disk( Detaching -> Diskless ) [go-diskless]
[12036.268337] drbd ha-iscsi/4 drbd4: drbd_bm_resize called with capacity == 0
[12036.299882] drbd /unregistered/ha-iscsi: Terminating worker thread

Could it be that during concurrent fence requests and CIB updates there is locking/race condition between two DRBD resources which are executing “/sbin/drbdadm fence-peer“ calling /usr/lib/drbd/crm-fence-peer.9.sh handler ? I have just played with AI describing initial request, so the generated “solution” is to modify /usr/lib/drbd/crm-fence-peer.9.sh handler.

Next is generated by AI

“The fix is to introduce an early check to see if the peer node is already marked as dead or fenced by the cluster, which would happen when the first resource (ha-iscsi) successfully runs the script. If the peer is already deemed fenced, the second resource (ha-nfs) must immediately return success (Exit Code 4) without attempting to place its own constraint.

Target Function and Location

Modify the function drbd_peer_fencing() around line 900. Insert the new check after the initial variable setup and before the main constraint logic.

case $1 in
	fence)

		local startup_fencing stonith_enabled
		check_cluster_properties
		setup_crm_timeout_unit_ms

        # === START OF PERMANENT FIX ===
        # Use existing function to check if the peer is already marked clean/fenced.
        check_peer_node_reachable
        if $all_excluded_peers_fenced ; then
            echo INFO "Concurrency check: Peer is already marked clean/fenced by another resource. Returning success (Exit 4)."
            drbd_fence_peer_exit_code=4
            rc=0
            return $rc
        fi
        # === END OF PERMANENT FIX ===

		if ! $had_constraint_on_entry ; then
            # ... existing logic ...

Explanation of the Fix

  1. check_peer_node_reachable: This existing script function polls the CIB and determines the status of the peer node(s).

  2. $all_excluded_peers_fenced: This variable is set by the function above. It returns true if the cluster manager (Pacemaker) considers the peer to be cleanly offline or fenced (state is likely expected="down" and in_ccm="false").

  3. Idempotency: By checking this status first, when ha-nfs runs:

    • It sees that ha-iscsi already caused Pacemaker to declare the peer dead/fenced.

    • It logs a message, sets its internal exit code (drbd_fence_peer_exit_code=4), and returns 0 (success) immediately.

    • This satisfies DRBD’s requirement for a fencing handler (Peer Fenced), allowing ha-nfs to become primary without causing the script to fail.”

I’m far from thinking that it is the best approach, but at least it works.

It might be useful for the real solution.

It also allows use together with a pacemaker qdevice and DRBD “fencing resource-and-stonith” achieving highest possible data consistency protection on all levels at any given time.

Anton