# DRBD quorum doesn't work with kind of disconnection

**URL:** <https://forums.linbit.com/t/drbd-quorum-doesnt-work-with-kind-of-disconnection/95>\
**Category:** DRBD\
**Created:** [May 16, 2024, 10:21am UTC](https://forums.linbit.com/t/drbd-quorum-doesnt-work-with-kind-of-disconnection/95 "2024-05-16T10:21:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![PetrCerny](https://avatars.discourse-cdn.com/v4/letter/p/a5b964/32.png) [@PetrCerny](https://forums.linbit.com/u/PetrCerny)\
**Post date:** [May 16, 2024, 10:21am UTC](https://forums.linbit.com/t/drbd-quorum-doesnt-work-with-kind-of-disconnection/95/1 "2024-05-16T10:21:44Z")

</div>

I’m new to DRBD, so maybe it’s just ignorance, but I ran into a problem with using quorum.  
My environment. three AlmaLinux 9.4 servers, kmod-drbd9x-9.1.20-1.el9\_4.elrepo, drbd9x-utils-9.28.0-1.el9.elrepo. Two servers have disks, the third is diskless. My configuration.

```auto
resource "md0r5" {
   device minor 1;
   disk "/dev/md0";
   meta-disk internal;

   net {
     protocol C;
     transport "tcp";
     load-balance-paths yes;
   }
options {
     majority quorum;
}

on "pcs1" {
     node-id 0;
   }
   on "pcs2" {
     node-id 1;
   }
   on "pcs3" {
     disk none;
     node-id 2;
   }

   connection {
     path {
       host pcs1 address 192.168.18.11:7789;
       host pcs2 address 192.168.18.21:7789;
     }
     path {
       host pcs1 address 192.168.19.11:7789;
       host pcs2 address 192.168.19.21:7789;
     }
   }
   connection {
     path {
       host pcs1 address 192.168.18.11:7789;
       host pcs3 address 192.168.18.31:7789;
     }
     path {
       host pcs1 address 192.168.19.11:7789;
       host pcs3 address 192.168.19.31:7789;
     }
   }
   connection {
     path {
       host pcs2 address 192.168.18.21:7789;
       host pcs3 address 192.168.18.31:7789;
     }
     path {
       host pcs2 address 192.168.19.21:7789;
       host pcs3 address 192.168.19.31:7789;
     }
   }
}

```

After starting DRBD, everything is online. If I disconnect the network of one server (both connections), the cluster maintains quorum and everything works. If I disconnect another one, the system reacts as I would expect, losing quorum and not allowing further changes to the disk. The problem occurs when I reconnect the system. The cluster reports that everything is OK, the disks are UpToDate and all systems are connected. When I disconnect the two systems again, the cluster no longer loses quorum and disk writes are enabled.

Is this standard behavior? Can I somehow influence it by changing the configuration?

Thank you

---

<div class="post-metadata">

**Author:** ![ionescu77](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/ionescu77/32/40_2.png) [@ionescu77](https://forums.linbit.com/u/ionescu77)\
**Post date:** [May 17, 2024, 4:48pm UTC](https://forums.linbit.com/t/drbd-quorum-doesnt-work-with-kind-of-disconnection/95/2 "2024-05-17T16:48:26Z")

</div>

Hi,

3y ago I’ve setup a Centos 7.9 apache/postgresql cluster with corosync/pacemaker & DRBD.  
DRBD was used to create a distributed filesystem to store the www and the PG `data/` and corosync/pacemaker to start apache and postgresql on the back-up node.

I’ve had a hard time to get DRBD in the free version (client had no budget) to play nice. Because the free version did not handle async and I had network delays in the LAN.

If it helps:  
I remember there were some scripts that I’ve had to tweak, in order to handle fail-over events and to get alerts if it would get into split brain state (if nodes failed and only one node was left). In souch cases recovery should be a manual process.  
But the client insisted they wanted automatic recovery.

I remember those helper scripts (`handlers`) were not included anymore in more recent versions of DRBD.

> in resource cfg file:

```auto
handlers {
  split-brain "/usr/lib/drbd/notify-split-brain.sh root";
}
# ...
net {
  protocol C;
  after-sb-0pri discard-zero-changes;
  after-sb-1pri discard-secondary;
  after-sb-2pri call-pri-lost-after-sb;
}

```

This was the location on Centos 7.9 with DRBD install via normal repos (with yum).

```auto
ls -ltr /usr/lib/drbd/notify-out-of-sync.sh
lrwxrwxrwx 1 root root 9 Nov 1 14:18 /usr/lib/drbd/notify-out-of-sync.sh -> notify.sh

```

Notice it was a symlink to the `notify.sh` script.

Hope it points you in some direction.

Good luck and let me know, I am curious about people experiences with DRBD in production.

---

<div class="post-metadata">

**Author:** ![Ryan](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/ryan/32/42_2.png) [@Ryan](https://forums.linbit.com/u/Ryan)\
**Post date:** [May 17, 2024, 6:43pm UTC](https://forums.linbit.com/t/drbd-quorum-doesnt-work-with-kind-of-disconnection/95/3 "2024-05-17T18:43:10Z")

</div>

Hi there,

At first glance, the `majority quorum;` is an incorrect setting. I’m guessing your cluster never had quorum configured? You can verify this with `drbdsetup show`, if you want to see the default settings also in use, run `drbdsetup show --show-defaults` instead.

```plaintext
options {
    majority quorum;
}

```

Should be (taken from [this section of the UG](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-force-secondary)):

```plaintext
options {
    quorum majority;
    on-no-quorum suspend-io;
    on-no-data-accessible suspend-io;
    on-suspended-primary-outdated force-secondary;
}

```

Keep in mind that the order of disconnecting/reconnecting nodes will affect quorum in different ways depending on diskless/diskful nodes, graceful vs non-graceful disconnection (think unplugging the ethernet cable versus running `drbdadm disconnect <res>`, etc. For example, connecting a _Diskful_ node that has lost quorum to a _Diskless/Tiebreaker_ node will not restore quorum. These scenarios are well documented [here](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-configuring-quorum-tiebreaker).

After making changes to your DRBD resource config (run `drbdadm adjust <res>` on each node after changes), quorum should function as expected. If not, we can dig into it further.

---

<div class="post-metadata">

**Author:** ![Ryan](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/ryan/32/42_2.png) [@Ryan](https://forums.linbit.com/u/Ryan)\
**Post date:** [May 17, 2024, 6:49pm UTC](https://forums.linbit.com/t/drbd-quorum-doesnt-work-with-kind-of-disconnection/95/4 "2024-05-17T18:49:28Z")

</div>

Just note that we never really recommend automatic split-brain recovery in production environments.

The DRBD User’s Guide says it best:

> You rather want to look into fencing policies, quorum settings, cluster manager integration, and redundant cluster manager communication links to **avoid** data divergence in the first place.

---

<div class="post-metadata">

**Author:** ![phil\_reisner](https://avatars.discourse-cdn.com/v4/letter/p/779978/32.png) [@phil\_reisner](https://forums.linbit.com/u/phil_reisner)\
**Post date:** [May 22, 2024, 8:59am UTC](https://forums.linbit.com/t/drbd-quorum-doesnt-work-with-kind-of-disconnection/95/5 "2024-05-22T08:59:51Z")

</div>

Oh, you have a three-node setup, with one diskless “tiebreaker” node and a redundant network. Nice. Keep in mind that as long as one path works, the complete connection is considered healthy.  
See the [documentation on quorum tiebreaker](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-configuring-quorum-tiebreaker) on when DRBD keeps quorum. Please also note that it makes a difference if you gracefully disconnect a node (with `drbdadm disconnect res:peer_name`) or if a connection is lost unexpectedly. See the documentation on [last man standing](https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/#s-quorum_last_man_standing).