DRBD 9.3.4 bug?: diskless witness cannot rejoin a dual-Primary resource, both Primaries decline the connect (rv -10) in a loop

DRBD
1

We hit a situation where a diskless quorum witness could not reconnect to two resources for over 20 minutes, although the network was fine again and TLS/HMAC handshakes succeeded. Each Primary kept declining the other Primary’s cluster-wide state change for that connection. Demoting one of the two Primaries made the witness connect within 3 seconds. I would like to understand whether this is expected, a known issue, or a bug, and what the declining side is checking.

Versions

  • DRBD 9.3.4 (DKMS), drbd-utils 9.35.0, kernel 7.2.6 (Debian 13 + backports)
  • LINSTOR 1.35.2 (controller under Pacemaker), Pacemaker 3.0, corosync 3 with qdevice

Topology (3 nodes, per resource)

  • SAN1 (node-id 0) and SAN2 (node-id 1): diskful, both Primary (allow-two-primaries yes, protocol C)
  • SAN-TIEBREAKER (node-id 2): diskless, Secondary, never opened; there for quorum only
  • Options: quorum majority, on-no-quorum suspend-io, on-suspended-primary-outdated force-secondary, fencing resource-and-stonith (custom fence-peer handler), auto-promote no, tls yes (kTLS via tlshd) and cram-hmac-alg sha256 on all connections
  • A third resource with the same three nodes but a single Primary (LINSTOR controller database, no fencing) behaved normally: it reconnected the witness by itself.

What happened

  1. 22:11:44: the witness became unreachable for both Primaries (an IP address conflict on the management network: another machine answered on the witness’s address for about 14 minutes, so connection attempts and TLS handshakes partly reached the wrong host). SAN1 and SAN2 stayed connected to each other, kept quorum (2 of 3) and kept writing.
  2. From 22:12:40 on, every attempt to reconnect the witness was declined, on both resources that are dual-Primary, about once per second, until 22:35:40 (SAN1 logged 1935 declines, SAN2 1819).
  3. After the address conflict was gone (about 22:25), TCP, TLS and the DRBD handshake to the witness succeeded each time, but the cluster-wide state change was then declined:

SAN1 (initiating the connect to the witness):

22:24:00.296268 drbd lun01: Preparing cluster-wide state change 4079589166: 0->2 role( Primary ) conn( Connected )
22:24:00.304272 drbd lun01/0 drbd1000 SAN-TIEBREAKER: self 08D47CA1B40A8EA5:33D6BF045BF35EFD:7182C6D4C3CF717C:94F292AAA97BC53A bits:0 flags:0
22:24:00.304322 drbd lun01/0 drbd1000 SAN-TIEBREAKER: peer's exposed UUID: 08D47CA1B40A8EA4
22:24:00.304343 drbd lun01: Declined by peer SAN2 (id: 1), see the kernel log there
22:24:00.304361 drbd lun01: Aborting cluster-wide state change 4079589166 (8ms) rv = -10

SAN2, at the same time, initiating its own connect to the witness:

22:24:00.090565 drbd lun01: Preparing cluster-wide state change 2102150485: 1->2 role( Primary ) conn( Connected )
22:24:00.098662 drbd lun01: Declined by peer SAN1 (id: 0), see the kernel log there
22:24:00.098677 drbd lun01: Aborting cluster-wide state change 2102150485 (8ms) rv = -10

SAN1’s log for SAN2’s change (the declining side) gives no reason:

22:24:00.088270 drbd lun01: Preparing remote state change 2102150485: 1->2 role( Primary ) conn( Connected )
22:24:00.096309 drbd lun01 SAN2: Aborting remote state change 2102150485

The witness:

22:24:00.087787 drbd lun01: Preparing remote state change 2102150485: 1->2 role( Primary ) conn( Connected )
22:24:00.095811 drbd lun01/0 drbd1000 SAN2: my exposed UUID: 08D47CA1B40A8EA4
22:24:00.095917 drbd lun01 SAN2: Aborting remote state change 2102150485

Then Connect keeps failing (State change was refused by peer node); retrying after connect-int, and the same again. Earlier in the outage the witness exposed 33D6BF045BF35EFC (the Primaries’ bitmap UUID) instead.

What did not help

  • drbdsetup disconnect of SAN2’s connection to the witness, so that only SAN1 tried: SAN2 was StandAlone towards the witness from 22:26:21.8 to 22:26:42.6, and SAN1’s attempts in that window were still Declined by peer SAN2 (id: 1) (22:26:22.064, 22:26:23.096, …).
  • drbdadm down / up of the resource on the witness (diskless, not open).

What helped

Demoting one Primary per resource for a moment (Pacemaker ban on the non-preferred node). With a single Primary the witness connected within 3 s; promoting the second node again afterwards worked normally and all connections stayed up. Online verify afterwards: no out-of-sync blocks.

Questions

  1. What does a Primary check when it declines another Primary’s state change that connects a diskless peer? Nothing is logged on the declining side, even at the default log level.
  2. Is a diskless peer that was away while two Primaries kept writing (and whose exposed UUID went from the bitmap UUID to the current UUID) expected to be refused until only one Primary is left?
  3. Is there a better way to let it rejoin than demoting a Primary, which moves host I/O off the optimized path for a moment?

I have not yet reproduced it in a clean setup without the address conflict; I can do that and collect drbdsetup events2 --timestamps --statistics or debug logs if that helps. Full kernel logs from all three nodes for 22:05-22:45 are available.

Thanks!