Cannot recover drdb volume - drbdadm primary --force doesn't work

DRBD
, ,
1

I have a drbd volume which won’t come up after a reboot

image
image1910×135 11.7 KB

I am trying to tell drbd to select the volume on 1 of the nodes as the “source of truth” but I am unable to.

root@node02.cloud10:~# drbdadm primary --force volume-0eb1b3c1-13d3-4d72-9b07-15ebc5194a3a
volume-0eb1b3c1-13d3-4d72-9b07-15ebc5194a3a: State change failed: (-2) Need access to UpToDate data
Command 'drbdsetup primary volume-0eb1b3c1-13d3-4d72-9b07-15ebc5194a3a --force' terminated with exit code 17

How can I recover from this split-brain situation?

2

All your nodes are in a Diskless state for that resource.

Is the backing storage for the resource present and healthy?

linstor storage-pool list
3

Looking OK

(vqcp) root@node01:# linstor storage-pool list
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool          ┊ Node                           ┊ Driver   ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName                                          ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ node01.xyz ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊ node01.xyz;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ node02.xyz ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊ node02.xyz;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ node03.xyz ┊ DISKLESS ┊          ┊              ┊               ┊ False        ┊ Ok    ┊ node03.xyz;DfltDisklessStorPool ┊
┊ pool1                ┊ node01.xyz ┊ ZFS_THIN ┊ pool1    ┊     1.22 TiB ┊     10.47 TiB ┊ True         ┊ Ok    ┊ node01.xyz;pool1                ┊
┊ pool1                ┊ node02.xyz ┊ ZFS_THIN ┊ pool1    ┊     1.09 TiB ┊     10.47 TiB ┊ True         ┊ Ok    ┊ node02.xyz;pool1                ┊
┊ pool1                ┊ node03.xyz ┊ ZFS_THIN ┊ pool1    ┊     3.36 TiB ┊     10.47 TiB ┊ True         ┊ Ok    ┊ node03.xyz;pool1                ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
4

You could try a mass adjust on the resource-group those resources belong to:

linstor resource-group adjust <rg-name>

Running a drbdadm adjust all on node2 and node3 might give you more hints as to what’s going on.

Either way, none of your node’s DRBD devices are attached to their local backing storage (all in a Diskless state). This isn’t a split-brain based on what’s been shared so far.