I’ve now finished updating a three-node Proxmox+linstor/drbd cluster, which is now on PVE 9.2.21, drbd 9.3.4, linstor 1.35.2
At the end of this upgrade, I needed to reboot node virtual5 to go from kernel 7.0.14-22-pve to 7.0.14-23-pve (Proxmox have released a new kernel in the last couple of days). I moved all the VMs off and rebooted it.
However, one of the volumes, which is primary on virtual6 and secondary on virtual1 and virtual5, has gone slightly out of sync. dmesg on virtual6 showed the following when virtual5 reconnected:
[202913.274911] drbd pm-4b8bd822/0 drbd1017 virtual5: expected n_oos:93 to be 0
which was also reflected in Prometheus metrics and drbdadm status:
drbd_peerdevice_outofsync_bytes{conn_name="virtual5", instance="virtual6.nsrc.org", job="linstor-node", name="pm-4b8bd822", node="virtual6", peer_node_id="1", volume="0"} 380928
brian@virtual6:~$ sudo drbdadm status pm-4b8bd822 --statistics
pm-4b8bd822 role:Primary
disk:UpToDate open:yes
read:60262872 written:27087784
virtual1 role:Secondary
peer-disk:UpToDate
received:0 sent:915403
virtual5 role:Secondary
peer-disk:UpToDate
received:0 sent:357467 out-of-sync:372
(93 x 4K blocks = 372 x 1K blocks = 380928 bytes)
I left it overnight, and this morning did an online verify. The number of out-of-sync blocks has reduced from 93 to 71, but it’s still clearly not in sync.
[243430.785626] drbd pm-4b8bd822/0 drbd1017 virtual5: Online verify done (total 596 sec; paused 0 sec; 87976 K/sec)
[243430.785936] drbd pm-4b8bd822/0 drbd1017 virtual5: Online verify found 71 4k blocks out of sync!
Questions
- What’s my next step to resolve this safely? I am thinking that I’d probably need to do
drbdadm invalidateordrbdadm outdateon the secondary (virtual5) but I’m not sure of the difference between these commands. - Is there something that I should have done to prevent this? For example, should I do
drbdadm down allon a node just before rebooting it? Or are the systemd units indrbd-utilsdoing this already? - Should this have happened at all, given that drbd on both nodes is the latest 9.3.4? Is it worth digging down further into the underlying cause?
Additional info
Full kernel logs from both the reboot (disconnect/reconnect) and the online verify are here: https://gist.github.com/candlerb/ce5d5dd7d904773bf53cea8ec9f6a4f9
Oddly, I’m seeing no messages of the form “Out of sync: start=XXX, size=YYY (sectors)” which means I can’t do a manual check of the affected blocks as per this thread.
General config:
brian@virtual1:~$ linstor v l -r pm-4b8bd822
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Resource │ Node │ StoragePool │ VolNr │ MinorNr │ DeviceName │ Allocated │ InUse │ State │ Repl │
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
│ pm-4b8bd822 │ virtual1 │ sata_ssd │ 0 │ 1017 │ /dev/drbd1017 │ 50.02 GiB │ Unused │ UpToDate │ Established(2) │
│ pm-4b8bd822 │ virtual5 │ sata_ssd │ 0 │ 1017 │ /dev/drbd1017 │ 50.02 GiB │ Unused │ UpToDate │ Established(2) │
│ pm-4b8bd822 │ virtual6 │ sata_ssd │ 0 │ 1017 │ /dev/drbd1017 │ 50.02 GiB │ InUse │ UpToDate │ Established(2) │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
brian@virtual5:~$ uname -a
Linux virtual5 7.0.14-23-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-23 (2026-10-07T20:25Z) x86_64 GNU/Linux
brian@virtual5:~$ cat /sys/module/drbd/version
9.3.4
brian@virtual5:~$ dpkg-query -l | egrep -i 'drbd|linstor'
hi drbd-dkms 9.3.4-1 all RAID 1 over TCP/IP for Linux module source
hi drbd-reactor 1.12.0-1 amd64 Monitors DRBD resources via plugins.
hi drbd-utils 9.35.0-1 amd64 RAID 1 over TCP/IP for Linux (user utilities)
ii linstor-client 1.29.1-1 all Linstor client command line tool
hi linstor-common 1.35.2-1 all DRBD distributed resource management utility
hi linstor-controller 1.35.2-1 all DRBD distributed resource management utility
ii linstor-gui 2.6.0-1 all Administration GUI for LINSTOR clusters.
ii linstor-proxmox 8.3.2-1 all DRBD distributed resource management utility
hi linstor-satellite 1.35.2-1 all DRBD distributed resource management utility
ii python-linstor 1.29.1-1 all Linstor python api library
brian@virtual5:~$ sudo drbdadm dump pm-4b8bd822
# resource pm-4b8bd822 on virtual5: not ignored, not stacked
# defined at /var/lib/linstor.d/pm-4b8bd822.res:10
resource pm-4b8bd822 {
on virtual5 {
node-id 1;
volume 0 {
disk {
discard-granularity 4096;
discard-zeroes-if-aligned no;
rs-discard-granularity 4096;
}
device minor 1017;
disk /dev/sata_ssd/pm-4b8bd822_00000;
meta-disk internal;
tiebreaker yes;
}
}
on virtual1 {
node-id 0;
volume 0 {
disk {
discard-granularity 4096;
discard-zeroes-if-aligned no;
rs-discard-granularity 4096;
}
device minor 1017;
disk /dev/drbd/this/is/not/used;
meta-disk internal;
tiebreaker yes;
}
}
on virtual6 {
node-id 2;
volume 0 {
disk {
discard-granularity 4096;
discard-zeroes-if-aligned no;
rs-discard-granularity 4096;
}
device minor 1017;
disk /dev/drbd/this/is/not/used;
meta-disk internal;
tiebreaker yes;
}
}
connection {
host virtual5 address ipv4 10.0.0.5:7017;
host virtual1 address ipv4 10.0.0.1:7017;
net {
_name virtual1;
allow-two-primaries yes;
verify-alg sha256;
}
}
connection {
host virtual5 address ipv4 10.0.0.5:7017;
host virtual6 address ipv4 10.0.0.6:7017;
net {
_name virtual6;
allow-two-primaries yes;
verify-alg sha256;
}
}
options {
on-no-quorum io-error;
quorum majority;
}
net {
cram-hmac-alg sha1;
shared-secret XXXXXXXX;
allow-two-primaries yes;
verify-alg sha256;
}
}
brian@virtual6:~$ uname -a
Linux virtual6 7.0.14-23-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-23 (2026-10-07T20:25Z) x86_64 GNU/Linux
brian@virtual6:~$ cat /sys/module/drbd/version
9.3.4
brian@virtual6:~$ dpkg-query -l | egrep -i 'drbd|linstor'
hi drbd-dkms 9.3.4-1 all RAID 1 over TCP/IP for Linux module source
hi drbd-reactor 1.12.0-1 amd64 Monitors DRBD resources via plugins.
hi drbd-utils 9.35.0-1 amd64 RAID 1 over TCP/IP for Linux (user utilities)
ii linstor-client 1.29.1-1 all Linstor client command line tool
hi linstor-common 1.35.2-1 all DRBD distributed resource management utility
hi linstor-controller 1.35.2-1 all DRBD distributed resource management utility
ii linstor-gui 2.6.0-1 all Administration GUI for LINSTOR clusters.
ii linstor-proxmox 8.3.2-1 all DRBD distributed resource management utility
hi linstor-satellite 1.35.2-1 all DRBD distributed resource management utility
ii python-linstor 1.29.1-1 all Linstor python api library
brian@virtual6:~$ sudo drbdadm dump pm-4b8bd822
# resource pm-4b8bd822 on virtual6: not ignored, not stacked
# defined at /var/lib/linstor.d/pm-4b8bd822.res:10
resource pm-4b8bd822 {
on virtual6 {
node-id 2;
volume 0 {
disk {
discard-granularity 4096;
discard-zeroes-if-aligned no;
rs-discard-granularity 4096;
}
device minor 1017;
disk /dev/sata_ssd/pm-4b8bd822_00000;
meta-disk internal;
tiebreaker yes;
}
}
on virtual1 {
node-id 0;
volume 0 {
disk {
discard-granularity 4096;
discard-zeroes-if-aligned no;
rs-discard-granularity 4096;
}
device minor 1017;
disk /dev/drbd/this/is/not/used;
meta-disk internal;
tiebreaker yes;
}
}
on virtual5 {
node-id 1;
volume 0 {
disk {
discard-granularity 4096;
discard-zeroes-if-aligned no;
rs-discard-granularity 4096;
}
device minor 1017;
disk /dev/drbd/this/is/not/used;
meta-disk internal;
tiebreaker yes;
}
}
connection {
host virtual6 address ipv4 10.0.0.6:7017;
host virtual1 address ipv4 10.0.0.1:7017;
net {
_name virtual1;
allow-two-primaries yes;
verify-alg sha256;
}
}
connection {
host virtual6 address ipv4 10.0.0.6:7017;
host virtual5 address ipv4 10.0.0.5:7017;
net {
_name virtual5;
allow-two-primaries yes;
verify-alg sha256;
}
}
options {
on-no-quorum io-error;
quorum majority;
}
net {
cram-hmac-alg sha1;
shared-secret XXXXXXXX;
allow-two-primaries yes;
verify-alg sha256;
}
}
I also note that Linstor did produce some error reports at the time that virtual5 was rebooted:
brian@virtual1:~$ linstor error-reports list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Id │ Datetime │ Node │ Exception │
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
│ 6ACBBCF7-B63BE-000000 │ 2026-10-11 18:33:26 │ S|virtual5 │ ResourceException: Failed to adjust DRBD resource linstor_db │
│ 6ACBBCF7-B63BE-000001 │ 2026-10-11 18:33:27 │ S|virtual5 │ ResourceException: Failed to adjust DRBD resource pm-4628dd67 │
│ 6ACBBCF7-B63BE-000002 │ 2026-10-11 18:33:28 │ S|virtual5 │ StorageException: Failed to query symlinks of device /dev/drbd1013 │
│ 6ACBBCF7-B63BE-000003 │ 2026-10-11 18:33:28 │ S|virtual5 │ StorageException: Failed to query symlinks of device /dev/drbd1008 │
│ 6ACBBCF7-B63BE-000004 │ 2026-10-11 18:33:28 │ S|virtual5 │ StorageException: Failed to query symlinks of device /dev/drbd1009 │
│ 6ACBBCF7-B63BE-000005 │ 2026-10-11 18:33:28 │ S|virtual5 │ StorageException: Failed to query symlinks of device /dev/drbd1011 │
│ 6ACBBCF7-B63BE-000006 │ 2026-10-11 18:33:29 │ S|virtual5 │ StorageException: Failed to query symlinks of device /dev/drbd1018 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
However, I presume these were transient as the satellite came online: for example, the missing /dev/drbdXXXX devices are now present. And the two ResourceException resources are not the one which has gone out of sync. So I don’t think this is related.
Aside: it would be nice if Linstor could report on out-of-sync volumes, and/or schedule periodic online verify - if it already can, I’ve not found it. I only see a mention in passing the manual: “Perhaps you have to run nightly online verifications of your volumes’ consistency” which talks about configuring verify-alg.