Has anyone seen this problem before? After upgrading a node we have some resources which won’t connect, and this is because two nodes have “cram-hmac-alg” and “shared-secret” set, but the third (most recently updated one) does not have those settings configured on the resources.
root@virtual1:~# ls /var/lib/linstor.d/*.res | wc -l
17
root@virtual1:~# grep shared-secret /var/lib/linstor.d/*.res | wc -l
14
root@virtual5:~# ls /var/lib/linstor.d/*.res | wc -l
17
root@virtual5:~# grep shared-secret /var/lib/linstor.d/*.res | wc -l
14
root@virtual6:~# ls /var/lib/linstor.d/*.res | wc -l
17
root@virtual6:~# grep shared-secret /var/lib/linstor.d/*.res | wc -l
0
DETAILS
- virtual1 is not yet upgraded, still on PVE8 + drbd 9.2.15
- virtual5 was upgraded yesterday, now on PVE9 with drbd-dkms 9.3.4-1 under kernel 7.0.14-22-pve
- virtual6 upgraded today, now on PVE9 with drbd-dkms 9.3.4-1 under kernel 7.0.14-23-pve (note that Proxmox released a one click upgrade between the two)
There are no primaries on virtual6 - everything was shunted off prior to upgrade.
Problem observed: lots of resources won’t connect. Specifically, those which are still primary on virtual1:
root@virtual6:~# drbdadm status linstor_db
linstor_db role:Secondary
disk:Outdated quorum:no open:no
virtual1 connection:Connecting
virtual5 connection:Connecting
I get this kernel message on the primary:
[31703144.418159] drbd linstor_db virtual6: conn( Unconnected -> Connecting ) [connecting]
[31703156.741598] drbd linstor_db/0 drbd1002: new current UUID: 5B055C6F1206E6C9 weak: FFFFFFFFFFFFFFFC
[31703288.167071] drbd linstor_db virtual6: Handshake to peer 2 successful: Agreed network protocol version 123
[31703288.167429] drbd linstor_db virtual6: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
[31703288.176696] drbd linstor_db virtual6: expected AuthChallenge packet, received: P_PROTOCOL (0x000b)
[31703288.176958] drbd linstor_db virtual6: Authentication of peer failed
[31703288.177212] drbd linstor_db virtual6: conn( Connecting -> Disconnecting )
which led me to this post (from 2013!) which hinted the problem is mismatched auth configuration, and indeed this is the case:
root@virtual1:~# drbdadm dump linstor_db
...
options {
auto-promote no;
on-no-data-accessible io-error;
on-no-quorum io-error;
on-suspended-primary-outdated force-secondary;
quorum majority;
}
net {
cram-hmac-alg sha1;
shared-secret XXXXXXXX;
verify-alg sha256;
}
}
root@virtual5:~# drbdadm dump linstor_db
...
options {
auto-promote no;
on-no-data-accessible io-error;
on-no-quorum io-error;
on-suspended-primary-outdated force-secondary;
quorum majority;
}
net {
cram-hmac-alg sha1;
shared-secret XXXXXXXX;
verify-alg sha256;
}
}
root@virtual6:~# drbdadm dump linstor_db
...
options {
auto-promote no;
on-no-data-accessible io-error;
on-no-quorum io-error;
on-suspended-primary-outdated force-secondary;
quorum majority;
}
net {
verify-alg sha256;
}
}
Note the missing “cram-hmac-alg” and “shared-secret” on virtual6.
But those which are primary on virtual5 do connect, and these have matching resource definitions across all three hosts (all without authentication)
root@virtual1:~# drbdadm dump pm-eb40ca57
...
options {
on-no-quorum io-error;
quorum majority;
}
net {
allow-two-primaries yes;
verify-alg sha256;
}
}
root@virtual5:~# drbdadm dump pm-eb40ca57
...
options {
on-no-quorum io-error;
quorum majority;
}
net {
allow-two-primaries yes;
verify-alg sha256;
}
}
root@virtual6:~# drbdadm dump pm-eb40ca57
...
options {
on-no-quorum io-error;
quorum majority;
}
net {
allow-two-primaries yes;
verify-alg sha256;
}
}
The only thing which I can think of which might be related is:
-
Yesterday I upgraded virtual5
-
Linstor was failing to configure drbd resources at all, because it was trying to use the “crct10dif” verify algorithm which is no longer available - see thread
-
I gave the following commands:
linstor c sp DrbdOptions/auto-verify-algo-disable true linstor c opt --verify-alg sha256and drbd sprang back into life, or so I thought (actually I ended up with corruption after live migrating some VMs and had to restore them from backups, but that’s a different story).
That is, linstor pushed out the new drbd configs immediately via the satellites. But I did not reboot virtual5 after this. In any case, this is a verify algorithm change, not authentication algorithm.
Then there is the kernel difference between 7.0.14-22-pve and 7.0.14-23-pve but that seems unlikely.
linstor versions are pinned to exactly the same everywhere (of course, since the satellite won’t talk to a different version controller)
root@virtual6:~# dpkg-query -l | grep -v ^ii
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-====================================-====================================-============-================================================================================
hi drbd-dkms 9.3.4-1 all RAID 1 over TCP/IP for Linux module source
hi drbd-reactor 1.12.0-1 amd64 Monitors DRBD resources via plugins.
hi drbd-utils 9.35.0-1 amd64 RAID 1 over TCP/IP for Linux (user utilities)
hi linstor-common 1.32.3-1 all DRBD distributed resource management utility
hi linstor-controller 1.32.3-1 all DRBD distributed resource management utility
hi linstor-satellite 1.32.3-1 all DRBD distributed resource management utility
hi openjdk-17-jre-headless:amd64 17.0.16+8-1~deb12u1 amd64 OpenJDK Java runtime, using Hotspot JIT (headless)
(Oddly, there is no tag v1.32.3 in the linstor-server repository: there are v1.32.2 and v1.33.0)
