I managed to run into this myself, and Iโm not sure what caused it - similarly, this used to work fine. Basically, everything looks good, no errors anywhere, but proxmox will not promote a volume to primary for itself. This affects both migrations (migrations, or rather the start step of them, will always fail unless you drbdadm primary the volume on the target node first, ideally followed by drbdadm secondary on the source node), and creation of new disks (the whole thing gets created properly, but is left on all secondaries, so the attach to the VM fails with the same โWrong medium typeโ failure until you set primary). This affects every volume on every resource.
I completed a reboot of each node (one-by-one for no loss of service), no help.
Additional outputs, for clarity cut to a single volume, but all are similar:
linstor resource list
:
โ pm-555df3d0 โ vbt-paw1 โ DRBD,STORAGE โ Unused โ Ok โ TieBreaker โ 2024-12-18 13:25:36 โ
โ pm-555df3d0 โ vbt-prox1 โ DRBD,STORAGE โ InUse โ Ok โ UpToDate โ 2024-12-17 12:11:35 โ
โ pm-555df3d0 โ vbt-prox2 โ DRBD,STORAGE โ Unused โ Ok โ UpToDate โ 2024-12-17 12:11:35 โ
drbdadm status
on the source, target says exactly what youโd expect:
pm-555df3d0 role:Primary
disk:UpToDate open:yes
vbt-paw1 role:Secondary
peer-disk:Diskless
vbt-prox2 role:Secondary
peer-disk:UpToDate
drbdsetup show
on source:
resource "pm-555df3d0" {
options {
auto-promote no;
quorum majority;
on-no-quorum io-error;
}
_this_host {
node-id 0;
volume 0 {
device minor 1001;
disk "/dev/zvol/faststor/pm-555df3d0_00000";
meta-disk internal;
disk {
rs-discard-granularity 16384; # bytes
}
}
}
connection {
_peer_node_id 2;
path {
_this_host ipv4 192.168.199.4:7001;
_remote_host ipv4 192.168.199.7:7001;
}
net {
allow-two-primaries yes;
cram-hmac-alg "sha1";
shared-secret "Msdh1RTQzBSm/SWOHx+o";
verify-alg "sha256";
_name "vbt-paw1";
}
volume 0 {
disk {
bitmap no;
}
}
}
connection {
_peer_node_id 1;
path {
_this_host ipv4 192.168.199.4:7001;
_remote_host ipv4 192.168.199.5:7001;
}
net {
allow-two-primaries yes;
cram-hmac-alg "sha1";
shared-secret "Msdh1RTQzBSm/SWOHx+o";
verify-alg "sha256";
_name "vbt-prox2";
}
}
}
โฆand on target:
resource "pm-555df3d0" {
options {
auto-promote no;
quorum majority;
on-no-quorum io-error;
}
_this_host {
node-id 1;
volume 0 {
device minor 1001;
disk "/dev/zvol/faststor/pm-555df3d0_00000";
meta-disk internal;
disk {
rs-discard-granularity 16384; # bytes
}
}
}
connection {
_peer_node_id 2;
path {
_this_host ipv4 192.168.199.5:7001;
_remote_host ipv4 192.168.199.7:7001;
}
net {
allow-two-primaries yes;
cram-hmac-alg "sha1";
shared-secret "Msdh1RTQzBSm/SWOHx+o";
verify-alg "sha256";
_name "vbt-paw1";
}
volume 0 {
disk {
bitmap no;
}
}
}
connection {
_peer_node_id 0;
path {
_this_host ipv4 192.168.199.5:7001;
_remote_host ipv4 192.168.199.4:7001;
}
net {
allow-two-primaries yes;
cram-hmac-alg "sha1";
shared-secret "Msdh1RTQzBSm/SWOHx+o";
verify-alg "sha256";
_name "vbt-prox1";
}
}
}
Complete journalctl
output during an attempted migration, source:
Jan 03 14:44:46 vbt-prox1 pvedaemon[4215]: <root@pam> starting task UPID:vbt-prox1:00003E17:000416CC:67785A4E:qmigrate:100:root@pam:
Jan 03 14:44:47 vbt-prox1 pmxcfs[2508]: [status] notice: received log
Jan 03 14:44:47 vbt-prox1 Controller[7082]: 2025-01-03 14:44:47.627 [grizzly-http-server-8] INFO LINSTOR/Controller/52350a SYSTEM - REST/API RestClient(192.168.199.5; 'linstor-proxmox/8.0.4')/ModRscDfn
Jan 03 14:44:47 vbt-prox1 Controller[7082]: 2025-01-03 14:44:47.628 [grizzly-http-server-8] INFO LINSTOR/Controller/52350a SYSTEM - Resource definition modified pm-555df3d0/false
Jan 03 14:44:47 vbt-prox1 Controller[7082]: 2025-01-03 14:44:47.631 [grizzly-http-server-9] INFO LINSTOR/Controller/c5cff2 SYSTEM - REST/API RestClient(192.168.199.5; 'linstor-proxmox/8.0.4')/LstVlm
Jan 03 14:44:47 vbt-prox1 Controller[7082]: 2025-01-03 14:44:47.797 [grizzly-http-server-11] INFO LINSTOR/Controller/19db8e SYSTEM - REST/API RestClient(192.168.199.5; 'linstor-proxmox/8.0.4')/LstVlm
Jan 03 14:44:47 vbt-prox1 pmxcfs[2508]: [status] notice: received log
Jan 03 14:44:48 vbt-prox1 pmxcfs[2508]: [status] notice: received log
Jan 03 14:44:48 vbt-prox1 Controller[7082]: 2025-01-03 14:44:48.859 [grizzly-http-server-13] INFO LINSTOR/Controller/4fb1b1 SYSTEM - REST/API RestClient(192.168.199.5; 'linstor-proxmox/8.0.4')/LstVlm
Jan 03 14:44:48 vbt-prox1 pmxcfs[2508]: [status] notice: received log
Jan 03 14:44:48 vbt-prox1 pvedaemon[15895]: migration problems
Jan 03 14:44:48 vbt-prox1 pvedaemon[4215]: <root@pam> end task UPID:vbt-prox1:00003E17:000416CC:67785A4E:qmigrate:100:root@pam: migration problems
โฆand target:
Jan 03 14:44:46 vbt-prox2 pmxcfs[3156]: [status] notice: received log
Jan 03 14:44:46 vbt-prox2 sshd[13767]: Accepted publickey for root from 192.168.199.4 port 60002 ssh2: RSA SHA256:1gcdoIRJdl4nTg4pc39CCy4vOqf370AYOCnjkVMZs7o
Jan 03 14:44:46 vbt-prox2 sshd[13767]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jan 03 14:44:46 vbt-prox2 systemd-logind[2579]: New session 8 of user root.
Jan 03 14:44:46 vbt-prox2 systemd[1]: Started session-8.scope - Session 8 of User root.
Jan 03 14:44:46 vbt-prox2 sshd[13767]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 03 14:44:46 vbt-prox2 sshd[13767]: Received disconnect from 192.168.199.4 port 60002:11: disconnected by user
Jan 03 14:44:46 vbt-prox2 sshd[13767]: Disconnected from user root 192.168.199.4 port 60002
Jan 03 14:44:46 vbt-prox2 sshd[13767]: pam_unix(sshd:session): session closed for user root
Jan 03 14:44:46 vbt-prox2 systemd[1]: session-8.scope: Deactivated successfully.
Jan 03 14:44:46 vbt-prox2 systemd-logind[2579]: Session 8 logged out. Waiting for processes to exit.
Jan 03 14:44:46 vbt-prox2 systemd-logind[2579]: Removed session 8.
Jan 03 14:44:46 vbt-prox2 sshd[13791]: Accepted publickey for root from 192.168.199.4 port 60010 ssh2: RSA SHA256:1gcdoIRJdl4nTg4pc39CCy4vOqf370AYOCnjkVMZs7o
Jan 03 14:44:46 vbt-prox2 sshd[13791]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jan 03 14:44:46 vbt-prox2 systemd-logind[2579]: New session 9 of user root.
Jan 03 14:44:46 vbt-prox2 systemd[1]: Started session-9.scope - Session 9 of User root.
Jan 03 14:44:46 vbt-prox2 sshd[13791]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 03 14:44:47 vbt-prox2 qm[13801]: <root@pam> starting task UPID:vbt-prox2:000035EA:0002F997:67785A4F:qmstart:100:root@pam:
Jan 03 14:44:47 vbt-prox2 qm[13802]: start VM 100: UPID:vbt-prox2:000035EA:0002F997:67785A4F:qmstart:100:root@pam:
Jan 03 14:44:47 vbt-prox2 systemd[1]: Started 100.scope.
Jan 03 14:44:47 vbt-prox2 systemd[1]: 100.scope: Deactivated successfully.
Jan 03 14:44:47 vbt-prox2 qm[13802]: start failed: QEMU exited with code 1
Jan 03 14:44:47 vbt-prox2 qm[13801]: <root@pam> end task UPID:vbt-prox2:000035EA:0002F997:67785A4F:qmstart:100:root@pam: start failed: QEMU exited with code 1
Jan 03 14:44:47 vbt-prox2 sshd[13791]: Received disconnect from 192.168.199.4 port 60010:11: disconnected by user
Jan 03 14:44:47 vbt-prox2 sshd[13791]: Disconnected from user root 192.168.199.4 port 60010
Jan 03 14:44:47 vbt-prox2 sshd[13791]: pam_unix(sshd:session): session closed for user root
Jan 03 14:44:47 vbt-prox2 systemd-logind[2579]: Session 9 logged out. Waiting for processes to exit.
Jan 03 14:44:47 vbt-prox2 systemd[1]: session-9.scope: Deactivated successfully.
Jan 03 14:44:47 vbt-prox2 systemd-logind[2579]: Removed session 9.
Jan 03 14:44:48 vbt-prox2 sshd[13816]: Accepted publickey for root from 192.168.199.4 port 60022 ssh2: RSA SHA256:1gcdoIRJdl4nTg4pc39CCy4vOqf370AYOCnjkVMZs7o
Jan 03 14:44:48 vbt-prox2 sshd[13816]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jan 03 14:44:48 vbt-prox2 systemd-logind[2579]: New session 10 of user root.
Jan 03 14:44:48 vbt-prox2 systemd[1]: Started session-10.scope - Session 10 of User root.
Jan 03 14:44:48 vbt-prox2 sshd[13816]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 03 14:44:48 vbt-prox2 qm[13822]: <root@pam> starting task UPID:vbt-prox2:000035FF:0002FA17:67785A50:qmstop:100:root@pam:
Jan 03 14:44:48 vbt-prox2 qm[13823]: stop VM 100: UPID:vbt-prox2:000035FF:0002FA17:67785A50:qmstop:100:root@pam:
Jan 03 14:44:48 vbt-prox2 qm[13822]: <root@pam> end task UPID:vbt-prox2:000035FF:0002FA17:67785A50:qmstop:100:root@pam: OK
Jan 03 14:44:48 vbt-prox2 sshd[13816]: Received disconnect from 192.168.199.4 port 60022:11: disconnected by user
Jan 03 14:44:48 vbt-prox2 sshd[13816]: Disconnected from user root 192.168.199.4 port 60022
Jan 03 14:44:48 vbt-prox2 sshd[13816]: pam_unix(sshd:session): session closed for user root
Jan 03 14:44:48 vbt-prox2 systemd[1]: session-10.scope: Deactivated successfully.
Jan 03 14:44:48 vbt-prox2 systemd-logind[2579]: Session 10 logged out. Waiting for processes to exit.
Jan 03 14:44:48 vbt-prox2 systemd-logind[2579]: Removed session 10.
โฆand status output:
2025-01-03 14:44:46 starting migration of VM 100 to node 'vbt-prox2' (192.168.199.5)
2025-01-03 14:44:46 starting VM 100 on remote node 'vbt-prox2'
2025-01-03 14:44:47 [vbt-prox2] kvm: -drive file=/dev/drbd/by-res/pm-555df3d0/0,if=none,id=drive-ide0,format=raw,cache=none,aio=io_uring,detect-zeroes=on: Could not open '/dev/drbd/by-res/pm-555df3d0/0': Wrong medium type
2025-01-03 14:44:47 [vbt-prox2] start failed: QEMU exited with code 1
2025-01-03 14:44:47 ERROR: online migrate failure - remote command failed with exit code 255
2025-01-03 14:44:47 aborting phase 2 - cleanup resources
2025-01-03 14:44:47 migrate_cancel
2025-01-03 14:44:48 ERROR: migration finished with problems (duration 00:00:02)
TASK ERROR: migration problems
The verify works properly with no errors.
Please let me know if youโd like to see anything else from these systems, Iโd really like to get this one fixed. Thanks in advance.