Migrating Proxmox-VM(s) fails with 'Wrong medium type' on the target node (Linstor-DRBD-VMs)

LINBIT SDS Integrations Proxmox VE
1

Dear community,

running here is a three node cluster of Proxmox VE 8.2 with the Linstor-DRBD-Backend enabled (great, I like it!).

Live migration of VMs from one node to another has successfully being tested a few week ago.

Now, when I want to migrate a VM (any of them) to another node, the task produces an error that reads:

[node3] kvm: -drive file=/dev/drbd/by-res/vm-101-disk-1/0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on: Could not open '/dev/drbd/by-res/vm-101-disk-1/0': Wrong medium type

The system log of the source node says:

Oct 14 14:47:11 ttivs1 pvedaemon[276824]: <root@pam> starting task UPID:ttivs1:00048289:00625DEC:670D12CF:qmigrate:101:root@pam:
Oct 14 14:47:13 ttivs1 pmxcfs[1867]: [status] notice: received log
Oct 14 14:47:14 ttivs1 Controller[4651]: 2024-10-14 14:47:14.013 [grizzly-http-server-16] INFO LINSTOR/Controller/58bb4b SYSTEM - REST/API RestClient(10.10.10.3; 'linstor-proxmox/8.0.4')/ModRscDfn
Oct 14 14:47:14 ttivs1 Controller[4651]: 2024-10-14 14:47:14.014 [grizzly-http-server-16] INFO LINSTOR/Controller/58bb4b SYSTEM - Resource definition modified vm-101-disk-1/false
Oct 14 14:47:14 ttivs1 Controller[4651]: 2024-10-14 14:47:14.017 [grizzly-http-server-17] INFO LINSTOR/Controller/08d41b SYSTEM - REST/API RestClient(10.10.10.3; 'linstor-proxmox/8.0.4')/LstVlm
Oct 14 14:47:14 ttivs1 Controller[4651]: 2024-10-14 14:47:14.305 [grizzly-http-server-19] INFO LINSTOR/Controller/988e30 SYSTEM - REST/API RestClient(10.10.10.3; 'linstor-proxmox/8.0.4')/LstVlm
Oct 14 14:47:14 ttivs1 pmxcfs[1867]: [status] notice: received log
Oct 14 14:47:15 ttivs1 pmxcfs[1867]: [status] notice: received log
Oct 14 14:47:15 ttivs1 Controller[4651]: 2024-10-14 14:47:15.743 [grizzly-http-server-21] INFO LINSTOR/Controller/769f8b SYSTEM - REST/API RestClient(10.10.10.3; 'linstor-proxmox/8.0.4')/LstVlm
Oct 14 14:47:15 ttivs1 pmxcfs[1867]: [status] notice: received log
Oct 14 14:47:15 ttivs1 pvedaemon[295561]: migration problems
Oct 14 14:47:15 ttivs1 pvedaemon[276824]: <root@pam> end task UPID:ttivs1:00048289:00625DEC:670D12CF:qmigrate:101:root@pam: migration problems

Updates have been made since the last successful attempt of moving a VM, so that may be a reason for the problem?

Does or did anyone experience the same issue?

What kind of additional information would you need from me?

Any help is appreciated!

Kind regards
Matthias

2

What does cat /proc/drbd; drbdadm status show from the target node?
I’m Wondering if the update installed a new kernel without installing a new DRBD kernel module.

3

Hi Kermat,

thanks for your answer!

That has been checked immediately, and the output is perfectly OK: kernel modules are loaded, and both target nodes are active secondaries.

Best regards
Matthias

4

I think the reason is that DRBD doesn’t change to primary before the VM is being started.

But still it’s not clear why it doesn’t …

DRBD-reactor is configured and runs on all nodes, the linstor-controller only runs on one.

It has worked before …

The resource files of the vm disks need to be identical on all nodes?
They are …

Best regards
Matthias

5

How is your Proxmox storage.cfg configured? Does it use the virtual IP address configured in DRBD Reactor for the HA LINSTOR Controller?

Can you post the output of the following commands:

  • linstor resource list from the LINSTOR controller
  • drbdadm status from the source and destination VM migration nodes
  • drbdsetup show from the source and destination VM migration nodes
  • grep -ie satellite -ie controller -ie drbd -ie pve /var/log/syslog from the source and destination VM migration nodes as well as the LINSTOR Controller after an attempted migration.
6

Hi Kermat,

thanks for your reply!

The storage.cfg:

–
dir: local
path /var/lib/vz
content backup,iso,vztmpl

zfspool: local-zfs
pool rpool/data
content rootdir,images
sparse 1

zfspool: datapool
pool datapool
content rootdir,images
mountpoint /datapool
nodes ttivs3,ttivs2,ttivs1

drbd: drbdstorage
content images, rootdir
controller 10.10.10.1,10.10.1.2,10.10.10.3
resourcegroup pve-rg
preferlocal yes

The file /etc/drbd-reactor.toml ist somewhat unconfigured while there is /etc/drbd-reactor.d/linstor_db.toml which reads:

–
[[promoter]]
id = “linstor_db”
[promoter.resources.linstor_db]
start = [“var-lib-linstor.mount”, “linstor-controller.service”]

On the source node:

linstor resource list :

–
╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ linstor_db ┊ ttivs1 ┊ 7011 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2024-08-05 19:05:41 ┊
┊ linstor_db ┊ ttivs2 ┊ 7011 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-08-05 19:05:48 ┊
┊ linstor_db ┊ ttivs3 ┊ 7011 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-08-05 19:05:46 ┊
┊ vm-100-disk-1 ┊ ttivs1 ┊ 7005 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-03-12 09:10:19 ┊
┊ vm-100-disk-1 ┊ ttivs2 ┊ 7005 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-03-12 09:10:16 ┊
┊ vm-100-disk-1 ┊ ttivs3 ┊ 7005 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-25 14:01:39 ┊
.
.
. and so on for more disks.

linstor_db role:Primary
disk:UpToDate
ttivs2 role:Secondary
peer-disk:UpToDate
ttivs3 role:Secondary
peer-disk:UpToDate

vm-100-disk-1 role:Secondary
disk:UpToDate
ttivs2 role:Secondary
peer-disk:UpToDate
ttivs3 role:Secondary
peer-disk:UpToDate
.
.
. and so on for more disks.

drbdsetup show :

–
resource “linstor_db” {
options {
auto-promote no;
quorum majority;
on-no-quorum io-error;
on-suspended-primary-outdated force-secondary;
}
_this_host {
node-id 0;
volume 0 {
device minor 1011;
disk “/dev/zvol/datapool/linstor_db_00000”;
meta-disk internal;
disk {
rs-discard-granularity 16384; # bytes
}
}
}
connection {
_peer_node_id 1;
path {
_this_host ipv4 10.10.10.1:7011;
_remote_host ipv4 10.10.10.2:7011;
}
net {
cram-hmac-alg “sha1”;
shared-secret “wSnUpqB2wWzQ3FUtycSs”;
rr-conflict retry-connect;
verify-alg “crct10dif”;
_name “ttivs2”;
}
}
connection {
_peer_node_id 2;
path {
_this_host ipv4 10.10.10.1:7011;
_remote_host ipv4 10.10.10.3:7011;
}
net {
cram-hmac-alg “sha1”;
shared-secret “wSnUpqB2wWzQ3FUtycSs”;
rr-conflict retry-connect;
verify-alg “crct10dif”;
_name “ttivs3”;
}
}
}
resource “vm-100-disk-1” {
options {
auto-promote no;
quorum majority;
on-no-quorum io-error;
on-suspended-primary-outdated force-secondary;
}
_this_host {
node-id 0;
volume 0 {
device minor 1005;
disk “/dev/zvol/datapool/vm-100-disk-1_00000”;
meta-disk internal;
disk {
rs-discard-granularity 16384; # bytes
}
}
}
connection {
_peer_node_id 1;
path {
_this_host ipv4 10.10.10.1:7005;
_remote_host ipv4 10.10.10.2:7005;
}
net {
cram-hmac-alg “sha1”;
shared-secret “MljCzHjD1cUBxZvNfm7n”;
rr-conflict retry-connect;
verify-alg “crct10dif”;
_name “ttivs2”;
}
}
connection {
_peer_node_id 2;
path {
_this_host ipv4 10.10.10.1:7005;
_remote_host ipv4 10.10.10.3:7005;
}
net {
cram-hmac-alg “sha1”;
shared-secret “MljCzHjD1cUBxZvNfm7n”;
rr-conflict retry-connect;
verify-alg “crct10dif”;
_name “ttivs3”;
}
}
}
.
.
. and so on for more disks.

On the destination node:

linstor resource list:

–
╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ linstor_db ┊ ttivs1 ┊ 7011 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2024-08-05 19:05:41 ┊
┊ linstor_db ┊ ttivs2 ┊ 7011 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-08-05 19:05:48 ┊
┊ linstor_db ┊ ttivs3 ┊ 7011 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-08-05 19:05:46 ┊
┊ vm-100-disk-1 ┊ ttivs1 ┊ 7005 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-03-12 09:10:19 ┊
┊ vm-100-disk-1 ┊ ttivs2 ┊ 7005 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-03-12 09:10:16 ┊
┊ vm-100-disk-1 ┊ ttivs3 ┊ 7005 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-25 14:01:39 ┊
.
.
. and so on for more disks.

drbdadm status:

–
linstor_db role:Secondary
disk:UpToDate
ttivs1 role:Primary
peer-disk:UpToDate
ttivs2 role:Secondary
peer-disk:UpToDate

vm-100-disk-1 role:Secondary
disk:UpToDate
ttivs1 role:Secondary
peer-disk:UpToDate
ttivs2 role:Secondary
peer-disk:UpToDate
.
.
. and so on for more disks.

drbdsetup show:

–
resource “linstor_db” {
options {
auto-promote no;
quorum majority;
on-no-quorum io-error;
on-suspended-primary-outdated force-secondary;
}
_this_host {
node-id 2;
volume 0 {
device minor 1011;
disk “/dev/zvol/datapool/linstor_db_00000”;
meta-disk internal;
disk {
rs-discard-granularity 16384; # bytes
}
}
}
connection {
_peer_node_id 0;
path {
_this_host ipv4 10.10.10.3:7011;
_remote_host ipv4 10.10.10.1:7011;
}
net {
cram-hmac-alg “sha1”;
shared-secret “wSnUpqB2wWzQ3FUtycSs”;
rr-conflict retry-connect;
verify-alg “crct10dif”;
_name “ttivs1”;
}
}
connection {
_peer_node_id 1;
path {
_this_host ipv4 10.10.10.3:7011;
_remote_host ipv4 10.10.10.2:7011;
}
net {
cram-hmac-alg “sha1”;
shared-secret “wSnUpqB2wWzQ3FUtycSs”;
rr-conflict retry-connect;
verify-alg “crct10dif”;
_name “ttivs2”;
}
}
}
resource “vm-100-disk-1” {
options {
auto-promote no;
quorum majority;
on-no-quorum io-error;
on-suspended-primary-outdated force-secondary;
}
_this_host {
node-id 2;
volume 0 {
device minor 1005;
disk “/dev/zvol/datapool/vm-100-disk-1_00000”;
meta-disk internal;
disk {
rs-discard-granularity 16384; # bytes
}
}
}
connection {
_peer_node_id 0;
path {
_this_host ipv4 10.10.10.3:7005;
_remote_host ipv4 10.10.10.1:7005;
}
net {
cram-hmac-alg “sha1”;
shared-secret “MljCzHjD1cUBxZvNfm7n”;
rr-conflict retry-connect;
verify-alg “crct10dif”;
_name “ttivs1”;
}
}
connection {
_peer_node_id 1;
path {
_this_host ipv4 10.10.10.3:7005;
_remote_host ipv4 10.10.10.2:7005;
}
net {
cram-hmac-alg “sha1”;
shared-secret “MljCzHjD1cUBxZvNfm7n”;
rr-conflict retry-connect;
verify-alg “crct10dif”;
_name “ttivs2”;
}
}
}
.
.
. and so on for more disks.

Unfortunately there is no /var/log/syslog !
Actually I wondered where it is before - do you have an idea?

Best regards
Matthias

7

Sorry - I can’t tell the reason for the changes of font sizes …

8

Hi Kermar,

journalctl is being used instead of syslog. The output of

journalctl | grep -ie satellite -ie controller -ie drbd -ie pve

after an attempted migration on the source node is:

–

Oct 27 09:27:51 ttivs1 pveproxy[3903032]: Clearing outdated entries from certificate cache
Oct 27 09:27:55 ttivs1 pvedaemon[1100032]: root@pam starting task UPID:ttivs1:000FA757:06FC7F4A:671DF98B:qmigrate:104:root@pam:
Oct 27 09:27:57 ttivs1 Controller[3780222]: 2024-10-27 09:27:57.342 [grizzly-http server-10] INFO LINSTOR/Controller/85c397 SYSTEM - REST/API RestClient(10.10.10.3; ‘linstor-proxmox/8.0.4’)/ModRscDfn
Oct 27 09:27:57 ttivs1 Controller[3780222]: 2024-10-27 09:27:57.342 [grizzly-http server-10] INFO LINSTOR/Controller/85c397 SYSTEM - Resource definition modified vm-104-disk-1/false
Oct 27 09:27:57 ttivs1 Controller[3780222]: 2024-10-27 09:27:57.345 [grizzly-http server-11] INFO LINSTOR/Controller/22dfab SYSTEM - REST/API RestClient(10.10.10.3; ‘linstor-proxmox/8.0.4’)/LstVlm
Oct 27 09:27:57 ttivs1 Controller[3780222]: 2024-10-27 09:27:57.632 [grizzly-http server-13] INFO LINSTOR/Controller/2aa837 SYSTEM - REST/API RestClient(10.10.10.3; ‘linstor-proxmox/8.0.4’)/LstVlm
Oct 27 09:27:58 ttivs1 Controller[3780222]: 2024-10-27 09:27:58.898 [grizzly-http server-15] INFO LINSTOR/Controller/932f9e SYSTEM - REST/API RestClient(10.10.10.3; ‘linstor-proxmox/8.0.4’)/LstVlm
Oct 27 09:27:58 ttivs1 pvedaemon[1025879]: migration problems
Oct 27 09:27:58 ttivs1 pvedaemon[1100032]: root@pam end task UPID:ttivs1:000FA757:06FC7F4A:671DF98B:qmigrate:104:root@pam: migration problems
Oct 27 09:28:06 ttivs1 Controller[3780222]: 2024-10-27 09:28:06.538 [grizzly-http server-17] INFO LINSTOR/Controller/ce2f27 SYSTEM - REST/API RestClient(10.10.10.3; ‘linstor-proxmox/8.0.4’)/QryAllSizeInfo
–

There’s no output on the destination node.

Best regards
Matthias

ps Point is, somehow: … it worked before …

9

Hi there,

maybe(!) I made one mistake in setting up the highly available controller.

The documentation reads:

–

The last but nevertheless important step is to configure the LINSTOR satellite services to not delete (and then regenerate) the resource file for the LINSTOR controller DB at its startup. Do not edit the service files directly, but use systemctl edit. Edit the service file on all nodes that could become a LINSTOR controller and that are also LINSTOR satellites.

systemctl edit linstor-satellite

[Service]
Environment=LS_KEEP_RES=linstor_db

Now it’s configured that way, but maybe some resource files came out of sync?
How can I check this?

Migration fails with:

2024-11-06 16:05:02 starting migration of VM 104 to node ‘ttivs3’ (10.10.10.3)
2024-11-06 16:05:02 starting VM 104 on remote node ‘ttivs3’
2024-11-06 16:05:03 [ttivs3] kvm: -drive file=/dev/drbd/by-res/vm-104-disk-1/0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on: Could not open ‘/dev/drbd/by-res/vm-104-disk-1/0’: Wrong medium type
2024-11-06 16:05:04 [ttivs3] start failed: QEMU exited with code 1
2024-11-06 16:05:04 ERROR: online migrate failure - remote command failed with exit code 255
2024-11-06 16:05:04 aborting phase 2 - cleanup resources
2024-11-06 16:05:04 migrate_cancel
2024-11-06 16:05:05 ERROR: migration finished with problems (duration 00:00:05)
TASK ERROR: migration problems

I think the underlying DRBD device doesn’t do the primary/secondary switch, because the ‘wrong medium type’ error seems to happen when the device of the secondary is being accessed.

Can I test it manually?

Best regards
Matthias

10

You can see the LINSTOR generated configurations in the following directory:

ls -hasl /var/lib/linstor.d/*.res

LINSTOR should show you whether or not the vm-104-disk-1 resource exists on ttivs3 node in the resource list:

linstor resource list --resources vm-104-disk-1

You should be able to, yes. Would be a simple, drbdadm primary vm-104-disk-1, followed by a drbdadm secondary vm-104-disk-1.

Maybe the allow-two-primaries setting somehow got reset (to false)? What does the configuration of that resource look like?

cat /var/lib/linstor.d/vm-104-disk-1.res
11

Hi Kermat,

thanks very much for your reply.

Apart from the host names and node ids the .res files are identical.

The resource looks perfectly OK to me:

# linstor resource list --resources vm-104-disk-1
╭─────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName  ┊ Node   ┊ Port ┊ Usage  ┊ Conns ┊    State ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════╡
┊ vm-104-disk-1 ┊ ttivs1 ┊ 7003 ┊ InUse  ┊ Ok    ┊ UpToDate ┊ 2024-03-12 18:23:17 ┊
┊ vm-104-disk-1 ┊ ttivs2 ┊ 7003 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2024-03-12 18:23:20 ┊
┊ vm-104-disk-1 ┊ ttivs3 ┊ 7003 ┊ Unused ┊ Ok    ┊ UpToDate ┊ 2024-07-25 14:57:39 ┊

drbdadm primary vm-104-disk-1, followed by a drbdadm secondary vm-104-disk-1

Doing that on the secondary ttivs3 also looks OK to me.
It goes primary (additionally to ttivs1 and then back to secondary.

‘allow-two-primaries’ is set to ‘yes’ in the resource file.

Best regards
Matthias

12

Interesting. I’m wondering if the resources have somehow silently become out of sync with one another.

Please try running an online verification from the Primary node (ttivs1) to verify that everything is actually synchronized.

drbdadm verify vm-104-disk-1:ttivs3

After running the command, you can watch the progress of the verification using watch drbdadm status vm-104-disk-1. You’ll see the replication status change to replication:VerifyS peer-disk:UpToDate done:4.75 on ttivs1.

Once complete you can check the logs to see if any inconsistent blocks were found:

grep "Out of sync" /var/log/syslog

If DRBD did detect differences, you will see one or more entries similar to this in the logs:

drbd dr-res-0/0 drbd1000 linbit-dr: Out of sync: start=1232, size=24 (sectors)

To get the “remote” node (ttivs3) back in sync with the “local” node (ttivs1) after out of sync blocks were discovered, you would run the following command from the “local” node:

drbdadm invalidate-remote vm-104-disk-1:ttivs3 --reset-bitmap=no
13

Hi Kermat,

thanks very much!

I did as you told and that can be found in the log:

–
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: repl( Off → WFBitMapS ) [connected]
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 28(1), total 28; compression: 100.0%
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 28(1), total 28; compression: 100.0%
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: helper command: /sbin/drbdadm before-resync-source
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: helper command: /sbin/drbdadm before-resync-source exit code 0
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: pdsk( Outdated → Inconsistent ) repl( WFBitMapS → SyncSource ) [receive-bitmap]
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: Began resync as SyncSource (will sync 12 KB [3 bits set]).
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: updated UUIDs 0343549A6773A387:0000000000000000:2BDF1781118BB5F2:50225B991A0D347C
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: Resync done (total 1 sec; paused 0 sec; 12 K/sec)
Nov 15 10:32:48 ttivs1 kernel: drbd vm-104-disk-1/0 drbd1003 ttivs3: pdsk( Inconsistent → UpToDate ) repl( SyncSource → Established ) [resync-finished]

So no ‘Out of sync’ to be found but there’s someting happening and UUIDs are updated.

Thing is: I can execute the command again and that will happen similarly again.
I can execute it many times and it always run for a different amount of time.
When I run it to verify the other secondary (ttivs2) it behaves similarly.

And unfortunately: It doesn’t change the migration behavior which still is failing.

Do you habe further ideas?
Is it OK that the verify command ‘does something’ any time it’s being run?

Should I ‘invalidate’?

Best regards
Matthias

14

Matthias, please use

Code blocks

for logs, content of configuration files, and CLI output. It’s much easier on the eyes. :slight_smile:

1 Like
15

I managed to run into this myself, and I’m not sure what caused it - similarly, this used to work fine. Basically, everything looks good, no errors anywhere, but proxmox will not promote a volume to primary for itself. This affects both migrations (migrations, or rather the start step of them, will always fail unless you drbdadm primary the volume on the target node first, ideally followed by drbdadm secondary on the source node), and creation of new disks (the whole thing gets created properly, but is left on all secondaries, so the attach to the VM fails with the same ‘Wrong medium type’ failure until you set primary). This affects every volume on every resource.

I completed a reboot of each node (one-by-one for no loss of service), no help.

Additional outputs, for clarity cut to a single volume, but all are similar:

linstor resource list:

┊ pm-555df3d0  ┊ vbt-paw1  ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2024-12-18 13:25:36 ┊
┊ pm-555df3d0  ┊ vbt-prox1 ┊ DRBD,STORAGE ┊ InUse  ┊ Ok    ┊   UpToDate ┊ 2024-12-17 12:11:35 ┊
┊ pm-555df3d0  ┊ vbt-prox2 ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2024-12-17 12:11:35 ┊

drbdadm status on the source, target says exactly what you’d expect:

pm-555df3d0 role:Primary
  disk:UpToDate open:yes
  vbt-paw1 role:Secondary
    peer-disk:Diskless
  vbt-prox2 role:Secondary
    peer-disk:UpToDate

drbdsetup show on source:

resource "pm-555df3d0" {
    options {
        auto-promote            no;
        quorum                  majority;
        on-no-quorum            io-error;
    }
    _this_host {
        node-id                 0;
        volume 0 {
            device                      minor 1001;
            disk                        "/dev/zvol/faststor/pm-555df3d0_00000";
            meta-disk                   internal;
            disk {
                rs-discard-granularity  16384; # bytes
            }
        }
    }
    connection {
        _peer_node_id 2;
        path {
            _this_host ipv4 192.168.199.4:7001;
            _remote_host ipv4 192.168.199.7:7001;
        }
        net {
            allow-two-primaries yes;
            cram-hmac-alg       "sha1";
            shared-secret       "Msdh1RTQzBSm/SWOHx+o";
            verify-alg          "sha256";
            _name               "vbt-paw1";
        }
        volume 0 {
            disk {
                bitmap                  no;
            }
        }
    }
    connection {
        _peer_node_id 1;
        path {
            _this_host ipv4 192.168.199.4:7001;
            _remote_host ipv4 192.168.199.5:7001;
        }
        net {
            allow-two-primaries yes;
            cram-hmac-alg       "sha1";
            shared-secret       "Msdh1RTQzBSm/SWOHx+o";
            verify-alg          "sha256";
            _name               "vbt-prox2";
        }
    }
}

…and on target:

resource "pm-555df3d0" {
    options {
        auto-promote            no;
        quorum                  majority;
        on-no-quorum            io-error;
    }
    _this_host {
        node-id                 1;
        volume 0 {
            device                      minor 1001;
            disk                        "/dev/zvol/faststor/pm-555df3d0_00000";
            meta-disk                   internal;
            disk {
                rs-discard-granularity  16384; # bytes
            }
        }
    }
    connection {
        _peer_node_id 2;
        path {
            _this_host ipv4 192.168.199.5:7001;
            _remote_host ipv4 192.168.199.7:7001;
        }
        net {
            allow-two-primaries yes;
            cram-hmac-alg       "sha1";
            shared-secret       "Msdh1RTQzBSm/SWOHx+o";
            verify-alg          "sha256";
            _name               "vbt-paw1";
        }
        volume 0 {
            disk {
                bitmap                  no;
            }
        }
    }
    connection {
        _peer_node_id 0;
        path {
            _this_host ipv4 192.168.199.5:7001;
            _remote_host ipv4 192.168.199.4:7001;
        }
        net {
            allow-two-primaries yes;
            cram-hmac-alg       "sha1";
            shared-secret       "Msdh1RTQzBSm/SWOHx+o";
            verify-alg          "sha256";
            _name               "vbt-prox1";
        }
    }
}

Complete journalctl output during an attempted migration, source:

Jan 03 14:44:46 vbt-prox1 pvedaemon[4215]: <root@pam> starting task UPID:vbt-prox1:00003E17:000416CC:67785A4E:qmigrate:100:root@pam:
Jan 03 14:44:47 vbt-prox1 pmxcfs[2508]: [status] notice: received log
Jan 03 14:44:47 vbt-prox1 Controller[7082]: 2025-01-03 14:44:47.627 [grizzly-http-server-8] INFO  LINSTOR/Controller/52350a SYSTEM - REST/API RestClient(192.168.199.5; 'linstor-proxmox/8.0.4')/ModRscDfn
Jan 03 14:44:47 vbt-prox1 Controller[7082]: 2025-01-03 14:44:47.628 [grizzly-http-server-8] INFO  LINSTOR/Controller/52350a SYSTEM - Resource definition modified pm-555df3d0/false
Jan 03 14:44:47 vbt-prox1 Controller[7082]: 2025-01-03 14:44:47.631 [grizzly-http-server-9] INFO  LINSTOR/Controller/c5cff2 SYSTEM - REST/API RestClient(192.168.199.5; 'linstor-proxmox/8.0.4')/LstVlm
Jan 03 14:44:47 vbt-prox1 Controller[7082]: 2025-01-03 14:44:47.797 [grizzly-http-server-11] INFO  LINSTOR/Controller/19db8e SYSTEM - REST/API RestClient(192.168.199.5; 'linstor-proxmox/8.0.4')/LstVlm
Jan 03 14:44:47 vbt-prox1 pmxcfs[2508]: [status] notice: received log
Jan 03 14:44:48 vbt-prox1 pmxcfs[2508]: [status] notice: received log
Jan 03 14:44:48 vbt-prox1 Controller[7082]: 2025-01-03 14:44:48.859 [grizzly-http-server-13] INFO  LINSTOR/Controller/4fb1b1 SYSTEM - REST/API RestClient(192.168.199.5; 'linstor-proxmox/8.0.4')/LstVlm
Jan 03 14:44:48 vbt-prox1 pmxcfs[2508]: [status] notice: received log
Jan 03 14:44:48 vbt-prox1 pvedaemon[15895]: migration problems
Jan 03 14:44:48 vbt-prox1 pvedaemon[4215]: <root@pam> end task UPID:vbt-prox1:00003E17:000416CC:67785A4E:qmigrate:100:root@pam: migration problems

…and target:

Jan 03 14:44:46 vbt-prox2 pmxcfs[3156]: [status] notice: received log
Jan 03 14:44:46 vbt-prox2 sshd[13767]: Accepted publickey for root from 192.168.199.4 port 60002 ssh2: RSA SHA256:1gcdoIRJdl4nTg4pc39CCy4vOqf370AYOCnjkVMZs7o
Jan 03 14:44:46 vbt-prox2 sshd[13767]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jan 03 14:44:46 vbt-prox2 systemd-logind[2579]: New session 8 of user root.
Jan 03 14:44:46 vbt-prox2 systemd[1]: Started session-8.scope - Session 8 of User root.
Jan 03 14:44:46 vbt-prox2 sshd[13767]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 03 14:44:46 vbt-prox2 sshd[13767]: Received disconnect from 192.168.199.4 port 60002:11: disconnected by user
Jan 03 14:44:46 vbt-prox2 sshd[13767]: Disconnected from user root 192.168.199.4 port 60002
Jan 03 14:44:46 vbt-prox2 sshd[13767]: pam_unix(sshd:session): session closed for user root
Jan 03 14:44:46 vbt-prox2 systemd[1]: session-8.scope: Deactivated successfully.
Jan 03 14:44:46 vbt-prox2 systemd-logind[2579]: Session 8 logged out. Waiting for processes to exit.
Jan 03 14:44:46 vbt-prox2 systemd-logind[2579]: Removed session 8.
Jan 03 14:44:46 vbt-prox2 sshd[13791]: Accepted publickey for root from 192.168.199.4 port 60010 ssh2: RSA SHA256:1gcdoIRJdl4nTg4pc39CCy4vOqf370AYOCnjkVMZs7o
Jan 03 14:44:46 vbt-prox2 sshd[13791]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jan 03 14:44:46 vbt-prox2 systemd-logind[2579]: New session 9 of user root.
Jan 03 14:44:46 vbt-prox2 systemd[1]: Started session-9.scope - Session 9 of User root.
Jan 03 14:44:46 vbt-prox2 sshd[13791]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 03 14:44:47 vbt-prox2 qm[13801]: <root@pam> starting task UPID:vbt-prox2:000035EA:0002F997:67785A4F:qmstart:100:root@pam:
Jan 03 14:44:47 vbt-prox2 qm[13802]: start VM 100: UPID:vbt-prox2:000035EA:0002F997:67785A4F:qmstart:100:root@pam:
Jan 03 14:44:47 vbt-prox2 systemd[1]: Started 100.scope.
Jan 03 14:44:47 vbt-prox2 systemd[1]: 100.scope: Deactivated successfully.
Jan 03 14:44:47 vbt-prox2 qm[13802]: start failed: QEMU exited with code 1
Jan 03 14:44:47 vbt-prox2 qm[13801]: <root@pam> end task UPID:vbt-prox2:000035EA:0002F997:67785A4F:qmstart:100:root@pam: start failed: QEMU exited with code 1
Jan 03 14:44:47 vbt-prox2 sshd[13791]: Received disconnect from 192.168.199.4 port 60010:11: disconnected by user
Jan 03 14:44:47 vbt-prox2 sshd[13791]: Disconnected from user root 192.168.199.4 port 60010
Jan 03 14:44:47 vbt-prox2 sshd[13791]: pam_unix(sshd:session): session closed for user root
Jan 03 14:44:47 vbt-prox2 systemd-logind[2579]: Session 9 logged out. Waiting for processes to exit.
Jan 03 14:44:47 vbt-prox2 systemd[1]: session-9.scope: Deactivated successfully.
Jan 03 14:44:47 vbt-prox2 systemd-logind[2579]: Removed session 9.
Jan 03 14:44:48 vbt-prox2 sshd[13816]: Accepted publickey for root from 192.168.199.4 port 60022 ssh2: RSA SHA256:1gcdoIRJdl4nTg4pc39CCy4vOqf370AYOCnjkVMZs7o
Jan 03 14:44:48 vbt-prox2 sshd[13816]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Jan 03 14:44:48 vbt-prox2 systemd-logind[2579]: New session 10 of user root.
Jan 03 14:44:48 vbt-prox2 systemd[1]: Started session-10.scope - Session 10 of User root.
Jan 03 14:44:48 vbt-prox2 sshd[13816]: pam_env(sshd:session): deprecated reading of user environment enabled
Jan 03 14:44:48 vbt-prox2 qm[13822]: <root@pam> starting task UPID:vbt-prox2:000035FF:0002FA17:67785A50:qmstop:100:root@pam:
Jan 03 14:44:48 vbt-prox2 qm[13823]: stop VM 100: UPID:vbt-prox2:000035FF:0002FA17:67785A50:qmstop:100:root@pam:
Jan 03 14:44:48 vbt-prox2 qm[13822]: <root@pam> end task UPID:vbt-prox2:000035FF:0002FA17:67785A50:qmstop:100:root@pam: OK
Jan 03 14:44:48 vbt-prox2 sshd[13816]: Received disconnect from 192.168.199.4 port 60022:11: disconnected by user
Jan 03 14:44:48 vbt-prox2 sshd[13816]: Disconnected from user root 192.168.199.4 port 60022
Jan 03 14:44:48 vbt-prox2 sshd[13816]: pam_unix(sshd:session): session closed for user root
Jan 03 14:44:48 vbt-prox2 systemd[1]: session-10.scope: Deactivated successfully.
Jan 03 14:44:48 vbt-prox2 systemd-logind[2579]: Session 10 logged out. Waiting for processes to exit.
Jan 03 14:44:48 vbt-prox2 systemd-logind[2579]: Removed session 10.

…and status output:

2025-01-03 14:44:46 starting migration of VM 100 to node 'vbt-prox2' (192.168.199.5)
2025-01-03 14:44:46 starting VM 100 on remote node 'vbt-prox2'
2025-01-03 14:44:47 [vbt-prox2] kvm: -drive file=/dev/drbd/by-res/pm-555df3d0/0,if=none,id=drive-ide0,format=raw,cache=none,aio=io_uring,detect-zeroes=on: Could not open '/dev/drbd/by-res/pm-555df3d0/0': Wrong medium type
2025-01-03 14:44:47 [vbt-prox2] start failed: QEMU exited with code 1
2025-01-03 14:44:47 ERROR: online migrate failure - remote command failed with exit code 255
2025-01-03 14:44:47 aborting phase 2 - cleanup resources
2025-01-03 14:44:47 migrate_cancel
2025-01-03 14:44:48 ERROR: migration finished with problems (duration 00:00:02)
TASK ERROR: migration problems

The verify works properly with no errors.

Please let me know if you’d like to see anything else from these systems, I’d really like to get this one fixed. Thanks in advance.

16

Ah, a classic case of user error - need to get just a bit more sleep before setting these things up for the first time!

I did this to myself while setting up the HA Linstor Controller - apparently I read this:

It is crucial that your cluster qualifies for auto-quorum and uses the io-error policy (see Section Auto-Quorum Policies), and that auto-promote is disabled.

…in the user guide, and totally failed to realize that auto-promote is only supposed to be “no” on the linstor-db-grp. Instead, I interpreted that auto-promote needed to be off for the whole cluster, so after the documented commands, I ended up giving linstor controller drbd-options --auto-promote no.

Anyway, it did take a bit to verify that auto-promote WAS supposed to be on for this type of Proxmox setup, but after giving linstor controller drbd-options --auto-promote yes (and resetting the option on linstor_db), all works properly (and in retrospect, of course it works this way - in fact it’s extremely telling that the only command the plugin runs to switch primaries is blockdev --setrw <dev> - oh and I suppose I should have taken the time to understand section 2.5 of the DRBD User Guide beforehand too!)…

One last issue: since I had been manually promoting these, the first migration after fixing that option left me in dual-primary on everything. I simply had to give drbdadm secondary on all the dual-primaries from the node that’s no longer running them - now auto-promote works exactly correctly again in both directions.

This appears to be OP’s problem as well, since auto_promote is off for his vm disks.

4 Likes
17

mkernvbt,

thanks very much - this solved the issue!

But I’ve just been running into another one that somehow seems to be related to the new naming scheme of the disk resources.

I’ll open a new thread about it and try to describe it properly …

Best regards
Matthias