Can't make proxmox-linbit plugin to use multiple controller

Hello,

I have a 3 node cluster of proxmox and followed the following guide to establish HA controller.

1: How to Setup LINSTOR on Proxmox VE - LINBIT
2: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-linstor_ha
3: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-proxmox-ls-HA

Everything works, when one node goes down. Another node controller is being auto started by reactor daemon.

But only issue is that in the gui I donโ€™t see the available storage capacity. But it appears again when the node 1 comes back. I can migrate vm/LXC without any issue from node 2 to node 3 in the meantime.

This is what happens when the controller node goes down. (pvetest1 was controller node first)

root@pvetest2:~# linstor node list
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”Š Node     โ”Š NodeType  โ”Š Addresses                    โ”Š State                                        โ”Š
โ•žโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ก
โ”Š pvetest1 โ”Š SATELLITE โ”Š 192.168.102.201:3366 (PLAIN) โ”Š OFFLINE (Auto-eviction: 2024-09-21 23:44:04) โ”Š
โ”Š pvetest2 โ”Š SATELLITE โ”Š 192.168.102.202:3366 (PLAIN) โ”Š Online                                       โ”Š
โ”Š pvetest3 โ”Š SATELLITE โ”Š 192.168.102.203:3366 (PLAIN) โ”Š Online                                       โ”Š
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
To cancel automatic eviction please consider the corresponding DrbdOptions/AutoEvict* properties on controller and / or node level
See 'linstor controller set-property --help' or 'linstor node set-property --help' for more details

root@pvetest2:~# drbd-reactorctl status 
/etc/drbd-reactor.d/linstor_db.toml:
Promoter: Currently active on this node
โ— drbd-services@linstor_db.target
โ— โ”œโ”€ drbd-promote@linstor_db.service
โ— โ”œโ”€ var-lib-linstor.mount 
โ— โ””โ”€ linstor-controller.service 
root@pvetest2:~# drbdadm status
linstor_db role:Primary
  disk:UpToDate
  pvetest1 connection:Connecting
  pvetest3 role:Secondary
    peer-disk:UpToDate

pm-c40c9e19 role:Secondary
  disk:UpToDate
  pvetest1 connection:Connecting
  pvetest3 role:Secondary
    peer-disk:UpToDate

pm-db8d36d7 role:Secondary
  disk:UpToDate
  pvetest1 connection:Connecting
  pvetest3 role:Secondary
    peer-disk:UpToDate

pm-e29692bb role:Primary
  disk:UpToDate
  pvetest1 connection:Connecting
  pvetest3 role:Secondary
    peer-disk:UpToDate
root@pvetest2:~# cat /etc/pve/storage.cfg 
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images


drbd: linstor_storage
        content images, rootdir
        controller 192.168.102.202,192.168.102.201,192.168.102.203
        resourcegroup pve-rg
root@pvetest2:~# linstor storage-pool list
โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”Š StoragePool          โ”Š Node     โ”Š Driver   โ”Š PoolName            โ”Š FreeCapacity โ”Š TotalCapacity โ”Š CanSnapshots โ”Š State   โ”Š SharedName                    โ”Š
โ•žโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ก
โ”Š DfltDisklessStorPool โ”Š pvetest1 โ”Š DISKLESS โ”Š                     โ”Š              โ”Š               โ”Š False        โ”Š Warning โ”Š pvetest1;DfltDisklessStorPool โ”Š
โ”Š DfltDisklessStorPool โ”Š pvetest2 โ”Š DISKLESS โ”Š                     โ”Š              โ”Š               โ”Š False        โ”Š Ok      โ”Š pvetest2;DfltDisklessStorPool โ”Š
โ”Š DfltDisklessStorPool โ”Š pvetest3 โ”Š DISKLESS โ”Š                     โ”Š              โ”Š               โ”Š False        โ”Š Ok      โ”Š pvetest3;DfltDisklessStorPool โ”Š
โ”Š pve-storage          โ”Š pvetest1 โ”Š LVM_THIN โ”Š linstor_vg/thinpool โ”Š              โ”Š               โ”Š True         โ”Š Warning โ”Š pvetest1;pve-storage          โ”Š
โ”Š pve-storage          โ”Š pvetest2 โ”Š LVM_THIN โ”Š linstor_vg/thinpool โ”Š    25.31 GiB โ”Š     25.54 GiB โ”Š True         โ”Š Ok      โ”Š pvetest2;pve-storage          โ”Š
โ”Š pve-storage          โ”Š pvetest3 โ”Š LVM_THIN โ”Š linstor_vg/thinpool โ”Š    25.31 GiB โ”Š     25.54 GiB โ”Š True         โ”Š Ok      โ”Š pvetest3;pve-storage          โ”Š
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ
WARNING:
Description:
    No active connection to satellite 'pvetest1'
Details:
    The controller is trying to (re-) establish a connection to the satellite. The controller stored the changes and as soon the satellite is connected, it will receive this update.
root@pvetest2:~# 
root@pvetest2:~# pvesm list linstor_storage 
Use of uninitialized value $owner in concatenation (.) or string at /usr/share/perl5/LINBIT/PluginHelper.pm line 65.
Use of uninitialized value $owner in concatenation (.) or string at /usr/share/perl5/LINBIT/PluginHelper.pm line 65.
Use of uninitialized value $owner in concatenation (.) or string at /usr/share/perl5/LINBIT/PluginHelper.pm line 65.
Use of uninitialized value $owner in concatenation (.) or string at /usr/share/perl5/LINBIT/PluginHelper.pm line 65.
Volid                           Format  Type            Size VMID
linstor_storage:pm-e29692bb_100 raw     rootdir   8592252928 100

While the original/first controller is offline, if you run something like the following on each node:

systemctl restart pve-cluster pvedaemon pvestatd pveproxy pve-ha-lrm

Does the available storage return in the Proxmox web interface?


One potential workaround would be to use a virtual IP address for the LINSTOR controller endpoint in the DRBD Reactor configuration:

Instead of:

[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = ["var-lib-linstor.mount", "linstor-controller.service"]

Use something similar to:

[[promoter]]
id = "linstor_db"
[promoter.resources.linstor_db]
start = [
"""
ocf:heartbeat:IPaddr2 db_virtip ip=192.168.102.200 \
cidr_netmask=24 iflabel=virtualip""",
"var-lib-linstor.mount",
"linstor-controller.service"
]

And finally, update the /etc/pve/storage.cfg file to use the new virtual IP address accordingly:

drbd: linstor_storage
        content images, rootdir
        controller 192.168.102.200
        resourcegroup pve-rg

I ran those command before posting and it did not help.

In fact I believe the issue is placement count. My place-count is 3. If i make it 2, them when the controller node or any other single node fails (in that case active nodes become 2) everything works perfectly.

But what Iโ€™m not sure about if this is the desired behavior.

Now the question is, if your node count become less than your placement count than the Proxmox api will not work until all 3 nodes are online?

Is that how it meant to be ?

No, the placement count of 2 or 3 should be irrelevant, but for a highly available LINSTOR controller, I would recommend using a placement count of 3 for 3 full replicas. Itโ€™s just extra insurance in case you canโ€™t recover 2/3 nodes, the database is small, and there arenโ€™t really any downsides to using 3 replicas provided you have at least 3 nodes with backing storage in your LINSTOR cluster.

If youโ€™re only losing the visual graphs in the GUI during a node failure, we can take a look to see if thereโ€™s a weird interaction between our Proxmox plugin and the Proxmox GUI. If youโ€™re able to perform actions that require a running LINSTOR controller (live migration, creating a new virtual disk backed by LINSTOR, etc) and everything is working correctly except the Proxmox GUI, your configuration should be correct.

However, I personally think using a virtual IP address is a more elegant approach (see my first reply). It is a supported configuration as mentioned in the LINSTOR users guide:

An alternative is to configure a virtual IP (VIP) address for the LINSTOR controller by using an OCF resource agent, ocf:heartbeat:IPaddr2, added to the DRBD Reactor promoter pluginโ€™s services start list. If you do this, you could then specify the VIP address for the controller parameter value.

In your cluster the Proxmox GUI seems to be getting confused when it needs to point to a new LINSTOR Controller. Using a virtual IP address should eliminate this behavior as there will only be a single endpoint for the LINSTOR controller regardless of which node is actively running the LINSTOR controller. Again, weโ€™ll take a look at testing this interaction. Hope that helps.

My LINSTOR controller db count is 3 and I have allocated 10G for that. I only put placement count 2 for the vm_data as putting count 3 causes gui and cli to report storage as 0%

Today I applied it. now storage.cfg is using virtual ip.

However, when 1 node fails both the gui and cli shows the available storage 0

root@pvetest2:~# pvesm status
Name                   Type     Status           Total            Used       Available        %
linstor_storage        drbd     active               0               0               0    0.00%
local                   dir     active         6866092         4367388         2128632   63.61%
local-lvm           lvmthin     active         4972544               0         4972544    0.00%

in the meantime, like mentioned before migration works.

my new config:

3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether bc:24:11:3f:15:85 brd ff:ff:ff:ff:ff:ff
    inet 192.168.102.201/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet 192.168.102.199/24 scope global secondary vmbr0:virtualip
       valid_lft forever preferred_lft forever
    inet6 fe80::be24:11ff:fe3f:1585/64 scope link 
       valid_lft forever preferred_lft forever
root@pvetest1:~# cat /etc/pve/storage.cfg 
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images


drbd: linstor_storage
        content images, rootdir
        controller 192.168.102.199
        resourcegroup pve-rg
root@pvetest2:~# apt list --installed linstor*
Listing... Done
linstor-client/unknown,now 1.23.0-1 all [installed,upgradable to: 1.23.2-1]
linstor-common/unknown,now 1.29.0-1 all [installed,upgradable to: 1.29.1-1]
linstor-controller/unknown,now 1.29.0-1 all [installed,upgradable to: 1.29.1-1]
linstor-proxmox/unknown,now 8.0.4-1 all [installed]
linstor-satellite/unknown,now 1.29.0-1 all [installed,upgradable to: 1.29.1-1]

I forgot to mention, even though the migration works, I canโ€™t create any new resource. It gives me the following error:

NOTICE
  Trying to create diskful resource (pm-6f58c7a9) on (pvetest2).
TASK ERROR: unable to create CT 101 - API Return-Code: 500. Message: Could not autoplace resource pm-6f58c7a9, because: [{"ret_code":-4611686018407201820,"message":"Not enough available nodes","details":"Not enough nodes fulfilling the following auto-place criteria:\n * has a deployed storage pool named [pve-storage]\n * the storage pools have to have at least '8388608' free space\n * the current access context has enough privileges to use the node and the storage pool\n * the node is online\n\nAuto-place configuration details:\n   Replica count: 3\n   Additional replica count: 2\n   Storage pool name:\n      pve-storage\n   Do not place with resource:\n      pm-6f58c7a9\n   Diskless on remaining: false\n\nAuto-placing resource: pm-6f58c7a9","obj_refs":{"RscDfn":"pm-6f58c7a9"},"created_at":"2024-09-26T10:47:51.714575548+02:00"}]  at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 433. 	PVE::Storage::Custom::LINSTORPlugin::alloc_image("PVE::Storage::Custom::LINSTORPlugin", "linstor_storage", HASH(0x58ea01c00a88), 101, "raw", undef, 8388608) called at /usr/share/perl5/PVE/Storage.pm line 1006 	eval {...} called at /usr/share/perl5/PVE/Storage.pm line 1006 	PVE::Storage::__ANON__() called at /usr/share/perl5/PVE/Cluster.pm line 648 	eval {...} called at /usr/share/perl5/PVE/Cluster.pm line 614 	PVE::Cluster::__ANON__("storage-linstor_storage", undef, CODE(0x58ea01a5b490)) called at /usr/share/perl5/PVE/Cluster.pm line 693 	PVE::Cluster::cfs_lock_storage("linstor_storage", undef, CODE(0x58ea01a5b490)) called at /usr/share/perl5/PVE/Storage/Plugin.pm line 623 	PVE::Storage::Plugin::cluster_lock_storage("PVE::Storage::Custom::LINSTORPlugin", "linstor_storage", 1, undef, CODE(0x58ea01a5b490)) called at /usr/share/perl5/PVE/Storage.pm line 1011 	PVE::Storage::vdisk_alloc(HASH(0x58ea01aac038), "linstor_storage", 101, "raw", undef, 8388608) called at /usr/share/perl5/PVE/LXC.pm line 2117 	eval {...} called at /usr/share/perl5/PVE/LXC.pm line 2103 	PVE::LXC::alloc_disk(HASH(0x58ea01aac038), 101, "linstor_storage", 8388608, 100000, 100000) called at /usr/share/perl5/PVE/LXC.pm line 2159 	PVE::LXC::__ANON__("rootfs", HASH(0x58ea01a5b1f0)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 475 	PVE::AbstractConfig::foreach_volume_full("PVE::LXC::Config", HASH(0x58ea01adf9d0), undef, CODE(0x58ea01b7f7a8)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 484 	PVE::AbstractConfig::foreach_volume("PVE::LXC::Config", HASH(0x58ea01adf9d0), CODE(0x58ea01b7f7a8)) called at /usr/share/perl5/PVE/LXC.pm line 2173 	eval {...} called at /usr/share/perl5/PVE/LXC.pm line 2141 	PVE::LXC::create_disks(HASH(0x58ea01aac038), 101, HASH(0x58ea01adf9d0), HASH(0x58ea01adf610)) called at /usr/share/perl5/PVE/API2/LXC.pm line 461 	eval {...} called at /usr/share/perl5/PVE/API2/LXC.pm line 383 	PVE::API2::LXC::__ANON__() called at /usr/share/perl5/PVE/AbstractConfig.pm line 299 	PVE::AbstractConfig::__ANON__() called at /usr/share/perl5/PVE/Tools.pm line 259 	eval {...} called at /usr/share/perl5/PVE/Tools.pm line 259 	PVE::Tools::lock_file_full("/run/lock/lxc/pve-config-101.lock", 10, 0, CODE(0x58ea01a5b0e8)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 302 	PVE::AbstractConfig::__ANON__("PVE::LXC::Config", 101, 10, 0, CODE(0x58ea01adfd28)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 322 	PVE::AbstractConfig::lock_config_full("PVE::LXC::Config", 101, 10, CODE(0x58ea01adfd28)) called at /usr/share/perl5/PVE/AbstractConfig.pm line 330 	PVE::AbstractConfig::lock_config("PVE::LXC::Config", 101, CODE(0x58ea01adfd28)) called at /usr/share/perl5/PVE/API2/LXC.pm line 528 	eval {...} called at /usr/share/perl5/PVE/API2/LXC.pm line 527 	PVE::API2::LXC::__ANON__("UPID:pvetest2:000020E2:00015EC3:66F51FB6:vzcreate:101:root\@pam:") called at /usr/share/perl5/PVE/RESTEnvironment.pm line 620 	eval {...} called at /usr/share/perl5/PVE/RESTEnvironment.pm line 611 	PVE::RESTEnvironment::fork_worker(PVE::RPCEnvironment=HASH(0x58e9fafefd88), "vzcreate", 101, "root\@pam", CODE(0x58ea01ac9d60)) called at /usr/share/perl5/PVE/API2/LXC.pm line 541 	PVE::API2::LXC::__ANON__(HASH(0x58ea01ae00b8)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499 	PVE::RESTHandler::handle("PVE::API2::LXC", HASH(0x58e9ff959990), HASH(0x58ea01ae00b8)) called at /usr/share/perl5/PVE/HTTPServer.pm line 180 	eval {...} called at /usr/share/perl5/PVE/HTTPServer.pm line 141 	PVE::HTTPServer::rest_handler(PVE::HTTPServer=HASH(0x58ea01a358e8), "::ffff:192.168.102.201", "POST", "/nodes/pvetest2/lxc", HASH(0x58ea01adebf0), HASH(0x58ea01a43da8), "extjs") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 938 	eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 912 	PVE::APIServer::AnyEvent::handle_api2_request(PVE::HTTPServer=HASH(0x58ea01a358e8), HASH(0x58ea01adf088), HASH(0x58ea01adebf0), "POST", "/api2/extjs/nodes/pvetest2/lxc") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1162 	eval {...} called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1154 	PVE::APIServer::AnyEvent::handle_request(PVE::HTTPServer=HASH(0x58ea01a358e8), HASH(0x58ea01adf088), HASH(0x58ea01adebf0), "POST", "/api2/extjs/nodes/pvetest2/lxc") called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1622 	PVE::APIServer::AnyEvent::__ANON__(AnyEvent::Handle=HASH(0x58ea01a35a38), "unprivileged=1&swap=512&rootfs=linstor_storage%3A8&memory=512"...) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1505 	AnyEvent::Handle::__ANON__(AnyEvent::Handle=HASH(0x58ea01a35a38)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 1315 	AnyEvent::Handle::_drain_rbuf(AnyEvent::Handle=HASH(0x58ea01a35a38)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Handle.pm line 2015 	AnyEvent::Handle::__ANON__() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Loop.pm line 248 	AnyEvent::Loop::one_event() called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent/Impl/Perl.pm line 46 	AnyEvent::CondVar::Base::_wait(AnyEvent::CondVar=HASH(0x58e9fafd5198)) called at /usr/lib/x86_64-linux-gnu/perl5/5.36/AnyEvent.pm line 2034 	AnyEvent::CondVar::Base::recv(AnyEvent::CondVar=HASH(0x58e9fafd5198)) called at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1936 	PVE::APIServer::AnyEvent::run(PVE::HTTPServer=HASH(0x58ea01a358e8)) called at /usr/share/perl5/PVE/Service/pvedaemon.pm line 52 	PVE::Service::pvedaemon::run(PVE::Service::pvedaemon=HASH(0x58ea0155a698)) called at /usr/share/perl5/PVE/Daemon.pm line 171 	eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 171 	PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x58ea0155a698)) called at /usr/share/perl5/PVE/Daemon.pm line 390 	eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 379 	PVE::Daemon::__ANON__(PVE::Service::pvedaemon=HASH(0x58ea0155a698), undef) called at /usr/share/perl5/PVE/Daemon.pm line 551 	eval {...} called at /usr/share/perl5/PVE/Daemon.pm line 549 	PVE::Daemon::start(PVE::Service::pvedaemon=HASH(0x58ea0155a698), undef) called at /usr/share/perl5/PVE/Daemon.pm line 659 	PVE::Daemon::__ANON__(HASH(0x58e9f98e74b0)) called at /usr/share/perl5/PVE/RESTHandler.pm line 499 	PVE::RESTHandler::handle("PVE::Service::pvedaemon", HASH(0x58ea0155a9e0), HASH(0x58e9f98e74b0), 1) called at /usr/share/perl5/PVE/RESTHandler.pm line 985 	eval {...} called at /usr/share/perl5/PVE/RESTHandler.pm line 968 	PVE::RESTHandler::cli_handler("PVE::Service::pvedaemon", "pvedaemon start", "start", ARRAY(0x58e9f990e500), ARRAY(0x58e9f9908640), undef, undef, undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 594 	PVE::CLIHandler::__ANON__(ARRAY(0x58e9f98e7678), CODE(0x58e9f9cf8c48), undef) called at /usr/share/perl5/PVE/CLIHandler.pm line 673 	PVE::CLIHandler::run_cli_handler("PVE::Service::pvedaemon", "prepare", CODE(0x58e9f9cf8c48)) called at /usr/bin/pvedaemon line 27

Oh, I see. There are a few different things going on here. I was also able to reproduce the behavior in the Proxmox GUI with a 3 replica storage pool in a 3 node cluster when one node is offline.

First, adding the virtual IP address to the LINSTOR controller HA configuration has no effect on the Proxmox GUI behavior. I initially suspected adding it might correct the behavior youโ€™re witnessing, but it does not. Oh well, I think itโ€™s a slightly better way to deploy the controller though.

Let me clarify my previous replies, when I suggested to use 3 replicas, it was in reference to the LINSTOR database only.

When it comes to the resource group used for Proxmox VMs, 2 replicas is perfectly fine, and is the LINSTOR default when the replica count is not specified.

This โ€œerrorโ€ is expected in this situation. In your currently configured 3 node cluster, youโ€™re attempting to create 3 replicas, however with only 2/3 nodes online, you cannot fulfill the autoplace requirements for creating a 3 replica resource. You would encounter a similar error trying to create a 2 replica resource when only 1/3 nodes are online.

So, to recap this has nothing to do with running LINSTORโ€™s controller in an HA fashion, but an effect from having a configured placement count higher than the available nodes. The Promox plugin is likely putting the LINSTOR storage into a semi-usable state. When a node fails, your existing workloads can keep running, you can migrate VMs, etc, but you cannot allocate new storage until all requirements are satisfied (3 replicas need 3 nodes with a storage pool in your case). Iโ€™ll check with the developers if the storage capacity dropping out of the GUI is intended, some side effect, or a bug, but there is nothing โ€œwrongโ€ with your cluster or configuration.

At the end of the day, you can use 3 replica resources with a 3 node cluster with no issues. With DRBDโ€™s replication, youโ€™ll be doubling your writes over the network with an extra replica, and in a saturated cluster with write-heavy workloads, 2 replicas may perform better. There are always tradeoffs, 2 replicas will consume less storage across the nodes, 3 replicas give you extra insurance, etc. You can also use both, simply configure two resource groups with similar settings, one with 2 replicas, and one with 3 replicas.

I have a little more information to pass on. The LINSTOR plugin for Proxmox passes information about the resource group from query-size-info to Proxmox.

Hereโ€™s the command:

linstor resource-group query-size-info pve-rg

When not enough nodes are online it will show 0 KiB:

โ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ
โ”Š MaxVolumeSize โ”Š AvailableSize โ”Š Capacity โ”Š Next Spawn Result โ”Š
โ•žโ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•ก
โ”Š         0 KiB โ”Š         0 KiB โ”Š    0 KiB โ”Š -                 โ”Š
โ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ

Thank you for the explanation.
I believe the issue is solved.
Would love to have and update regarding the GUI issue when possible.

1 Like

Sorry Iโ€™m not sure if itโ€™s the place or irrelevant,
Just wanted to know/share if you or anyone have any plan to write a blog/tutorial/video about how to make the controller highly available like you mentioned here.

I know you linked how to make the controller highly available and everything is well documented in the link. But for the last weeks, as a beginner I had some difficulties to follow HA-Controller part. And also there are lots of options to chose in the documentation and for a beginner I believe it will be helpful if there was a starting point, following which will make things work. Once itโ€™s up itโ€™s not hard to go deep.
So maybe consider doing a Part 2 or if the blog/Linbit allows community contribution I/someone else might contribute.

It can definitely be a lot to wrap your head around for the first time. There are a lot of moving components with LINSTOR, DRBD, and high availability.

Thanks for your interest in contributing. We do have an older blog post on this topic using Pacemaker. Iโ€™ve started some discussion on replacing the old post with the โ€œDRBD Reactor wayโ€ to align with the LINSTOR UG, but that is definitely some housekeeping weโ€™ll take care of on our end. DRBD Reactor is still fairly new compared to Pacemaker.