Linstor DRBD on Proxmox Backup Server

LINSTOR
, ,
1

Hello everyone.

I have a « small » question about Linstor’s DRBD installation.

Few years ago, I made a 3-node PVE cluster which was using 2 PVE node as the virtual machine hosts and 1 last PVE node for backups.

The backup node is the Linstor controller (diskless) and the other 2 are Linstor satellites. (SSDs)

Recently, I’ve been looking at Proxmox Backup Server and was wondering if it could replace the PVE backup node.

Is it possible to properly install Linstor DRBD and make a Linstor cluster if I’m using PBS on the 3rd node ? It will be declared as « diskless », and used only for the vote.

Also, I must specify that PVE + PBS cohabitation on the 3rd node isn’t possible, only PBS alone.

Thank you in advance for your time and help.

If you need more information, I will do my best to provide it.

Kind regards,

2

Linstor doesn’t care what other software is running on the satellite nodes, and if you configure the third node without any storage pools, it will only use it as diskless.

Proxmox VE probably doesn’t care that there’s a third Linstor node. I guess it would see it if it asks the Linstor API for a list of all nodes. But it’s never going to try to place a VM there, since it’s not a Proxmox node.

So I don’t see any problem with running a satellite on a PBS server, just for the tiebreaker role.

1 Like
3

Hi.

First of all, thank you for your fast reply.

Before reading your answer, I thought about it a little more. What could possibly prevent the PBS node to be the Linstor controller ?

For me, it just comes down to adding LINBIT’s repository, and installing the required packages on the nodes, right ? And maybe even the PBS “controller” node doesn’t need the drbd-dkms/utils packages.

So I tried it while filtering some of my older commands that were originally for a PVE node and it worked.

I suppose that as long as you have a debian base that is the same version as the added LINBIT repository, it should work almost as intended. (The PBS’s debian version probably must match the “proxmox-$PVERS” variable while adding the repository, right now it’s PVERS=8 for debian 12 PBS 3 is debian 12 too)

If I notice some weird things happening later, I’ll update in this post.

Kind regards,

4

I think those packages are required, because the satellite needs to create /dev/drbdXXXX devices which communicate with those on the other two nodes - even though they don’t directly manage any local disks - if you want the tiebreaker (anti split-brain) functionality.

Whether it’s a “controller” or not is another issue. You could run the controller on any of the three nodes; its node type would be set as “combined” rather than “satellite” in linstor node create.

1 Like
5

Your statement seems to be absolutely right.

I just checked on my older “backup node” which is a PVE as the linstor “controller” and it does have /var/lib/linstor.d/.backup/ populated with resources from the other 2 nodes even as a diskless member of the linstor cluster.

I will now try to just add the required packages on the PBS node and see if a delete/add of the test resource will create these files.

6

I need to recreate the Linstor cluster properly (linstor node create…) because it seems that it saved the node’s capability at its creation :

Details:
    Supported storage providers: [diskless, lvm, lvm_thin, zfs, zfs_thin, file, file_thin, remote_spdk, ebs_init, ebs_target]
    Supported resource layers  : [nvme, writecache, cache, storage]
    Unsupported storage providers:
        SPDK: IO exception occured when running 'rpc.py spdk_get_version': Cannot run program "rpc.py": error=2, No such file or directory
        EXOS: IO exception occured when running 'lsscsi --version': Cannot run program "lsscsi": error=2, No such file or directory
        STORAGE_SPACES: This tool does not exist on the Linux platform.
        STORAGE_SPACES_THIN: This tool does not exist on the Linux platform.

    Unsupported resource layers:
        DRBD: DRBD version has to be >= 9. Current DRBD version: 8.4.11
        LUKS: IO exception occured when running 'cryptsetup --version': Cannot run program "cryptsetup": error=2, No such file or directory
        BCACHE: IO exception occured when running 'make-bcache -h': Cannot run program "make-bcache": error=2, No such file or directory

Now I have another problem that could be because of the fact that I installed it in the “wrong” order ?

DRBD version on the PBS node is detected as 8.4.11, however I did install the right version :

drbd-dkms/unknown,now 9.2.14-1 all [installed]
drbd-utils/unknown,now 9.31.0-1 amd64 [installed]

For now, I must find why because the resources won’t be created on the PBS node if it’s not registered as [DRBD] “capable”.

7

/var/lib/linstor.d/.backup/

Interesting, I didn’t know about that until now :slight_smile:

DRBD version on the PBS node is detected as 8.4.11, however I did install the right version :frowning:

Try cat /sys/module/drbd/version. If it’s wrong, then rmmod drbd and modprobe drbd. If that fixes the version then restart the satellite. Recheck linstor node info -f.

If necessary linstor node reconnect ..., or at worst, linstor node delete and linstor node create to re-add the node.

1 Like
8

I knew about rmmod drbd and modprobe drbd but it didn’t work this time, weird.

I don’t know if it’s because I tried to apt install pve-headers-$(uname -r) on the PBS, then apt -y install --reinstall drbd-dkms drbd-utils and finally another rmmod drbd ; modprobe drbd and it worked :
version: 9.2.14 (api:2/proto:118-123)
GIT-hash: a1e7c10e591a844b327da120d169df7da7c933b7 build by root@backup, 2025-07-29 13:38:40
Transports (api:21): tcp (9.2.14)

I will probably reinstall the nodes cleanly later as this is just some tests to get everything working as intended.
I’m kinda worried about the PBS showing a “pve” kernel now :
uname -r -> 6.8.12-12-pve or was it always the case ?

After a little check on the actual kernel on my other PBS node, which is 100% alone from PVE environment, is also a “pve” one :
6.8.12-11-pve
I’m kinda relieved that it’s normal and they are just shipping “-pve” named kernel for PBS.

Regarding the linstor cluster ; after deleting everything old with linstor commands (nodes, resource-definition, resource-group…), I created the cluster again and now the PBS backup node is able to DRBD and in fact Tie-Breaking with my test resource :

╭────────────────────────────────────────────────────────────────────────╮
┊ Node   ┊ DRBD ┊ LUKS ┊ NVMe ┊ Cache ┊ BCache ┊ WriteCache ┊ Storage ┊
╞════════════════════════════════════════════════════════════════════════╡
┊ backup ┊ +    ┊ -    ┊ +    ┊ +     ┊ -      ┊ +          ┊ +       ┊
┊ srv1   ┊ +    ┊ -    ┊ +    ┊ +     ┊ -      ┊ +          ┊ +       ┊
┊ srv2   ┊ +    ┊ -    ┊ +    ┊ +     ┊ -      ┊ +          ┊ +       ┊
╰────────────────────────────────────────────────────────────────────────╯

root@backup:~# linstor r l
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node   ┊ Layers       ┊ Usage  ┊ Conns ┊      State ┊ CreatedOn           ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pm-61dae041  ┊ backup ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-07-29 14:32:32 ┊
┊ pm-61dae041  ┊ srv1   ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-07-29 14:32:32 ┊
┊ pm-61dae041  ┊ srv2   ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-07-29 14:32:33 ┊
┊ pm-d2da2116  ┊ backup ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-07-29 14:32:26 ┊
┊ pm-d2da2116  ┊ srv1   ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-07-29 14:32:26 ┊
┊ pm-d2da2116  ┊ srv2   ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-07-29 14:32:27 ┊
┊ pm-fa6753cf  ┊ backup ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊ TieBreaker ┊ 2025-07-29 14:32:29 ┊
┊ pm-fa6753cf  ┊ srv1   ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-07-29 14:32:29 ┊
┊ pm-fa6753cf  ┊ srv2   ┊ DRBD,STORAGE ┊ Unused ┊ Ok    ┊   UpToDate ┊ 2025-07-29 14:32:30 ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────╯

As we can see, the backup node also created the resource files (1 resource per Windows VM disk):

root@backup:~# ls -la /var/lib/linstor.d/
total 24
drwxr-xr-x  3 root root 4096 Jul 29 14:32 .
drwxr-xr-x 33 root root 4096 Jul 29 12:15 ..
drwxr-xr-x  2 root root 4096 Jul 29 14:32 .backup
-rw-r--r--  1 root root    0 Jul 29 14:32 loop_device_mapping
-rw-r--r--  1 root root 2010 Jul 29 14:32 pm-61dae041.res
-rw-r--r--  1 root root 2010 Jul 29 14:32 pm-d2da2116.res
-rw-r--r--  1 root root 2010 Jul 29 14:32 pm-fa6753cf.res

root@backup:~# ls -la /var/lib/linstor.d/.backup
total 20
drwxr-xr-x 2 root root 4096 Jul 29 14:32 .
drwxr-xr-x 3 root root 4096 Jul 29 14:32 ..
-rw-r--r-- 1 root root 2010 Jul 29 14:32 pm-61dae041.res
-rw-r--r-- 1 root root 2010 Jul 29 14:32 pm-d2da2116.res
-rw-r--r-- 1 root root 2010 Jul 29 14:32 pm-fa6753cf.res
9

Welcome to the forums.

Looks like you’re on the right track. LINSTOR needs at least two diskful nodes with storage, the 3rd node can be exclusively diskless. This means it is also a LINSTOR satellite node, and DRBD is required even if there isn’t local storage. This 3rd node is also a good candidate to run the LINSTOR controller as a combined LINSTOR node.

Your cluster configuration is exactly what we refer to as the “minimal” LINSTOR deployment pattern, you just so happen be using PBS on your 3rd node.

One thing to mention about kernels, it is always recommended to run the same kernels throughout a cluster if possible (not just the same DRBD kernel module versions). If the “PVE kernel” is easily installable on the PBS node, even if you have to add a Proxmox repo to make the same kernel available on PBS, that’s what I would recommend.

1 Like
10

Hi and thank you Ryan :slight_smile:

I will gladly remember the kernel advice.

I’m now going to try to make the SSL/TLS encryption work for satellites communication and also look into NIC failover.

For example : all of my 3 nodes have 2 NIC, 1 WAN and 1 LAN. If the LAN NIC goes down, I want the cluster to stay alive by switching the controller/satellites communications that were on the LAN NIC, to the WAN NIC. Even if the WAN NIC is several times slower, it should be better than risking having the cluster going down.

To keep things clear, I will make another post about it and if it is permitted by the forum rules, link it in this reply for the “post continuity”.

Kind regards,

1 Like