I’m currently able to use my setup with 2x Proxmox VE nodes, and an RPi5 as the tiebreaker. So far this seams to work. I’m also able to setup RDMA and get DRBD working over RDMA from one PVE node to the other. My questions:
This is an interesting question. Thanks for asking here. I checked in with some members of the team and received some feedback and guidance. What you propose doing should be possible, both for DRBD alone and using LINSTOR.
On a DRBD level, the transport is per connection {} , so yes.
For LINSTOR, you could try setting DRBD options on the network connections between nodes by using node-connection commands such as these:
linstor node-connection drbd-peer-options --transport rdma diskfulA diskfulB
linstor node-connection drbd-peer-options --transport tcp diskfulB disklessA
linstor node-connection drbd-peer-options --transport tcp diskfulA disklessA
If you introduce more nodes, you would need to enter more commands to create your desired mesh. You might want to review the Managing Network Interface Cards section within the LINSTOR User Guide if you have not already.
Report back your experiences after trying this with your setup if you would. It’s not a common (yet) setup so I’m interested in hearing how it goes for you.
Thanks for the quick response. I will report back as soon as I can try this out.
That works as expected. One thing that wasn’t obvious is that in your linstor commands the transport type (RDMA or TCP) are all capital letters. However, that will generate errors. It seems the drbd_transport_xxxx kernel driver is loaded based on the exact spelling of this option. So it must be lower case.
Further, and this is a bit off topic, sometime in the past I spent time figuring out how to tune the RDMA connections for DRBD. I don’t remember all the details, but I have the resulting options saved, and this command is the right place to put them, so I’m including them here to be complete. Caveat: these are tuned for my system/setup. Without tuning however, the RDMA connection is periodically dropped under load, in my case.
My diskfull nodes are pve4, and pve5 (PVE v8.3, ConnectX-4 NIC w/direct cable). My diskless tiebreaker is pve-rpi1 (which is an rpi5 w/ubuntu 24.04 LTS server).
These enable RDMA, and keep TCP for the diskless:
linstor node-connection drbd-peer-options --transport rdma --sndbuf-size 10485760 --rcvbuf-size 10485760 --max-buffers 131072 --max-epoch-size 16 --ping-timeout 150 --resync-rate 488636416 --c-fill-target 536870912 --c-max-rate 1468006400 --c-min-rate 488636416 pve4 pve5
linstor node-connection drbd-peer-options --transport tcp pve5 pve-rpi1
linstor node-connection drbd-peer-options --transport tcp pve4 pve-rpi1
To revert/unset these changes:
linstor node-connection drbd-peer-options --unset-transport --unset-sndbuf-size --unset-rcvbuf-size --unset-max-buffers --unset-max-epoch-size --unset-ping-timeout --unset-resync-rate --unset-c-fill-target --unset-c-max-rate --unset-c-min-rate pve4 pve5
linstor node-connection drbd-peer-options --unset-transport pve5 pve-rpi1
linstor node-connection drbd-peer-options --unset-transport pve4 pve-rpi1
and the proof that RDMA and TCP are both in use on the same resource:
root@pve5:~# cat /sys/kernel/debug/drbd/resources/pm-2a8c6505/connections/*/transport |grep transport_type
transport_type: rdma
transport_type: tcp
root@pve5:~# linstor volume list -r pm-2a8c6505 -p
+-------------------------------------------------------------------------------------------------------------------+
| Node | Resource | StoragePool | VolNr | MinorNr | DeviceName | Allocated | InUse | State |
|===================================================================================================================|
| pve-rpi1 | pm-2a8c6505 | DfltDisklessStorPool | 0 | 1003 | /dev/drbd1003 | | Unused | TieBreaker |
| pve4 | pm-2a8c6505 | pve-linstor-pool | 0 | 1003 | /dev/drbd1003 | 49.87 GiB | InUse | UpToDate |
| pve5 | pm-2a8c6505 | pve-linstor-pool | 0 | 1003 | /dev/drbd1003 | 49.87 GiB | Unused | UpToDate |
+-------------------------------------------------------------------------------------------------------------------+
It’s not a common (yet) setup
Yes, I agree. It feels odd to use Raspberry Pi and RDMA in the same sentence. 
Wow. That was fast turnaround on the testing and implementation. Appreciate all the details that you provided. I will bring these to the development team’s attention.
And sorry about the capitalized transport names in my original response. I should have verified those using linstor node-connection drbd-peer-options --help before my quick copy + paste from a chat with a colleague. My guess is that they were capitalized to draw my attention to them in the chat. Glad you caught that mistake though. Hopefully it did not waste too much of your time.
I’ll go back and edit my reply to correct the commands.
As for testing, there wasn’t much. I read/wrote data from/to the resource, saw packet counters increment where expected, and done.
I did just hit an issue, that I’m not fully able to describe yet, but it would seem that creating a new volume in PVE, ends up where the 2 diskfull nodes don’t connect at the drbd level. They both see the TCP tiebreaker, but not each other. I’m still trying to make sense of this, so I can report a more isolated issue, and what does/doesn’t work in this setup.
It’s odd because with RDMA on (and TCP TieBreaker) I was able to read/write without issue. And all the other existing resources configured identically seem to behave fine at that same time. So the connection is working, when it works.