I eventually did get it working and I think something like this was the issue! 2 node cluster with a witness is now working, I recently discovered linstor gateway which seems to be the solution to my problem of utilizing linstor/drbd with nvme/roce, but i’ve been completely unable to switch the nvme transport from tcp to rdma for whatever reason, I haven’t seen anything on the docs for it either
EDIT:
looks like this may be possible with drbd-reactorctl edit ?
drbd-reactorctl restart doesn’t same to always make the new config take for whatever reason though. I tried rebooting my node and it looked like it reverted back to tcp.
drbd-reactorctl disable
followed by drbd-reactorctl enable might be a more reliable way because I can see the transport change in dmesg this time
[ 683.575965] nvmet: adding nsid 1 to subsystem schemesec:nvme:rg-pve0
[ 683.586280] nvmet_rdma: enabling port 0 (192.168.20.51:4420)
Really unsure how to get this to stick if its not surviving reboot…
Being able to use nvme/RoCE would be super cool and i feel like im really close.