Linstor on Proxmox utilising NVMe-of with RDMA over RoCE

LINBIT SDS Integrations Proxmox VE
, ,
1

Hi Everyone,

This is my first post to the forum after sucssefully configuring Linstor on a 2 node Proxmox cluster.

I would now like to optimse the networking by implementing NVMe-of with RDMA over RoCE utilsiing Mellanox connectx-4 lx but have not been able to find any offical documentation and or google seaches that outlines exaclly what needs to be done.

Could anyone assit and or point be to a wesbite/guide/documentation on how to enable this?

Thanks

2

For the NIC: https://enterprise-support.nvidia.com/s/article/howto-configure-nvme-over-fabrics

For Linstor: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-nvme-layer

But you’d be losing the main benefit (IMO) of Linstor, which is DRBD replication.

NVMe-oF/NVMe-TCP allows LINSTOR to connect diskless resources to a node with the same resource where the data is stored over NVMe fabrics. This leads to the advantage that resources can be mounted without using local storage by accessing the data over the network. LINSTOR is not using DRBD in this case, and therefore NVMe resources provisioned by LINSTOR are not replicated, the data is stored on one node.

Linstor then just becomes a volume manager: e.g. it can create an LVM volume on node A, and you can access it from node B using NVMe-oF or NVMe-TCP.

(Note: I’ve never run Linstor in this way, this is just from reading the documentation)

2 Likes
3

This tech guide and this blog offer an alternative approach to using LINSTOR with Proxmox.

Neither use the LINSTOR driver to integrate directly with Proxmox, but instead create HA NVMe-oF clusters that you can attach Proxmox to.