NFS Ganesha on Proxmox via Gateway

Hi

I just saw this video form Linbit: https://www.youtube.com/watch?v=vsNpexPj-1w
This is exactly what I need.
Could someone please make a guide how to correctly install and configure this on Proxmox?

I’m not sure if:

  • I need DRBD reactor
  • I still need to install nfs-kernel-server if I only want to use Ganesha (nfs-ganesha)
  • how to setup DRBD controller and gateway in a redundant way.

So far I have not found a tutorial or guide describing how to configure redundant DRBD controllers on a Proxmox cluster.

Thank you very much!

I got it up and runnnig.

Here is my documentation, maybe it helps someone else. If you find issues please leave a comment and correction. Thank you.

prerequisite: drbd installed and configured. see Proxmox - intall Linstor DRBD

do the following on all cluster nodes (not required on qdevice)

install drbd-reactor and linstor gateway

apt install drbd-reactor linstor-gateway psmisc

install the resource-agents. the gateway requires them to mange failover in tandem with drbd-reactor.

apt install --no-install-recommends resource-agents

without --no-install-recommends this would install a cloud-api too. we dont need this.

install nfs-ganesha

apt install nfs-ganesha nfs-ganesha-vfs nfs-common

verify if the required ocf agent is present:

ls -la /usr/lib/ocf/resource.d/heartbeat/ | grep nfs

if ganesha-nfs does not exist we have to install it manually. (at the time of writing ganesha-nfs was not included in the resource-agents package)
download ganesha-nfs

curl -o /usr/lib/ocf/resource.d/heartbeat/ganesha-nfs \
     https://raw.githubusercontent.com/ClusterLabs/resource-agents/main/heartbeat/ganesha-nfs
chmod 755 /usr/lib/ocf/resource.d/heartbeat/ganesha-nfs

create/edit /etc/linstor/linstor_satellite.toml

[files]
allowExtFiles = [
  "/etc/systemd/system",
  "/etc/systemd/system/linstor-satellite.service.d",
  "/etc/drbd-reactor.d"
]

restart linstor-satellite service:

systemctl restart linstor-satellite

configure drbd-reactor auto-reload:

cp /usr/share/doc/drbd-reactor/examples/drbd-reactor-reload.path /etc/systemd/system/
cp /usr/share/doc/drbd-reactor/examples/drbd-reactor-reload.service /etc/systemd/system/

enable and start drbd-reactor reload service

systemctl enable --now drbd-reactor-reload.path

nfs-server and nfs-ganesha are mutually exclusive, we cannot run both at the same time.

if installed disable nfs-server service:

systemctl disable --now nfs-server.service

or uninstall it.

disable nfs-ganesha service:

systemctl disable --now nfs-ganesha.service

drbd-reactor manages service (start/stop) based on active/standby node. we do not cover active/active here since this requires a specialized file-system e.g. gluster-fs

verify all nfs server services are disabled and down:

systemctl status nfs-server.service
systemctl status nfs-ganesha.service

at this point the installation and configuration is done.
check the linstor-gateway health:

linstor-gateway check-health

expected output:

root@tvbox:~# linstor-gateway check-health  
Checking agent requirements.  
  
[✓] System Utilities  
[✓] LINSTOR  
[✓] drbd-reactor  
[✓] Resource Agents  
[!] iSCSI  
   ✗ The targetcli tool is not available  
     exec: "targetcli": executable file not found in $PATH  
     Please install the targetcli package  
     Hint: targetcli is only required for the LIO target (lio-t) backend. If you are not planning on using LIO target, try e  
xcluding it via `--iscsi-backends`.  
   ✗ The scstadmin tool is not available  
     exec: "scstadmin": executable file not found in $PATH  
     Please install the scstadmin package  
     Hint: scstadmin is only required for the SCST backend. If you are not planning on using SCST, try excluding it via `--i  
scsi-backends`.  
   ✗ Kernel module scst is not loaded  
     Execute modprobe scst or install package scst  
   ✗ Kernel module iscsi_scst is not loaded  
     Execute modprobe iscsi_scst or install package scst  
   ✗ Kernel module scst_vdisk is not loaded  
     Execute modprobe scst_vdisk or install package scst  
   ✗ Process iscsi-scstd is not running  
     iscsi-scstd not found in process list  
     Make sure that:  
     • the scst package is installed  
     • the iscsi-scstd process is started  
[!] NVMe-oF  
   ✗ The nvmetcli tool is not available  
     exec: "nvmetcli": executable file not found in $PATH  
     Please install the nvmetcli package  
     Hint: nvmetcli is not (yet) packaged on all distributions. See https://git.infradead.org/users/hch/nvmetcli.git for ins  
tructions on how to manually install it.  
[!] NFS  
   ✗ Service nfs-server.service is in the wrong state (not loaded).  
     This systemd service conflicts with LINSTOR Gateway.  
     It needs to be loaded, but not started.  
     Make sure that:  
     • the nfs-server package is installed  
     • the nfs-server.service systemd unit is stopped and disabled  
     Execute systemctl disable --now nfs-server.service to disable and stop the service.  
  
FATA[0000] Health check failed: found 3 issues

explanation:

  • ignore iSCSI, NVMe-oF states unless you need these options, then follow the instructions and install the missing packages
  • ignore the NFS state. (the current version 2.2 of linstor-gateway only searches for nfs-server but not for nfs-ganesha.
  • ignore the final FATAL status
  • debug and fix if System Utilities, LINSTOR, drbd-reactor or Ressource Agents have issues!

create nfs export (on ONE node only):

linstor-gateway nfs create <resource-name> <service-ip>/<cidr> <size> --implementation ganesha

# e.g.
# linstor-gateway nfs create nfs-test 192.168.10.25/23 10G --implementation ganesha

verify the export:

root@gateway:~# linstor-gateway nfs list  
┌──────────┬──────────────────┬─────────────────┬───────────────────────────────┬───────────────┐  
│ Resource │    Service IP    │  Service state  │          NFS export           │ LINSTOR state │  
├──────────┼──────────────────┼─────────────────┼───────────────────────────────┼───────────────┤  
│ nfs-test │ 192.168.10.25/23 │ Started (tvbox) │ /srv/gateway-exports/nfs-test │ OK            │  
└──────────┴──────────────────┴─────────────────┴───────────────────────────────┴───────────────┘  
root@gateway:~#

try to ping the service ip (vip): (from ALL nodes)

ping 192.168.10.25

verify drbd-reactor status: (on ALL nodes)

drbd-reactorctl status

expected output on the active node:

root@tvbox:~# drbd-reactorctl status  
/etc/drbd-reactor.d/linstor-gateway-nfs-nfs-test.toml:  
Promoter: Resource nfs-test currently active on this node  
● drbd-services@nfs\x2dtest.target  
● ├─ drbd-promote@nfs\x2dtest.service  
● ├─ ocf.rs@portblock_nfs\x2dtest.service  
● ├─ ocf.rs@fs_cluster_private_nfs\x2dtest.service  
● ├─ ocf.rs@fs_1_nfs\x2dtest.service  
● ├─ ocf.rs@service_ip_nfs\x2dtest.service  
● ├─ ocf.rs@nfsserver_nfs\x2dtest.service  
● └─ ocf.rs@portunblock_nfs\x2dtest.service

expected output on the standby nodes:

root@gateway:~# drbd-reactorctl status  
/etc/drbd-reactor.d/linstor-gateway-nfs-nfs-test.toml:  
Promoter: Resource nfs-test currently active on node 'tvbox'  
○ drbd-services@nfs\x2dtest.target  
○ ├─ drbd-promote@nfs\x2dtest.service  
○ ├─ ocf.rs@portblock_nfs\x2dtest.service  
○ ├─ ocf.rs@fs_cluster_private_nfs\x2dtest.service  
○ ├─ ocf.rs@fs_1_nfs\x2dtest.service  
○ ├─ ocf.rs@service_ip_nfs\x2dtest.service  
○ ├─ ocf.rs@nfsserver_nfs\x2dtest.service  
○ └─ ocf.rs@portunblock_nfs\x2dtest.service

verify drbd ressources for nfs: (on the linstor drbd controller)

linstor resource list

expected output:

root@gateway:~# linstor node list
╭──────────────────────────────────────────────────────────────────────────╮  
│ ResourceName │ Node    │ Layers       │ Usage  │ Conns │    State │ Vote │  
╞══════════════════════════════════════════════════════════════════════════╡  
│ nfs-test     │ gateway │ DRBD,STORAGE │ Unused │ Ok    │ UpToDate │ Yes  │  
│ nfs-test     │ tvbox   │ DRBD,STORAGE │ InUse  │ Ok    │ UpToDate │ Yes  │

verify quorum: (on the linstor drbd controller)

linstor node list

expected output:

root@gateway:~# linstor node list
╭──────────────────────────────────────────────────────────────────────╮  
│ Node    │ Platform │ NodeType  │ Addresses                  │ State  │  
╞══════════════════════════════════════════════════════════════════════╡  
│ gateway │ LINUX    │ COMBINED  │ 192.168.2.20:3366 (PLAIN)  │ Online │  
│ qdevice │ LINUX    │ SATELLITE │ 192.168.2.100:3366 (PLAIN) │ Online │  
│ tvbox   │ LINUX    │ COMBINED  │ 192.168.2.30:3366 (PLAIN)  │ Online │  
╰──────────────────────────────────────────────────────────────────────╯

verify failover:
start ping to nfs export (vip) e.g. on standby node

ping 192.168.10.25

force failover (on active node):

drbd-reactorctl evict

ping should not loose many packets
verify status on all nodes with:

drbd-reactorctl status

mount the export: (on a workstation, needs nfs-common installed)

sudo mkdir -p /mnt/nfs/test
sudo mount -t nfs4 -v 192.168.10.25:/srv/gateway-exports/nfs-test /mnt/nfs/test
ls -la /mnt/nfs/test
echo "hello nfs" > /mnt/nfs/test/test.txt
cat /mnt/nfs/test/test.txt