I got it up and runnnig.
Here is my documentation, maybe it helps someone else. If you find issues please leave a comment and correction. Thank you.
prerequisite: drbd installed and configured. see Proxmox - intall Linstor DRBD
do the following on all cluster nodes (not required on qdevice)
install drbd-reactor and linstor gateway
apt install drbd-reactor linstor-gateway psmisc
install the resource-agents. the gateway requires them to mange failover in tandem with drbd-reactor.
apt install --no-install-recommends resource-agents
without --no-install-recommends this would install a cloud-api too. we dont need this.
install nfs-ganesha
apt install nfs-ganesha nfs-ganesha-vfs nfs-common
verify if the required ocf agent is present:
ls -la /usr/lib/ocf/resource.d/heartbeat/ | grep nfs
if ganesha-nfs does not exist we have to install it manually. (at the time of writing ganesha-nfs was not included in the resource-agents package)
download ganesha-nfs
curl -o /usr/lib/ocf/resource.d/heartbeat/ganesha-nfs \
https://raw.githubusercontent.com/ClusterLabs/resource-agents/main/heartbeat/ganesha-nfs
chmod 755 /usr/lib/ocf/resource.d/heartbeat/ganesha-nfs
create/edit /etc/linstor/linstor_satellite.toml
[files]
allowExtFiles = [
"/etc/systemd/system",
"/etc/systemd/system/linstor-satellite.service.d",
"/etc/drbd-reactor.d"
]
restart linstor-satellite service:
systemctl restart linstor-satellite
configure drbd-reactor auto-reload:
cp /usr/share/doc/drbd-reactor/examples/drbd-reactor-reload.path /etc/systemd/system/
cp /usr/share/doc/drbd-reactor/examples/drbd-reactor-reload.service /etc/systemd/system/
enable and start drbd-reactor reload service
systemctl enable --now drbd-reactor-reload.path
nfs-server and nfs-ganesha are mutually exclusive, we cannot run both at the same time.
if installed disable nfs-server service:
systemctl disable --now nfs-server.service
or uninstall it.
disable nfs-ganesha service:
systemctl disable --now nfs-ganesha.service
drbd-reactor manages service (start/stop) based on active/standby node. we do not cover active/active here since this requires a specialized file-system e.g. gluster-fs
verify all nfs server services are disabled and down:
systemctl status nfs-server.service
systemctl status nfs-ganesha.service
at this point the installation and configuration is done.
check the linstor-gateway health:
linstor-gateway check-health
expected output:
root@tvbox:~# linstor-gateway check-health
Checking agent requirements.
[✓] System Utilities
[✓] LINSTOR
[✓] drbd-reactor
[✓] Resource Agents
[!] iSCSI
✗ The targetcli tool is not available
exec: "targetcli": executable file not found in $PATH
Please install the targetcli package
Hint: targetcli is only required for the LIO target (lio-t) backend. If you are not planning on using LIO target, try e
xcluding it via `--iscsi-backends`.
✗ The scstadmin tool is not available
exec: "scstadmin": executable file not found in $PATH
Please install the scstadmin package
Hint: scstadmin is only required for the SCST backend. If you are not planning on using SCST, try excluding it via `--i
scsi-backends`.
✗ Kernel module scst is not loaded
Execute modprobe scst or install package scst
✗ Kernel module iscsi_scst is not loaded
Execute modprobe iscsi_scst or install package scst
✗ Kernel module scst_vdisk is not loaded
Execute modprobe scst_vdisk or install package scst
✗ Process iscsi-scstd is not running
iscsi-scstd not found in process list
Make sure that:
• the scst package is installed
• the iscsi-scstd process is started
[!] NVMe-oF
✗ The nvmetcli tool is not available
exec: "nvmetcli": executable file not found in $PATH
Please install the nvmetcli package
Hint: nvmetcli is not (yet) packaged on all distributions. See https://git.infradead.org/users/hch/nvmetcli.git for ins
tructions on how to manually install it.
[!] NFS
✗ Service nfs-server.service is in the wrong state (not loaded).
This systemd service conflicts with LINSTOR Gateway.
It needs to be loaded, but not started.
Make sure that:
• the nfs-server package is installed
• the nfs-server.service systemd unit is stopped and disabled
Execute systemctl disable --now nfs-server.service to disable and stop the service.
FATA[0000] Health check failed: found 3 issues
explanation:
- ignore iSCSI, NVMe-oF states unless you need these options, then follow the instructions and install the missing packages
- ignore the NFS state. (the current version 2.2 of linstor-gateway only searches for nfs-server but not for nfs-ganesha.
- ignore the final FATAL status
- debug and fix if System Utilities, LINSTOR, drbd-reactor or Ressource Agents have issues!
create nfs export (on ONE node only):
linstor-gateway nfs create <resource-name> <service-ip>/<cidr> <size> --implementation ganesha
# e.g.
# linstor-gateway nfs create nfs-test 192.168.10.25/23 10G --implementation ganesha
verify the export:
root@gateway:~# linstor-gateway nfs list
┌──────────┬──────────────────┬─────────────────┬───────────────────────────────┬───────────────┐
│ Resource │ Service IP │ Service state │ NFS export │ LINSTOR state │
├──────────┼──────────────────┼─────────────────┼───────────────────────────────┼───────────────┤
│ nfs-test │ 192.168.10.25/23 │ Started (tvbox) │ /srv/gateway-exports/nfs-test │ OK │
└──────────┴──────────────────┴─────────────────┴───────────────────────────────┴───────────────┘
root@gateway:~#
try to ping the service ip (vip): (from ALL nodes)
ping 192.168.10.25
verify drbd-reactor status: (on ALL nodes)
drbd-reactorctl status
expected output on the active node:
root@tvbox:~# drbd-reactorctl status
/etc/drbd-reactor.d/linstor-gateway-nfs-nfs-test.toml:
Promoter: Resource nfs-test currently active on this node
● drbd-services@nfs\x2dtest.target
● ├─ drbd-promote@nfs\x2dtest.service
● ├─ ocf.rs@portblock_nfs\x2dtest.service
● ├─ ocf.rs@fs_cluster_private_nfs\x2dtest.service
● ├─ ocf.rs@fs_1_nfs\x2dtest.service
● ├─ ocf.rs@service_ip_nfs\x2dtest.service
● ├─ ocf.rs@nfsserver_nfs\x2dtest.service
● └─ ocf.rs@portunblock_nfs\x2dtest.service
expected output on the standby nodes:
root@gateway:~# drbd-reactorctl status
/etc/drbd-reactor.d/linstor-gateway-nfs-nfs-test.toml:
Promoter: Resource nfs-test currently active on node 'tvbox'
○ drbd-services@nfs\x2dtest.target
○ ├─ drbd-promote@nfs\x2dtest.service
○ ├─ ocf.rs@portblock_nfs\x2dtest.service
○ ├─ ocf.rs@fs_cluster_private_nfs\x2dtest.service
○ ├─ ocf.rs@fs_1_nfs\x2dtest.service
○ ├─ ocf.rs@service_ip_nfs\x2dtest.service
○ ├─ ocf.rs@nfsserver_nfs\x2dtest.service
○ └─ ocf.rs@portunblock_nfs\x2dtest.service
verify drbd ressources for nfs: (on the linstor drbd controller)
linstor resource list
expected output:
root@gateway:~# linstor node list
╭──────────────────────────────────────────────────────────────────────────╮
│ ResourceName │ Node │ Layers │ Usage │ Conns │ State │ Vote │
╞══════════════════════════════════════════════════════════════════════════╡
│ nfs-test │ gateway │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │
│ nfs-test │ tvbox │ DRBD,STORAGE │ InUse │ Ok │ UpToDate │ Yes │
verify quorum: (on the linstor drbd controller)
linstor node list
expected output:
root@gateway:~# linstor node list
╭──────────────────────────────────────────────────────────────────────╮
│ Node │ Platform │ NodeType │ Addresses │ State │
╞══════════════════════════════════════════════════════════════════════╡
│ gateway │ LINUX │ COMBINED │ 192.168.2.20:3366 (PLAIN) │ Online │
│ qdevice │ LINUX │ SATELLITE │ 192.168.2.100:3366 (PLAIN) │ Online │
│ tvbox │ LINUX │ COMBINED │ 192.168.2.30:3366 (PLAIN) │ Online │
╰──────────────────────────────────────────────────────────────────────╯
verify failover:
start ping to nfs export (vip) e.g. on standby node
ping 192.168.10.25
force failover (on active node):
drbd-reactorctl evict
ping should not loose many packets
verify status on all nodes with:
drbd-reactorctl status
mount the export: (on a workstation, needs nfs-common installed)
sudo mkdir -p /mnt/nfs/test
sudo mount -t nfs4 -v 192.168.10.25:/srv/gateway-exports/nfs-test /mnt/nfs/test
ls -la /mnt/nfs/test
echo "hello nfs" > /mnt/nfs/test/test.txt
cat /mnt/nfs/test/test.txt