Since you have a full 3 node setup, did you consider using ceph? If the hardware requirements are satisfied, I prefer a hyperconverged PVE+Ceph for a general purpose virtualization cluster. (There is a guide in the Proxmox Wiki how to install that). At least a 10 GbE Storage Network for Ceph is highly recommended. A small Cluster like yours will probably work very well with LinStor/DRBD with a dedicated 1 GbE Network.
A general recommendation from my side is to generally use 2 ssds for the OS either as hardware raid or software raid, as you prefer. (The SSDs for the OS may be cheaper models, not the expensive, high duration, high capacity, enterprise SSDs)
I would start with exploring LINSTOR and using our storage plugin for Proxmox:
Here’s a quick overview of some things you can do with LINSTOR and Proxmox:
Either way will work. If you have a hardware RAID card that pairs nicely with the drives in your servers, that is still a valid option. Some prefer ZFS, or spec their systems to build RAID-Z arrays.
A good question to ask yourself - If you built a single Proxmox node without replication or high availability, would you choose ZFS or your hardware RAID controller? DRBD/LINSTOR tends to be fairly agnostic about the storage layer underneath.
Keep in mind using ZFS with LINSTOR only uses ZVOLs, not the ZFS dataset/filesystem functionality. It is merely a way to slice up the storage in the system for replicated virtual machine disk images.
