While ZFS with ZVols is mentioned as supported by Linstor/DRBD in the official documentation, I cannot find any actual recommendation to prefer it over LVM, neither can I find any recommendations on how to set it up to get the most out of Linstor/DRBD.
Just in terms of features, ZFS is clearly superior to LVM. The following feature comparison table is taken from Linbit’s own Comparing LINSTOR Back-End Storage Types.
| Feature | LVM | LVM | ZFS |
| | | (Thin) | |
| ------------------------------------- | ---- | ------ | -----|
| Cache management | Yes | Yes | Yes |
| Checksumming of data | No | No | Yes |
| Data compression | No | No | Yes |
| Data integrity | No | No | Yes |
| Deduplication | No | No | Yes |
| Encryption | No | No | Yes |
| Multiple disk failures protection | Yes | Yes | Yes |
| Multiple RAID levels | Yes | Yes | Yes |
| Online capacity extend and shrink | Yes | Yes | Yes |
| Online scrubbing and resilvering | No | No | Yes |
| Partition resizing | Yes | Yes | Yes |
| POSIX access control lists (ACLs) | No | No | Yes |
| Snapshots (supported in LINSTOR) | No | Yes | Yes |
| Thin provisioning | No | Yes | Yes |
ZFS support in Linux distributions is not as common as LVM yet.
I understand that there are circumstances where ZFS should be avoided. I would not put ZFS on top of an existing (hardware/software) RAID. I would also not use ZFS on machines with very limited resources.
But if I have access to a powerful server with plenty of ECC RAM, it seems to me that ZFS should be a preferred option.
But, alas, there is no such recommendation available, and I’d like to understand why not. Maybe I’m missing something, my practical experience with Linstor/DRBD is admittedly close to 0.
More concretely, I’m in the process of building new machines for the purpose of running a LINSTOR cluster and I would love some guidance or recommendations on how to build the storage backend, preferrably without having to add a hardware RAID controller.
ZFS is fully supported as mentioned in the documentation. However, I wouldn’t necessarily suggest ZFS over LVM. As you have mentioned, ZFS is much more feature complete, but do I need, or will I even use all those features?
As already mentioned, ZFS is relatively new by comparison, and if we don’t need those features then why not just keep it simple? Additionally, as you also already mentioned, ZFS may require a bit more system resources.
Storage backends will vary widely in both their capabilities and what is required of them. Some people like shiny new vehicles with all the latest hi-tech features, and some people prefer a 20 year old work truck that has proven itself reliable and easy to work on for decades. In the end I think it comes down to preference, and what it is you need to achieve.
Don’t get me wrong, I think thin LVM is a great concept and I use it even in production, albeit on a very small scale.
For me, the big difference here is that with LVM, I’d still need another layer of software or hardware for local disk redundancy while this is built-in with ZFS. (I know that LVM can do RAID in theory but when it comes to soft RAID, mdraid seems more popular and more stable).
How do you build your RAIDs, by the way? Do you have a preference for hardware or software, and if software, is it mdraid?
Not being an expert but someone who wants to make sure that as little as possible can go wrong, things like “check-summing of data” and “data integrity” do sound like huge endorsements.
I’m genuinely interested to understand why you say we don’t need those features. In particular, when researching thin LVM, the words “metadata corruption” do come up quite often. It sounds scary and I guess such a corruption would be replicated to all other nodes. (As a side note, I myself lost a thin pool once because of such a corruption caused by a snap package that ran another version of LVM tools than my host).
In summary, you clearly seem to favor LVM over ZFS and I would love to better understand why.
I prefer battery/capicitor backed hardware RAID when possible, but I will use LVM’s raid feature if needed. I haven’t benchmarked or compared LVM Raid to mdraid, but will almost always lean toward the stack with fewer components when possible.
Checksumming is CPU intensive. Not everyone wants or can afford this overhead. I’m not too sure what exactly is meant there with the “data integrity” column, but it sounds redundant with the checksumming.
Well, don’t do that. I have had no such issues with LVM myself.
A LINSTOR cluster with what equipment and to provide storage for what use? There are almost countless combinations of hardware and data consumers.
That’s really helpful information. Can you maybe share an example config of LVM RAID you set up? I’m also curious what you do to make sure you have sufficient integrity. I read something about MD integrity that can be also turned on with LVM RAID, are you using that?
I got my hands on some powerful HP servers with a mix of SAS and NVMe bays. The SAS bay is attached to a RAID controller, NVMe (by definition, as I understand it) isn’t. Servers will be connected over SFP28 (2x) interfaces. My primary goal is to create a storage and virtualization cluster with these servers. I’ve worked with Incus (a fork of LXD) before and I like it a lot but it doesn’t support Linstor natively which is why I’m going to try out Proxmox. I’m just at the beginning of my journey with Linstor/DRBD so I expect to become a frequent visitor here.
I have had no such issues with LVM myself.