How can I use linstor as shared file storage for docker swarm, home directories, etc

General
,
1

I’m trying to find a distributed filesystem for my hyperconverged proxmox homelab with 4 nodes. Linstore seems interesting, but as far as I can tell, it only offers block storage and no filesystem a la CephFS, gluster etc.

What I’m trying to achieve is:
Shared storage for VM images to enable proxmox HA
This seems pretty straightforward with linstore. I saw the blog post about it and it was very comprehensive.

Shared storage for docker swarm
I saw the blog post about the linstore volume driver. Unfortunately, it seems the plugin repo hasn’t been updated in a while, and I saw a comment on one of the issues about it possibly getting archived because it’s not seeing a lot of use on the business side.

Furthermore, there’s a huge problem with using named docker volume, and that is that it makes permissions set up very painful and complicated. I run all my containers as non-root users, so you have to play tricks with the volume which I’d like to avoid. Currently, I just bind mount everything and set up permissions on the host.

Can I do anything with linstore like this?

Shared home directories
I currently have home directories on nfs across all my nodes, VMs, etc. I saw that I can export nfs shares, but apparently there is no support for permissions, which is a non-starter.

Is there an easy (or any, really) way to get what I want? It seems like if I’m going to use linstore, I’d have to set up a VM or LXC, create a giant disk on linstore storage for all my data, then put nfs in there to export to all my VMs, containers, etc. But this seems like it would introduce a lot of unnecessary overhead. It also means all disk access would go necessarily go through one node.

The alternatives I’m looking at are GlusterFS and CephFS, with virtiofsd to mount the data into VMs. This is much more straightforward, but generally performance is pretty awful.

2

Linstore seems interesting, but as far as I can tell, it only offers block storage and no filesystem a la CephFS, gluster etc.

Have a look at Linstor Gateway which gives you a highly-available NFS server.

Can you be more specific about what problem you see with NFS permissions?

3

I thought I’d read in one of the blog posts or the documentation somewhere that only a subset of NFS was implemented and all users have access to all files. But I can’t seem to find it. Is that not the case?

In any event, with the HA NFS export, I expect you’ll lose locality of the data because every node will always read from the VIP, and not the local disk even if the blocks reside there, right?

4

On the Linstor Gateway NFS server? I don’t use it and am unaware of that limitation.

Yes, because at the end of the day it has to go into a single filesystem stored in a (Linstor) block device. It seems a reasonable limitation for a shared home directory, IMO.

(In theory, it might be possible to use an esoteric cluster-aware filesystem which allows multiple readers and writers to the same block device, like GFS or OCFS2. Even then, I don’t know if it would work with DRBD multi-master. Don’t go there).