Resource connection path for ring topology

LINSTOR
1

I’m beginning to experiment with linstore on a 3-node proxmox cluster. So far this is looking really promising.

Here’s my context and questions. Each node has 3 nics: a management/master nic, and two other nics that are direct-connect to other nodes.

I want to configure the data replication to use the direct-connected nics and fall back to the mgmt nic if the link is down. Some questions or issues have arisen that I would like advice on.

To elaborate, the data replication would use these links if they are up and only fall back to using the mgmt nic if a link is down. The normal drbd repl traffic would use the direct connects like:

A → B
A → C
B → C

1: I cannot figure out how to create resource-connection paths for a resource group. I have done it for the individual resource “linstor_db” I created from the HA docs. The problem is that I have a resource group for replicated storage created when I create a VM. I had rather not have to do that every time I create a new VM, but had rather instead do it at the resource group level. Any ideas or suggestions?

2: Given my desired connection topology, what would be the right setting for the PrefNic property since that seems to apply to single-nic connected systems instead of direct connected ring topology?

As a side note, I’m running OSPF and BFD routing protocols on these nodes so they detect and reroute link failures very quickly. I’m not sure how that would interplay with linstor/drbd’s own failure handling logic.

Thanks for any and all advice!

2

You can only create a resource-connection path on a specific resource. However, you can use node-connection path to configure multiple paths between all cluster resources in a cluster.

For example, a test cluster I currently have up has nodes with the following names and IP addresses:

  • swarm-0
    • eth0: 192.168.121.120
    • eth1: 192.168.222.50
  • swarm-1
    • eth0: 192.168.121.209
    • eth1: 192.168.222.51
  • swarm-2
    • eth0: 192.168.121.51
    • eth1: 192.168.222.52

The following commands are used to create multiple paths between all resource in the cluster:

# create interfaces on all nodes
linstor node interface create swarm-0 eth0 192.168.121.120
linstor node interface create swarm-0 eth1 192.168.222.50
linstor node interface create swarm-1 eth0 192.168.121.209
linstor node interface create swarm-1 eth1 192.168.222.51
linstor node interface create swarm-2 eth0 192.168.121.51
linstor node interface create swarm-2 eth1 192.168.222.52

# create all paths that use eth0
linstor node-connection path create swarm-0 swarm-1 repl0 eth0 eth0
linstor node-connection path create swarm-0 swarm-2 repl0 eth0 eth0
linstor node-connection path create swarm-1 swarm-2 repl0 eth0 eth0

# create all paths that use eth1
linstor node-connection path create swarm-0 swarm-1 repl1 eth1 eth1
linstor node-connection path create swarm-0 swarm-2 repl1 eth1 eth1
linstor node-connection path create swarm-1 swarm-2 repl1 eth1 eth1

Does that answer your questions?

Also, the path-mesh feature should be coming in the next LINSTOR release:

1 Like
3

Hi kermat,

Thanks for the detailed answer! I think that might do mostly what I’m trying to achieve. However, I can’t see how I could configure it to prefer the direct paths as opposed to the “backup” path over the switched network.

I have three nics. From your example, lets say eth0 and eth1 are specified exactly as you show and those are direct cable links between the nodes (no switch etc. in the middle).

Now I have a third interface which is the broader network-attached interface eth2 that is available on the general network. All three nodes are available via this interface as well. I could add eth3 as well:

linstore node-connection path create swarm-0 swarm-1 eth2 eth2
linstore node-connection path create swarm-0 swarm-2 eth2 eth2
linstore node-connection path create swarm-1 swarm-1 eth2 eth2

What I would like to do is have the data replication only take the eth2 path if their direct link is down. I’ll watch that path-mesh feature to see what they say.

I had another idea I may experiment with given that I’m running OSPF and BFD routing protocols on these nodes. I might try creating the node IP on the “lo” interface instead of directly on the network adapters and let the OSPF and BFD link failure detection figure this out. That may be too much voodoo but I may play with it :slight_smile:

Again, thanks for your detailed reply.

Regards

4

Hi kermat, So how is the physical connectivity…Can you provide a diagram of actual physical connectivity… I was planning to use a 10Gig switch which seems to be very expensive…I think with your concept I can avoid the switch to reduce the cost…

Thanks…

5

I am sorry that this post of 7 months old,if someone can address the query would be really helpful…

Thanks in Advance…

6

Sorry, been pretty busy… looking back on this I realize for direct connections without a switch you’d need something more like this:

  • swarm-0
    • eth0: 192.168.201.50/24
    • eth1: 192.168.202.150/24
  • swarm-1
    • eth0: 192.168.201.51/24
    • eth1: 192.168.212.151/24
  • swarm-2
    • eth0: 192.168.212.52/24
    • eth1: 192.168.202.152/24

I lazily and wastefully used 3x /24 subnets, but optimally these would be 3x /30 subnets.

linstor node interface create swarm-0 eth0 192.168.201.50
linstor node interface create swarm-0 eth1 192.168.202.150
linstor node interface create swarm-1 eth0 192.168.201.51
linstor node interface create swarm-1 eth1 192.168.212.151
linstor node interface create swarm-2 eth0 192.168.212.52
linstor node interface create swarm-2 eth1 192.168.202.152

# create all paths
linstor node-connection path create swarm-0 swarm-1 repl_0_1 eth0 eth0
linstor node-connection path create swarm-0 swarm-2 repl_0_2 eth1 eth1
linstor node-connection path create swarm-1 swarm-2 repl_1_2 eth1 eth0

Physical connections would look something like this:

image
image1808×992 10.5 KB

You could use whatever management network you have (not pictured here) when adding nodes to the LINSTOR cluster. LINSTOR would then use the management network as its management/control network.

7

Hi Kermat, Thankyou and appreciate for your prompt reply. I am sorry that was old post…

I have couple of more questions,

  1. In this scenario the redundancy is builtin and that the OS level nic bonding may not be required…?
  2. have you implemented this architecture in a production environment, or anyone else have done whom you might know…just curious to know…
  3. what would be the architecture in a 2 node cluster with the 3rd node being qdevice/diskless…

Thanks…

8

In this scenario there would be no redundancy on the replication links. You would need to configure multiple paths between node pairs to achieve redundancy.

I have not implemented exactly this for someone in production but I wouldn’t hesitate to if the need was there. These network addresses end up in the DRBD resource configurations as paths, and we’ve configured back-to-back connections for DRBD peers countless times for 2-nodes. This is just adding more of those same types of connections.

The same could be used. The third node just won’t have a diskful replica.

9

Hi Kermat, Thanks for you reply, was really helpful, sorry for the delayed response from my end…So if redundancy is not builtin, I have to provide 2 more additional ports for each existing ports(eth0 and eth1) for redundancy for each host ? which means 4 ports on each host and plus 2 ports for Mgmt/controller ?

any idea where I can look for the documentation…

Thanks in advance…

10

Yes, with direct connections, connection meshes become “port heavy” :sweat_smile:

The relevant documentation for what we’re talking about can be found here:

1 Like