DRBD failover TCP links?

General
1

Hi

There are two links for sync TCP DRBD replication.

from *.res file

connection
    {
        path
        {
            host "memverge" address ipv4 192.168.0.6:7900;
            host "memverge2" address ipv4 192.168.0.8:7900;
        }
        path
        {
            host "memverge" address ipv4 1.1.1.6:7900;
            host "memverge2" address ipv4 1.1.1.8:7900;
        }
net
    {
#       load-balance-paths      yes;
        transport tcp;
        protocol  C;
        sndbuf-size 10M;
        rcvbuf-size 10M;
        max-buffers 80K;
        max-epoch-size 20000;
        timeout 90;
        ping-timeout 10;
        ping-int 15;
        connect-int 15;
        fencing resource-and-stonith;
    }
    }

from corosync.conf

nodelist {
    node {
        name: memverge
        nodeid: 27
        ring0_addr: 192.168.0.6
        ring1_addr: 1.1.1.6
    }

    node {
        name: memverge2
        nodeid: 28
        ring0_addr: 192.168.0.8
        ring1_addr: 1.1.1.8
    }
}

but when I unplug cable from one link, DRBD replication freezes until I plug the cable back.

[ 1268.618724] mlx5_core 0000:86:00.0: Port module event: module 0, Cable unplugged
[ 1268.643924] mlx5_core 0000:86:00.0 ens5f0np0: Link down
[ 1284.707415] drbd ha-nfs memverge2: PingAck did not arrive in time.
[ 1284.707448] drbd ha-nfs: susp-io( no -> fencing )
[ 1284.707455] drbd ha-nfs memverge2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
[ 1284.707462] drbd ha-nfs/29 drbd1 memverge2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
[ 1284.707470] drbd ha-nfs/30 drbd2 memverge2: repl( SyncSource -> Off )
[ 1284.707725] drbd ha-nfs tcp:memverge2: dtt_send_page: size=4096 len=2900 sent=-4
[ 1284.707943] drbd ha-nfs memverge2: Terminating sender thread
[ 1284.708011] drbd ha-nfs memverge2: Starting sender thread (peer-node-id 28)
[ 1284.716036] drbd ha-nfs memverge2: Connection closed
[ 1284.716119] drbd ha-nfs memverge2: helper command: /sbin/drbdadm disconnected
[ 1284.716138] drbd ha-nfs memverge2: helper command: /sbin/drbdadm fence-peer
[ 1284.717060] drbd ha-nfs memverge2: helper command: /sbin/drbdadm disconnected exit code 0
[ 1284.717069] drbd ha-nfs memverge2: conn( NetworkFailure -> Unconnected ) [disconnected]
[ 1284.717075] drbd ha-nfs memverge2: Restarting receiver thread
[ 1284.717079] drbd ha-nfs memverge2: conn( Unconnected -> Connecting ) [connecting]
[ 1284.717093] drbd ha-nfs memverge2: Configured local address not found, retrying every 15 sec, err=-99
[ 1284.756086] drbd ha-nfs memverge2: helper command: /sbin/drbdadm fence-peer exit code 4 (0x400)
[ 1284.756109] drbd ha-nfs memverge2: fence-peer helper returned 4 (peer was fenced)
[ 1284.756115] drbd ha-nfs/29 drbd1 memverge2: pdsk( DUnknown -> Outdated ) [outdate-async]
[ 1284.756313] drbd ha-nfs/29 drbd1: new current UUID: C81E77D9A9B8DE3B weak: FFFFFFFFF7FFFFFF
[ 1284.756321] drbd ha-nfs: susp-io( fencing -> no ) [after-fencing]

So there is no failover to second healthy link ?

Anton

2

@Anton thanks for providing your testing. I need to refresh myself on how/when this works, and when it does not before I can thoughtfully reply.

In the meantime, I will ask that you please format your posts for better readability. Please use standard markdown for code blocks on configurations, commands, and log output. Editing posts for readability is “one more thing” I need to do before thinking about the actual problem you’re reporting.

3

Okay, I took some time to look into this again. Thanks for your patience.

If losing the link state (when ip link show shows state DOWN) causes your IP address to become unassigned, the network configuration within DRBD will become “invalid”, and DRBD won’t be able to resume replication over the “other path”.

If the IP address remains assigned regardless of the link state, as it would with a statically assigned IP address, then DRBD will be able to recover the replication over the “other path”.

For example, I have eth1 configured with a static IP address, but with the network cable “unplugged” I still have an IP address:

3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 52:54:00:6b:78:00 brd ff:ff:ff:ff:ff:ff
    altname enp0s11
    altname ens11
    inet 192.168.230.60/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe6b:7800/64 scope link 
       valid_lft forever preferred_lft forever

This works for DRBD’s load-balance-paths yes;.

However, if I allow the IP address to be configured automatically based on the link state, I see the same results as you (no “failover” or recovery of the replication link) when the link drops.

Without changing the environment, you could change the test, and instead use iptables to block and unblock one of the replication paths. This would keep the link state UP, and therefore the IP address remains configured, allowing DRBD to detect traffic is not getting through and subsequently switch to the “other path”.

I used a script similar to this to test blocking and unblocking with dynamically assigned IP addresses:

#!/bin/sh

usage() {
    echo "Usage: $0 -b|-u -i <interface> -p <port>"
    exit 1
}

INTERFACE=""
PORT=""
ACTION=""

while getopts "bu i:p:" opt; do
    case "$opt" in
        b) ACTION="block" ;;
        u) ACTION="unblock" ;;
        i) INTERFACE="$OPTARG" ;;
        p) PORT="$OPTARG" ;;
        *) usage ;;
    esac
done

if [ -z "$ACTION" ] || [ -z "$INTERFACE" ] || [ -z "$PORT" ]; then
    usage
fi

case "$ACTION" in
    "block")
        iptables -A OUTPUT -o "$INTERFACE" -p tcp --dport "$PORT" -j DROP
        iptables -A INPUT -i "$INTERFACE" -p tcp --dport "$PORT" -j DROP
        echo "Port $PORT blocked on interface $INTERFACE."
        ;;
    "unblock")
        iptables -D OUTPUT -o "$INTERFACE" -p tcp --dport "$PORT" -j DROP
        iptables -D INPUT -i "$INTERFACE" -p tcp --dport "$PORT" -j DROP
        echo "Port $PORT unblocked on interface $INTERFACE."
        ;;
    *)
        usage
        ;;
esac

Using it like this while writing to the DRBD device:

root@linbit-1:~# ./path_block.sh -b -i eth1 -p 7900
Port 7900 blocked on interface eth1.
root@linbit-1:~# ./path_block.sh -u -i eth1 -p 7900
Port 7900 unblocked on interface eth1.
2 Likes
4

So the root case is a very important nuance of the network configuration.

I made changes and failover works!

It may not be a bad idea to mention this in the DRBD9 user’s guide.

Thank you Matt, you are the best!! :+1:

Anton

3 Likes