# DRBD failover TCP links?

**URL:** <https://forums.linbit.com/t/drbd-failover-tcp-links/625>\
**Category:** General\
**Tags:** drbd\
**Created:** [February 15, 2025, 8:03am UTC](https://forums.linbit.com/t/drbd-failover-tcp-links/625 "2025-02-15T08:03:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anton](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Anton](https://forums.linbit.com/u/Anton)\
**Post date:** [February 15, 2025, 8:03am UTC](https://forums.linbit.com/t/drbd-failover-tcp-links/625/1 "2025-02-15T08:03:19Z")

</div>

Hi

There are two links for sync TCP DRBD replication.

from \*.res file

```txt
connection
    {
        path
        {
            host "memverge" address ipv4 192.168.0.6:7900;
            host "memverge2" address ipv4 192.168.0.8:7900;
        }
        path
        {
            host "memverge" address ipv4 1.1.1.6:7900;
            host "memverge2" address ipv4 1.1.1.8:7900;
        }
net
    {
# load-balance-paths yes;
        transport tcp;
        protocol C;
        sndbuf-size 10M;
        rcvbuf-size 10M;
        max-buffers 80K;
        max-epoch-size 20000;
        timeout 90;
        ping-timeout 10;
        ping-int 15;
        connect-int 15;
        fencing resource-and-stonith;
    }
    }

```

from corosync.conf

```txt
nodelist {
    node {
        name: memverge
        nodeid: 27
        ring0_addr: 192.168.0.6
        ring1_addr: 1.1.1.6
    }

    node {
        name: memverge2
        nodeid: 28
        ring0_addr: 192.168.0.8
        ring1_addr: 1.1.1.8
    }
}

```

but when I unplug cable from one link, DRBD replication freezes until I plug the cable back.

```txt
[1268.618724] mlx5_core 0000:86:00.0: Port module event: module 0, Cable unplugged
[1268.643924] mlx5_core 0000:86:00.0 ens5f0np0: Link down
[1284.707415] drbd ha-nfs memverge2: PingAck did not arrive in time.
[1284.707448] drbd ha-nfs: susp-io( no -> fencing )
[1284.707455] drbd ha-nfs memverge2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
[1284.707462] drbd ha-nfs/29 drbd1 memverge2: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
[1284.707470] drbd ha-nfs/30 drbd2 memverge2: repl( SyncSource -> Off )
[1284.707725] drbd ha-nfs tcp:memverge2: dtt_send_page: size=4096 len=2900 sent=-4
[1284.707943] drbd ha-nfs memverge2: Terminating sender thread
[1284.708011] drbd ha-nfs memverge2: Starting sender thread (peer-node-id 28)
[1284.716036] drbd ha-nfs memverge2: Connection closed
[1284.716119] drbd ha-nfs memverge2: helper command: /sbin/drbdadm disconnected
[1284.716138] drbd ha-nfs memverge2: helper command: /sbin/drbdadm fence-peer
[1284.717060] drbd ha-nfs memverge2: helper command: /sbin/drbdadm disconnected exit code 0
[1284.717069] drbd ha-nfs memverge2: conn( NetworkFailure -> Unconnected ) [disconnected]
[1284.717075] drbd ha-nfs memverge2: Restarting receiver thread
[1284.717079] drbd ha-nfs memverge2: conn( Unconnected -> Connecting ) [connecting]
[1284.717093] drbd ha-nfs memverge2: Configured local address not found, retrying every 15 sec, err=-99
[1284.756086] drbd ha-nfs memverge2: helper command: /sbin/drbdadm fence-peer exit code 4 (0x400)
[1284.756109] drbd ha-nfs memverge2: fence-peer helper returned 4 (peer was fenced)
[1284.756115] drbd ha-nfs/29 drbd1 memverge2: pdsk( DUnknown -> Outdated ) [outdate-async]
[1284.756313] drbd ha-nfs/29 drbd1: new current UUID: C81E77D9A9B8DE3B weak: FFFFFFFFF7FFFFFF
[1284.756321] drbd ha-nfs: susp-io( fencing -> no ) [after-fencing]

```

So there is no failover to second healthy link ?

Anton

---

<div class="post-metadata">

**Author:** ![kermat](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/kermat/32/21_2.png) [@kermat](https://forums.linbit.com/u/kermat)\
**Post date:** [February 24, 2025, 2:53pm UTC](https://forums.linbit.com/t/drbd-failover-tcp-links/625/2 "2025-02-24T14:53:24Z")

</div>

@Anton thanks for providing your testing. I need to refresh myself on how/when this works, and when it does not before I can thoughtfully reply.

In the meantime, I will ask that you please format your posts for better readability. Please use standard markdown for code blocks on configurations, commands, and log output. Editing posts for readability is “one more thing” I need to do before thinking about the actual problem you’re reporting.

---

<div class="post-metadata">

**Author:** ![kermat](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/kermat/32/21_2.png) [@kermat](https://forums.linbit.com/u/kermat)\
**Post date:** [March 6, 2025, 9:32pm UTC](https://forums.linbit.com/t/drbd-failover-tcp-links/625/3 "2025-03-06T21:32:17Z")

</div>

Okay, I took some time to look into this again. Thanks for your patience.

If losing the link state (when `ip link show` shows `state DOWN`) causes your IP address to become unassigned, the network configuration within DRBD will become “invalid”, and DRBD won’t be able to resume replication over the “other path”.

If the IP address remains assigned regardless of the link state, as it would with a statically assigned IP address, then DRBD will be able to recover the replication over the “other path”.

For example, I have `eth1` configured with a static IP address, but with the network cable “unplugged” I still have an IP address:

```txt
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 52:54:00:6b:78:00 brd ff:ff:ff:ff:ff:ff
    altname enp0s11
    altname ens11
    inet 192.168.230.60/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe6b:7800/64 scope link 
       valid_lft forever preferred_lft forever

```

This works for DRBD’s `load-balance-paths yes;`.

However, if I allow the IP address to be configured automatically based on the link state, I see the same results as you (no “failover” or recovery of the replication link) when the link drops.

Without changing the environment, you could change the test, and instead use `iptables` to block and unblock one of the replication paths. This would keep the link `state UP`, and therefore the IP address remains configured, allowing DRBD to detect traffic is _not_ getting through and subsequently switch to the “other path”.

I used a script similar to this to test blocking and unblocking with dynamically assigned IP addresses:

```sh
#!/bin/sh

usage() {
    echo "Usage: $0 -b|-u -i <interface> -p <port>"
    exit 1
}

INTERFACE=""
PORT=""
ACTION=""

while getopts "bu i:p:" opt; do
    case "$opt" in
        b) ACTION="block" ;;
        u) ACTION="unblock" ;;
        i) INTERFACE="$OPTARG" ;;
        p) PORT="$OPTARG" ;;
        *) usage ;;
    esac
done

if [-z "$ACTION"] || [-z "$INTERFACE"] || [-z "$PORT"]; then
    usage
fi

case "$ACTION" in
    "block")
        iptables -A OUTPUT -o "$INTERFACE" -p tcp --dport "$PORT" -j DROP
        iptables -A INPUT -i "$INTERFACE" -p tcp --dport "$PORT" -j DROP
        echo "Port $PORT blocked on interface $INTERFACE."
        ;;
    "unblock")
        iptables -D OUTPUT -o "$INTERFACE" -p tcp --dport "$PORT" -j DROP
        iptables -D INPUT -i "$INTERFACE" -p tcp --dport "$PORT" -j DROP
        echo "Port $PORT unblocked on interface $INTERFACE."
        ;;
    *)
        usage
        ;;
esac

```

Using it like this while writing to the DRBD device:

```txt
root@linbit-1:~# ./path_block.sh -b -i eth1 -p 7900
Port 7900 blocked on interface eth1.
root@linbit-1:~# ./path_block.sh -u -i eth1 -p 7900
Port 7900 unblocked on interface eth1.

```

---

<div class="post-metadata">

**Author:** ![Anton](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Anton](https://forums.linbit.com/u/Anton)\
**Post date:** [March 7, 2025, 11:00am UTC](https://forums.linbit.com/t/drbd-failover-tcp-links/625/4 "2025-03-07T11:00:40Z")

</div>

So the root case is a very important nuance of the network configuration.

I made changes and failover works!

It may not be a bad idea to mention this in the DRBD9 user’s guide.

Thank you Matt, you are the best!! 👍

Anton