# Quick DR & HA through new x-replicas-on-different feature

**URL:** <https://forums.linbit.com/t/quick-dr-ha-through-new-x-replicas-on-different-feature/363>\
**Category:** LINSTOR\
**Tags:** kubernetes, quorum, drbd\
**Created:** [October 13, 2024, 5:04pm UTC](https://forums.linbit.com/t/quick-dr-ha-through-new-x-replicas-on-different-feature/363 "2024-10-13T17:04:12Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![boedy](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/boedy/32/124_2.png) [@boedy](https://forums.linbit.com/u/boedy)\
**Post date:** [February 21, 2025, 11:00pm UTC](https://forums.linbit.com/t/quick-dr-ha-through-new-x-replicas-on-different-feature/363/8 "2025-02-21T23:00:29Z")

</div>

Now that the [xReplicasOnDifferent](https://forums.linbit.com/t/linstor-operator-v2-8-0/630) feature is out in the latest operator release, I figured it was time to revisit this multi-datacenter DRBD setup and share my findings. I hope others experimenting with this feature can also find these insights useful!

## Test Environment

- **Cluster Topology:** A “stretched” cluster across two main datacenters (DC A & DC B) plus a third “tiebreaker” datacenter (DC C).

- **Hardware:** Five CCX33 Hetzner cloud instances.

- **Storage Backend:** LVM Thin

- **Workload:**

### Benchmark command

```bash
# paramerters
sysbench \
 --table-size=100000 \
 --tables=20 \
 --threads=64 \
 --events=100000 \
 --time=5000 \
 oltp_write_only run

```

Time to start testing! 🧪 🥼

## First Test: Baseline (2 Replicas, Same DC)

**Goal:** Get a baseline measurement by having only one extra replica in the same datacenter.

**StorageClass:**

```auto
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: lvm-thin
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  linstor.csi.linbit.com/storagePool: vg1-thin #LVM Thin
  linstor.csi.linbit.com/placementCount: "2"
  linstor.csi.linbit.com/resourceGroup: "lvm-thin"
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "false"

```

**Results:**

```txt
General statistics:
    total time: 19.8495s
    total number of events: 100000

Latency (ms):
         min: 1.25
         avg: 12.69
         max: 851.76
         95th percentile: 33.72
         sum: 1268680.55

```

**Performance:** Around `19.85s` total time. This serves as the baseline for comparison. I ran the benchmark multiple times, but results were consistent with little variance.

## Second Test: 4 Replicas (2 in Each DC) - Protocol C (Sync)

**Goal:** Measure the impact of fully synchronous replication (Protocol C) across two datacenters. This setup also includes a diskless tiebreaker node in the third DC.

**StorageClass:**

```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: high-available
provisioner: linstor.csi.linbit.com
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  linstor.csi.linbit.com/storagePool: vg1-thin #LVM Thin
  linstor.csi.linbit.com/placementCount: "4"
  linstor.csi.linbit.com/resourceGroup: "high-available"
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "false"
  linstor.csi.linbit.com/xReplicasOnDifferent: |
    topology.kubernetes.io/zone: 2

```

**Results:**

```plaintext
General statistics:
    total time: 31.4133s
    total number of events: 100000

Latency (ms):
         min: 4.38
         avg: 20.08
         max: 1227.20
         95th percentile: 50.11
         sum: 2008262.40

```

**Performance:** About **31.41s** total time, roughly 58% slower than the baseline.

## Third Test: 4 Replicas (2 in Each DC) - Protocol A (Async)

**Goal:** Same setup a previous test, but this time compare performance using asynchronous replication between the two primary datacenters. I applied the following `NodeConnection` to achieve this:

```yaml
apiVersion: piraeus.io/v1
kind: LinstorNodeConnection
metadata:
  name: selector
spec:
  selector:
    - matchLabels:
        - key: topology.kubernetes.io/region
          op: NotSame
  properties:
    - name: DrbdOptions/Net/protocol
      value: A

```

**Results:**

```txt
General statistics:
    total time: 27.3795s
    total number of events: 100000

Latency (ms):
         min: 1.29
         avg: 17.50
         max: 1155.95
         95th percentile: 49.21
         sum: 1750079.59

```

**Performance:** ~27.38s total time—around 38% slower than the baseline.

Tweaking DRBD options like `--al-extents=65534` `--sndbuf-size=0` didn’t significantly shift the results.

# Thoughts & Questions

I was hoping to see performance closer to the baseline when using asynchronous replication between the DC’s, but there’s still a noticeable slowdown. I guess some overhead is expected when adding more replicas, but I’m curious if there are any tuning parameters or best practices that could help close this gap further.

**DRBD Proxy?**  
Could [DRBD Proxy](https://linbit.com/drbd-proxy), bring performance closer to the baseline, or is some performance hit inevitable as soon as multiple replicas (even asynchronous ones) are involved?

**Methodology & Setup**  
I tried to keep each test consistent; same Sysbench parameters, same node scheduling, etc. But if you see anything that should be adjusted, let me know. If there’s any additional data or context you’d like me to provide, feel free to ask!

**Final Thoughts**  
In the end, I’m feeling a tiny bit disappointed. Though the performance drop isn’t catastrophic, I was hoping to have all my cake and eat it too! 🎂

@kermat, @ghernadi, I’d love to hear your insights on whether I might be missing any optimizations or if this is just the expected trade-off. Also, any guidance on diagnosing where the real bottleneck lies would be super helpful!

Either way, this was a lot of fun to test, and I’m looking forward to hearing your thoughts!

---

_[View the full topic](https://forums.linbit.com/t/quick-dr-ha-through-new-x-replicas-on-different-feature/363)._