Sluggishness in DRBD resync in Rocky Linux 9.8 (worked on Rocky Linux 9.7)

General
1

We are using DRBD to setup a high-availability environment.

  • Kernel: Rocky Linux 9.8 → 5.14.0-687.10.1.el9_8.0.1.x86_64
  • DRBD version: kmod-drbd9x-9.3.2-2.el9_8& drbd9x-utils-9.34.0-1.el9 in Rocky 9.8.
  • Symptom: untuned dynamic resync is very slow around 250k c-min-rate floor (~0.30 MB/s)

drbdsetup status --verbose --statistics r0

r0 node-id:1 role:Primary suspended:no force-io-failures:no

*write-ordering:flush*

volume:0 minor:0 disk:UpToDate backing_dev:/dev/vgpcp/lvbprhome quorum:yes

  *open:yes*

  *size:20970844 read:11558042 written:133969 al-writes:0 bm-writes:7125 upper-pending:0 lower-pending:1*

  *al-suspended:no blocked:no*

rocky98-xxx node-id:0 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0

  *rs-in-flight:0*

*volume:0 replication:SyncSource peer-disk:Inconsistent done:55.42 resync-suspended:no*

    *received:425 sent:11557952 out-of-sync:9349224 pending:0 unacked:1 **dbdt1:0.30** eta:30778*

Earlier when we tested with Rocky Linux 9.7, we got the desired results; the untuned dynamic resync ramped to ~27 MB/s.

DRBD version used in Rocky Linux 9.7 - kmod-drbd9x-9.3.0-1.el9_7 & drbd9x-utils-9.33.0-1.el9.

drbdsetup status --verbose --statistics r0

r0 node-id:1 role:Primary suspended:no force-io-failures:no

write-ordering:flus h

volume:0 minor:0 disk:UpToDate backing_dev:/dev/vgpcp/lvbprhome quorum:y es

open: yes

size:20970844 read:6972490 written:133969 al-writes:5 bm-writes:341 upper-pending:0 lower-pendi ng:1

al-suspended:no block ed:no

rocky97-xxx node-id:0 connection:Connected role:Secondary tls:no congested:no ap-in-fl ight:0

rs-in-flig ht:6144

volume:0 replication:SyncSource peer-disk:Inconsistent done:33.54 resync-susp ended:no

received:425 sent:6972400 out-of-sync:13937848 pending:3 unacked:1 dbdt1:27.5 1 eta:495

In both the Rocky Linux 9.8 and 9.7, we are having identical config & defaults. Same backing device, mq-deadline scheduler on both, vmxnet3 10G identical, iperf3 ~32 Gbit/s with ~0 retransmits.

disk {

resync-rate 250k; # bytes/second, default

c-plan-ahead 20; # 1/10 seconds, default

c-delay-target 10; # 1/10 seconds, default

c-fill-target 100s; # bytes, default

c-max-rate 102400k; # bytes/ second, default

c-min-rate 250k; # bytes /second, default

bitm ap yes; # default

resync-without-replication yes; # default

}

Why are we seeing very slow resync in Rocky 9.8?

2

My guess is likely just the tuning. The defaults are a tad conservative because that is what is safe in most environments. Try tuning things a bit.

I have a whole KB article on this exact topic here: Tuning the DRBD Resync Controller | Knowledge Base

3

Hi Devin,

Thank you for the response and the tuning suggestions.

Before applying them, I’d like to understand the root cause. These defaults have worked well across Linux 6.x, 7.x, 8.x, and up to 9.7 — ramping to ~27 MB/s on the same hardware. Only after the upgrade to Rocky 9.8 — with identical config and identical DRBD defaults — does the controller now stay pinned around ~0.30 MB/s.

The fact that 9.8 requires special tuning is bothering and hence this issue is reported to understand what changed so drastically in 9.8 to cause this.

4

Hi Devin,

Any insights on this?

Thanks again!