# High CPU Usage on drbd\_s\_writecac during benchmarking

**URL:** <https://forums.linbit.com/t/high-cpu-usage-on-drbd-s-writecac-during-benchmarking/175>\
**Category:** DRBD\
**Created:** [July 17, 2024, 11:12pm UTC](https://forums.linbit.com/t/high-cpu-usage-on-drbd-s-writecac-during-benchmarking/175 "2024-07-17T23:12:00Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheEvilApplePie](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@TheEvilApplePie](https://forums.linbit.com/u/TheEvilApplePie)\
**Post date:** [July 17, 2024, 11:12pm UTC](https://forums.linbit.com/t/high-cpu-usage-on-drbd-s-writecac-during-benchmarking/175/1 "2024-07-17T23:12:00Z")

</div>

When benchmarking to determine the viability of using drbd for a solution I’ve found that I can’t hit my raw disk performance due to the drbd\_s\_writecac process ( which appears to buffer the changes for outbound replication ) hitting the limits of a single core…

 ![drbd_s_writecac high cpu](https://canada1.discourse-cdn.com/flex003/uploads/linbit/original/1X/44cd2b1e88736e1f51e6b19f1ef8b6476851fea4.jpeg)

Benchmarking the local diskset I’m seeing ~600k IOPS and when placing behind DRBD ( with and without remote replica ) and benching I get ~190 IOPS.

Raw Disks:  
read: IOPS=631k, BW=4929MiB/s (5169MB/s)(32.0GiB/6645msec)  
write: IOPS=631k, BW=4933MiB/s (5173MB/s)(32.0GiB/6645msec); 0 zone resets

DRBD:  
read: IOPS=180k, BW=1408MiB/s (1476MB/s)(8438MiB/5994msec)  
write: IOPS=180k, BW=1406MiB/s (1475MB/s)(8429MiB/5994msec); 0 zone resets

I’ve tried every combination of settings I could imagine would have any impact including removing everything that wasn’t absolutely necessary, I’m at a loss as it seems like this may be a design issue with the cache/buffer process.

DRBD version: 9.2.10 (api:2/proto:86-122)

fio --name=fiotest --filename=/mnt/test/test --size=4Gb --rw=randrw --bs=8K --direct=1 --numjobs=32 --ioengine=libaio --iodepth=32 --group\_reporting --runtime=60

resource XXYY {

```
    volume 0 {
           device minor 1;
           disk /dev/md/XXYY;
           meta-disk internal;
    }

    on XXYY-a.local {
            node-id 0;
    }
    on XXYY-b.local {
            node-id 1;
    }

    disk {
            c-plan-ahead 0;
            al-updates no;
    }

    connection {
            path {
                    host XXYY-a.local address 123.123.123.1:7789;
                    host XXYY-b.local address 123.123.123.2:7789;
            }
            path {
                    host XXYY-a.local address 123.123.123.3:7789;
                    host XXYY-b.local address 123.123.123.4:7789;
            }
            net {
                    transport rdma;
                    protocol A;
                    max-buffers 128k;
                    sndbuf-size 2048k;
                    rcvbuf-size 2048k;
            }
    }

```

}

Any advice is welcome, thank you!

---

<div class="post-metadata">

**Author:** ![Devin](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/devin/32/26_2.png) [@Devin](https://forums.linbit.com/u/Devin)\
**Post date:** [July 19, 2024, 6:51pm UTC](https://forums.linbit.com/t/high-cpu-usage-on-drbd-s-writecac-during-benchmarking/175/2 "2024-07-19T18:51:10Z")

</div>

I believe you essentially have it figured out. You’re hitting the limitation of a single core. I tend to see IOPS max out around 350k when testing myself, but I test with 4k block size. You’re getting a little over half that with 8k blocks.

While it might not be the most graceful solution, I have seen people divide the storage into multiple DRBD resources, to be able to have multiple threads, and then aggregate these together using something like LVM.

---

<div class="post-metadata">

**Author:** ![TheEvilApplePie](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@TheEvilApplePie](https://forums.linbit.com/u/TheEvilApplePie)\
**Post date:** [July 19, 2024, 9:39pm UTC](https://forums.linbit.com/t/high-cpu-usage-on-drbd-s-writecac-during-benchmarking/175/3 "2024-07-19T21:39:36Z")

</div>

I appreciate your commentary, I was doing a last ditch effort here on the forums to see if there was something obvious I was missing. The solution you outlined I considered ( although as I’m not intimate with the innards not sure if it would experience the same issue, but figured it was worth a try anyway, so good to know it likely will give the result I want ) but was avoiding it as it adds complexity and overhead. Fixing this kind of issue usually is a heavy handed thing, so I doubt it will be resolved anytime soon, but good to know others have encountered a similar headwind.