# RDMA over IB is twice as slow as TCP with IP over IB **URL:** \ **Category:** DRBD\ **Tags:** drbd\ **Created:** [February 13, 2026, 2:21pm UTC](https://forums.linbit.com/t/rdma-over-ib-is-twice-as-slow-as-tcp-with-ip-over-ib/1152 "2026-02-13T14:21:34Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![abegorov](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@abegorov](https://forums.linbit.com/u/abegorov)\ **Post date:** [February 13, 2026, 2:21pm UTC](https://forums.linbit.com/t/rdma-over-ib-is-twice-as-slow-as-tcp-with-ip-over-ib/1152/1 "2026-02-13T14:21:34Z")
**DRBD 9.3.0** The RDMA transport driver is very slow on modern hardware like NVIDIA ConnectX-7. It struggles to utilize the adapter’s (400 Gb/s) bandwidth. It appears that `sndbuf-size` and `rcvbuf-size` maximum values are too small. It is impossible to specify buffer sizes larger than 25M (`sndbuf-size + rcvbuf-size + rdma-ctrl-sndbuf-size + rdma-ctrl-rcvbuf-size = 52424K max`), and even with 25M buffers, I see thousands of `Not sending flow_control msg, no receive window` messages during heavy load. Additionally, there are many messages about switching to WQ\_UNBOUND in dmesg (for both RDMA and TCP transports): ```auto workqueue: iomap_dio_complete_work hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND ``` Even when switching the transport from RDMA to TCP (using IP over Infiniband), I observe two times the performance improvement. I have configured 24 DRBD resources. For each DRBD instance, I allocated a dedicated CPU core (reserved only for DRBD): ```auto resource "disk0" { device minor 0; disk "/dev/disk/by-partlabel/drbd0"; meta-disk internal; options { cpu-mask "00000000,00000000,00000001"; } on "hosta" { node-id 0; } on "hostb" { node-id 1; } connection { host "hosta" address 192.168.1.1:7789; host "hostb" address 192.168.1.2:7789; } } ... resource "disk23" { device minor 23; disk "/dev/disk/by-partlabel/drbd23"; meta-disk internal; options { cpu-mask "00000000,00000008,00000000"; } on "hosta" { node-id 0; } on "hostb" { node-id 1; } connection { host "hosta" address 192.168.1.1:7812; host "hostb" address 192.168.1.2:7812; } } ``` For my write performance test, I used this FIO script: ```auto [seq-write-24] filename=/disks/0/fio:/disks/1/fio:/disks/2/fio:/disks/3/fio:/disks/4/fio:/disks/5/fio:/disks/6/fio:/disks/7/fio:/disks/8/fio:/disks/9/fio:/disks/10/fio:/disks/11/fio:/disks/12/fio:/disks/13/fio:/disks/14/fio:/disks/15/fio:/disks/16/fio:/disks/17/fio:/disks/18/fio:/disks/19/fio:/disks/20/fio:/disks/21/fio:/disks/22/fio:/disks/23/fio file_service_type=roundrobin:1 size=2400G rw=write bs=1M direct=1 buffered=0 numjobs=24 time_based=1 runtime=60 ioengine=libaio iodepth=24 group_reporting ``` **With RDMA and this configuration, I got 12.2 GB/s overall:** ```auto net { protocol C; transport "rdma"; sndbuf-size 25M; rcvbuf-size 25M; rdma-ctrl-sndbuf-size 600K; rdma-ctrl-rcvbuf-size 600K; max-buffers 128K; max-epoch-size 20000; } ``` **With TCP and this configuration, I got 24.3 GB/s overall:** ```auto net { protocol C; transport "tcp"; sndbuf-size 128M; rcvbuf-size 128M; max-buffers 128K; max-epoch-size 20000; } ``` **Without replication (DRBD down on hostb), I achieved 129 GB/s** ---
**Author:** ![kermat](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/kermat/32/21_2.png) [@kermat](https://forums.linbit.com/u/kermat)\ **Post date:** [February 23, 2026, 3:41pm UTC](https://forums.linbit.com/t/rdma-over-ib-is-twice-as-slow-as-tcp-with-ip-over-ib/1152/2 "2026-02-23T15:41:27Z")
Thank you for sharing these tests and results. I have ConnectX-5 cards available for my own testing and typically see around ~15% _better_ performance when using RDMA rather than TCP (IPoIB), but I usually test random writes with small block sizes. When testing with sequential writes and large block sizes, I am also seeing that RDMA is ~3% slower than TCP. So not as drastic as your results, but still counter intuitive. I will ask the LINBIT kernel module developers for input and report back with any insight. ---
**Author:** ![abegorov](https://avatars.discourse-cdn.com/v4/letter/a/eb9ed0/32.png) [@abegorov](https://forums.linbit.com/u/abegorov)\ **Post date:** [February 24, 2026, 8:09am UTC](https://forums.linbit.com/t/rdma-over-ib-is-twice-as-slow-as-tcp-with-ip-over-ib/1152/3 "2026-02-24T08:09:49Z")
Thank you for your feedback and for investigating further. I believe the core issue comes down to buffer limitations. Sending 25 MB over a 400 Gb/s link takes approximately half a millisecond, which means a single RDMA connection cannot keep up with the throughput. The real bottleneck appears to be the maximum number of Work Requests per RDMA connection, which constrains the total buffer size. With a Scatter-Gather Element of 8, the limit is 13,106 WRs. Assuming 4 KB buffers, the effective buffer capacity (send + receive) reaches only about 51 MB. I believe the problem could potentially be addressed in two ways: 1. Use multiple RDMA connections - NVMe over Fabrics (RDMA) uses one connection per CPU, which might distribute the load more effectively 2. Increase buffer sizes per Work Request - using buffers larger than 4 KB could maximize the WR capacity. ---
**Author:** ![kermat](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/kermat/32/21_2.png) [@kermat](https://forums.linbit.com/u/kermat)\ **Post date:** [March 20, 2026, 9:24pm UTC](https://forums.linbit.com/t/rdma-over-ib-is-twice-as-slow-as-tcp-with-ip-over-ib/1152/4 "2026-03-20T21:24:50Z")
Sorry for taking so long to reply. Life gets busy and I forget to check in on the forums, and then here we are 25 days later 😅 I did want to let you know that I did speak with our kernel module developer, and learned that we’re aware of some shortcomings in the RDMA transport, specifically with replicating larger writes, that will require a significant effort to address. Basically, when the RDMA module was initially created 10+ years ago, it seemed like a good idea to chunk up data into 4K pages before transmitting. What we know now is that ring buffers are a better approach, since they can transfer up to 1MiB in a single operation. I believe this is along the same lines as your 2nd suggestion, of increasing the buffer sizes per request, so I do think you and our kernel module developers are at least on the same page there.