During heavy IO disk write timeout leads to DRBD going to diskless mode

DRBD
1

Hi,
During heavy disk IO one of our raid sets experiences a write timeout (DID_TIME_OUT as reported by the kernel). This happens about once a week on our secondary node.

Next message in our logs is from kernel blk_update_request which reports this as an I/O error. This leads to DRBD saying “we had at least one MD IO ERROR during bitmap IO”. And then failing the disk and going into Diskless mode.
If there was a real IO error this is exactly what we want to happen. But this is a timeout, which should be handled differently.

Why does DRBD handle this as an IO error and not a disk timeout? How can we change this behaviour?

DRBD version 9.15

To recover from this we simply run: drbdadm attach
And DBRD resyncs for a bit and everything works again for a week or so.

Best regards,
Rami Lehti

2

Would you mind sharing your drbd resource config? It could be a matter of configuring the ko-count:

ko-count number

           If a secondary node fails to complete a write request in ko-count times the timeout
           parameter, it is excluded from the cluster. The primary node then sets the connection
           to this secondary node to Standalone. To disable this feature, you should explicitly
           set it to 0; defaults may change between versions.
3

resource kvmpool2 {
meta-disk internal;
device /dev/drbd2;
syncer {
verify-alg sha1;
c-max-rate 1024000;
c-min-rate 51200;
c-plan-ahead 10;
}
net {
}
on host1 {
disk /dev/disk/by-partuuid/13902e29-7e15-4465-9651-1abbd2ac341a;
address 192.168.120.2:7790;
}
on host2 {
disk /dev/disk/by-partuuid/7e9ce22d-29de-4fbc-9677-49af3dd18c3e;
address 192.168.120.3:7790;
}
}

4

The default for ko-count is 7; you could try increasing that and see if the issue still happens.

5

Thanks, I’ll try that.

6

One more thing, as mentioned in the man page for drbd.conf, the default might change between versions. But you can easily see it with the command drbdsetup show --show-defaults this will also show you all kinds of other fun things you can tune.