On a two node DRBD cluster we are experiencing the following error messages:
[11560350.682797] block drbd0: drbd_md_sync_page_io(,3742095912s,WRITE) failed with error -5
[11560350.682808] block drbd0: disk( UpToDate -> Failed )
[11560350.682819] block drbd0: Local IO failed in __al_write_transaction. Detaching...
[11560350.682832] block drbd0: local WRITE IO error sector 218585232+32 on md0p2
[11560350.682842] block drbd0: local WRITE IO error sector 218586616+8 on md0p2
[11560350.682849] block drbd0: local WRITE IO error sector 218588448+16 on md0p2
[11560350.682855] block drbd0: local WRITE IO error sector 218589224+32 on md0p2
[11560350.682864] block drbd0: local WRITE IO error sector 218589848+8 on md0p2
[11560350.682901] block drbd0: disk is Failed, cannot start al transaction
[11560350.683632] block drbd0: 0 KB (0 bits) marked out-of-sync by on disk bit-map.
[11560350.683641] block drbd0: disk( Failed -> Diskless )
[11560350.683810] block drbd0: receiver updated UUIDs to effective data uuid: 78189A5EBB2C6954
OS: Debian 12
DRBD Kernel Module: 8.4.11
DRBD Admin Tools: 9.22.0
The hardware vendor tells us, that the physical disks are healthy as they can be.
There were no obvious changes to the system, other than OS updates.
Does anyone have an idea where to look to isolate this problem? Right now, I am pretty much clueless on where to even start. Thanks in advance!