DRBD assertion errors and LINSTOR resources stuck in DELETING on Proxmox 9 / ZFS-Thin

DRBD
1

Hello,

I am running a three-node Proxmox cluster with LINSTOR, DRBD, and ZFS-Thin storage.

All three nodes are fresh installations with identical hardware and software. All packages were updated from the currently configured repositories.

Two of the three nodes repeatedly produce DRBD assertion errors in dmesg, while the third node does not show these errors. I have not been able to find any relevant difference between the nodes.

Example errors:

[91064.078391] drbd pm-6f25d6aa/0 drbd1007: ASSERTION atomic_read(&device->local_cnt) > 0 FAILED in drbd_al_begin_io_fastpath
[91064.078477] drbd pm-6f25d6aa/0 drbd1007: ASSERTION i >= 0 FAILED in put_ldev
[91064.082281] drbd pm-6f25d6aa/0 drbd1007: ASSERTION i >= 0 FAILED in put_ldev
[91064.093978] drbd pm-437e50b0/0 drbd1008: ASSERTION i >= 0 FAILED in put_ldev
[91064.093990] drbd pm-437e50b0/0 drbd1008: ASSERTION i >= 0 FAILED in put_ldev
[91064.093995] drbd pm-437e50b0/0 drbd1008: ASSERTION i >= 0 FAILED in put_ldev
[91064.193423] drbd pm-6f25d6aa/0 drbd1007: ASSERTION i >= 0 FAILED in put_ldev
[91064.193461] drbd pm-6f25d6aa/0 drbd1007: ASSERTION i >= 0 FAILED in put_ldev
[91064.194516] drbd pm-dfb4f624/0 drbd1001: ASSERTION atomic_read(&device->local_cnt) > 0 FAILED in drbd_al_begin_io_fastpath
[91064.194567] drbd pm-dfb4f624/0 drbd1001: ASSERTION i >= 0 FAILED in put_ldev
[91064.194996] drbd pm-dfb4f624/0 drbd1001: ASSERTION i >= 0 FAILED in put_ldev
[91064.304528] drbd pm-dfb4f624/0 drbd1001: ASSERTION i >= 0 FAILED in put_ldev
[91064.304541] drbd pm-dfb4f624/0 drbd1001: ASSERTION i >= 0 FAILED in put_ldev
[91065.375341] drbd pm-437e50b0/0 drbd1008: ASSERTION atomic_read(&device->local_cnt) > 0 FAILED in drbd_al_begin_io_fastpath
[91065.375357] drbd pm-437e50b0/0 drbd1008: ASSERTION i >= 0 FAILED in put_ldev
[91065.377937] drbd pm-437e50b0/0 drbd1008: ASSERTION i >= 0 FAILED in put_ldev
[91065.787224] drbd pm-437e50b0/0 drbd1008: ASSERTION i >= 0 FAILED in put_ldev
[91065.787232] drbd pm-437e50b0/0 drbd1008: ASSERTION i >= 0 FAILED in put_ldev
[91065.787235] drbd pm-437e50b0/0 drbd1008: ASSERTION i >= 0 FAILED in put_ldev

I am also seeing resources become stuck in the DELETING state after attempting to delete them:

│ pm-f82a68ca │ px-hq-core01 │ DRBD,STORAGE │ │ Ok │ DELETING │ Yes │
│ pm-f82a68ca │ px-hq-core02 │ DRBD,STORAGE │ │ Ok │ DELETING │ Yes │
│ pm-f82a68ca │ px-hq-core03 │ DRBD,STORAGE │ │ Ok │ DELETING │ Yes │

The resource remains in this state on all three nodes and is not removed automatically.

Versions from one affected node

proxmox-ve: 9.2.0
pve-manager: 9.2.4
running kernel: 7.0.14-3-pve
zfsutils-linux: 2.4.3-pve1


Linux px-hq-core01 7.0.14-3-pve #1 SMP PREEMPT_DYNAMIC PMX 7.0.14-3 (2026-07-03T19:51Z) x86_64 GNU/Linux

DRBD kernel module:

filename: /lib/modules/7.0.14-3-pve/updates/dkms/drbd.ko
version: 9.3.3
vermagic: 7.0.14-3-pve SMP preempt mod_unload modversions

LINSTOR client:

linstor-client 1.28.1; GIT-hash: UNKNOWN

Questions

  1. What is the recommended procedure for removing resources that are stuck in the DELETING state?

  2. How can I safely clean up the resource without leaving orphaned DRBD devices, ZFS datasets, or LINSTOR database entries?

  3. What could cause these DRBD assertion failures?

  4. Are the drbd_al_begin_io_fastpath and put_ldev assertion errors related to the stuck resource deletion?

  5. Is this a known issue with DRBD 9.3.3, Proxmox kernel 7.0.14, LINSTOR, or the ZFS-Thin backend?

  6. What additional logs or command outputs should I provide to troubleshoot this?

The third node have the same hardware, installed packages, kernel, DRBD version, and LINSTOR configuration, but it does not produce the assertion errors.

Any guidance would be appreciated.

2

I usually tear down the DRBD device using drbdadm or drbdsetup, then remove the backing volume (in your case the zvol using zfs destroy), and then try deleting the resource from LINSTOR to cleanup the LINSTOR database entries and DRBD configurations.

This isn’t an issue I’m familiar with, or one that I could find others talking about, but seems like like assertion could be related to the stuck resource deletion.

I’m wondering what was happening immediately before the assertion errors cropped up, or what operations you were performing when the error was observed? Perhaps you can share some logs surrounding the assertion messages?

Can you share some details about the backing volumes and trim settings?

# ZFS backing geometry
zfs get volblocksize,volmode,refreservation <pool>/<vol>
zpool get ashift <pool>

# DRBD discard-related options in effect
drbdsetup show --show-defaults <resource> | grep -iE 'discard|al-|rs-discard'
grep -riE 'discard|al-extents|rs-discard' /var/lib/linstor.d/*.res /etc/drbd.d/ 2>/dev/null

# Is fstrim / discard active?
systemctl status fstrim.timer; cat /etc/fstab | grep -i discard
lsblk -D                # DISC-GRAN / DISC-MAX columns down the stack
3

Hi, we have also noticed this behaviour. On proxmox 9.2.5 with the latest (right now 28-07-2026) drbd and linstor packages. We also use ZFS_THIN.

The thing is, that the deletion cannot finish succesfully because ‘something’ is keeping the zvol busy:

# zfs destroy pool1/pm-15b3d58f_00000
cannot destroy 'pool1/pm-15b3d58f_00000': dataset is busy

The result is that you have to reboot the physical node (move all payload from that node and then reboot it) before you can remove the zvol.

Unfortunately that leaves the linstor resource in a ‘undefined’ state still, since the deletion process could not complete in one go.

Perhaps linstor could be adjusted so that the deletion process (in case of ZFS thin) would time out after 5 seconds and then just let the zvol exist and/or put a message somewhere that it has to be removed by hand. That way your linstor resource keeps a sane state and the only thing you are left with is a ‘unused’ zvol.

4

Probably related to ASSERTION i>= 0 FAILED in put_ldev · Issue #137 · LINBIT/drbd · GitHub.

Hopefully fixed with the next DRBD Release.

Current best fix; Reboot affected Node.

After that try to minimize parallel actions related to syncs.