# Recover from abrupt server shutdown

**URL:** <https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173>\
**Category:** General\
**Created:** [July 17, 2024, 5:59pm UTC](https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173 "2024-07-17T17:59:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![dshiklov](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dshiklov](https://forums.linbit.com/u/dshiklov)\
**Post date:** [July 17, 2024, 5:59pm UTC](https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173/1 "2024-07-17T17:59:49Z")

</div>

After abrupt server shutdown drbd shows no resources and linstor sees no physical-storage. Is there a way to recover from this state?

---

<div class="post-metadata">

**Author:** ![Devin](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/devin/32/26_2.png) [@Devin](https://forums.linbit.com/u/Devin)\
**Post date:** [July 19, 2024, 6:56pm UTC](https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173/2 "2024-07-19T18:56:29Z")

</div>

It’s tough to say without any additional information. Anything in the logs?

If LINSTOR doesn’t see the physical storage, my first guess would there is likely something wrong with the physical storage.

---

<div class="post-metadata">

**Author:** ![dshiklov](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dshiklov](https://forums.linbit.com/u/dshiklov)\
**Post date:** [July 22, 2024, 5:27pm UTC](https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173/3 "2024-07-22T17:27:29Z")

</div>

Thanks for your reply Devin,

Just for context I’ve seen this happening on two separate occasions. Once on a system without UPS during power outage. Second time on a system being shutdown ungracefully from a lights out manager.

Before the abrupt shutdown the lsblk showed something similar to this output:

```auto
  ├─ubuntu--vg-drbdlv01_tmeta 253:8 0 76M 0 lvm  
  │ └─ubuntu--vg-drbdlv01-tpool 253:10 0 300G 0 lvm  
  │ ├─ubuntu--vg-drbdlv01 253:11 0 300G 1 lvm  
  │ ├─ubuntu--vg-pvc--c2f11b76--c176--4ec9--b0b7--acd78073067d_00000
  │ │ 253:12 0 8G 0 lvm  
  │ │ └─drbd1000 147:1000 0 8G 0 disk /var/snap/microk8s/common/var/lib/kubelet/pods/762387fe-fd0d-4c43-9246-0b6b300b6f5d/volumes/kubernetes.io~csi/pvc-c2f11b76-c176-4ec9-b0b7-acd78073
  │ ├─ubuntu--vg-pvc--8069500a--7c8c--45aa--8d4e--3ab04e9a94a2_00000
  │ │ 253:13 0 8G 0 lvm  
  │ │ └─drbd1001 147:1001 0 8G 0 disk 
  │ ├─ubuntu--vg-pvc--d7b2a928--c3a0--446c--a048--adabf1cecdbb_00000
  │ │ 253:14 0 2G 0 lvm  
  │ │ └─drbd1006 147:1006 0 2G 0 disk /var/snap/microk8s/common/var/lib/kubelet/pods/3e662704-4dbb-4fca-a508-c4d7e1c428d5/volumes/kubernetes.io~csi/pvc-d7b2a928-c3a0-446c-a048-adabf1ce
  │ └─ubuntu--vg-pvc--b6d0b34b--9887--4792--a1c1--b9dc0169af49_00000
  │ 253:15 0 20G 0 lvm  
  │ └─drbd1003 147:1003 0 20G 0 disk /var/snap/microk8s/common/var/lib/kubelet/pods/09997923-e64f-4e57-811d-d0f19b3f1ab2/volumes/kubernetes.io~csi/pvc-b6d0b34b-9887-4792-a1c1-b9dc0169
  └─ubuntu--vg-drbdlv01_tdata 253:9 0 300G 0 lvm  
    └─ubuntu--vg-drbdlv01-tpool 253:10 0 300G 0 lvm  
      ├─ubuntu--vg-drbdlv01 253:11 0 300G 1 lvm  
      ├─ubuntu--vg-pvc--c2f11b76--c176--4ec9--b0b7--acd78073067d_00000
      │ 253:12 0 8G 0 lvm  
      │ └─drbd1000 147:1000 0 8G 0 disk /var/snap/microk8s/common/var/lib/kubelet/pods/762387fe-fd0d-4c43-9246-0b6b300b6f5d/volumes/kubernetes.io~csi/pvc-c2f11b76-c176-4ec9-b0b7-acd78073
      ├─ubuntu--vg-pvc--8069500a--7c8c--45aa--8d4e--3ab04e9a94a2_00000
      │ 253:13 0 8G 0 lvm  
      │ └─drbd1001 147:1001 0 8G 0 disk 
      ├─ubuntu--vg-pvc--d7b2a928--c3a0--446c--a048--adabf1cecdbb_00000
      │ 253:14 0 2G 0 lvm  
      │ └─drbd1006 147:1006 0 2G 0 disk /var/snap/microk8s/common/var/lib/kubelet/pods/3e662704-4dbb-4fca-a508-c4d7e1c428d5/volumes/kubernetes.io~csi/pvc-d7b2a928-c3a0-446c-a048-adabf1ce
      └─ubuntu--vg-pvc--b6d0b34b--9887--4792--a1c1--b9dc0169af49_00000
                                  253:15 0 20G 0 lvm  
        └─drbd1003 147:1003 0 20G 0 disk /var/snap/microk8s/common/var/lib/kubelet/pods/09997923-e64f-4e57-811d-d0f19b3f1ab2/volumes/kubernetes.io~csi/pvc-b6d0b34b-9887-4792-a1c1-b9dc0169

```

After the system was powered up the lsblk does not show the drbdXXXX disks any longer

```auto
│ ├─ubuntu--vg-ubuntu--lv 253:0 0 300G 0 lvm /
│ ├─ubuntu--vg-drbdlv01_tmeta 253:1 0 344M 0 lvm  
│ │ └─ubuntu--vg-drbdlv01-tpool 253:3 0 1.3T 0 lvm  
│ │ ├─ubuntu--vg-drbdlv01 253:4 0 1.3T 1 lvm  
│ │ ├─ubuntu--vg-pvc--48402671--3490--4204--a9db--b1ce1e9e28f3_00000 253:5 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--4eb97c36--0db2--4ec9--8cbb--c859d6ca61c1_00000 253:6 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--1e8ea97e--7bcc--4ce9--ad20--a9bb1b73ac33_00000 253:7 0 2G 0 lvm  
│ │ ├─ubuntu--vg-pvc--6b64c702--2499--4dd1--9be0--76de0dc34f40_00000 253:8 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--561b15b2--cf1d--443e--8ad5--82ac39bda936_00000 253:9 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--da327e42--1c1f--49f0--b362--07688905577a_00000 253:10 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--ee9675b0--36bd--47f1--a265--d4ba7fff1343_00000 253:11 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--38d2ec2e--40c0--4397--99ce--5a99e3b01c59_00000 253:12 0 500.1G 0 lvm  
│ │ ├─ubuntu--vg-pvc--25c2c3c6--c449--47d6--8005--e8a4a52d80b5_00000 253:13 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--a1e4b63b--9a2a--472c--811c--3eb4e2426d9d_00000 253:14 0 500.1G 0 lvm  
│ │ ├─ubuntu--vg-pvc--f481ad78--0202--4349--8861--0fdd466db032_00000 253:15 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--d57e29df--3902--4e68--a519--43eafab2737e_00000 253:16 0 500.1G 0 lvm  
│ │ ├─ubuntu--vg-pvc--27659e8e--e3eb--4230--83ef--43c0eeb63cc9_00000 253:17 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--df6ae000--5cdf--47ff--b951--9aa606068f1a_00000 253:18 0 100G 0 lvm  
│ │ ├─ubuntu--vg-pvc--ac0213df--0c64--4546--ac56--725bfef7533b_00000 253:19 0 2G 0 lvm  
│ │ ├─ubuntu--vg-pvc--52490850--defe--4019--b594--5759c9d053aa_00000 253:20 0 2G 0 lvm  
│ │ ├─ubuntu--vg-pvc--acd26b2a--ab3d--457f--9ef3--c0f23e86fccc_00000 253:21 0 2G 0 lvm  
│ │ ├─ubuntu--vg-pvc--f1f727d4--514f--4529--b39a--762bafa8c5e1_00000 253:22 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--52ca1165--750c--405e--a354--6c3c1ae57cbf_00000 253:23 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--c339697b--6069--4acc--bcd1--91647d1eb29b_00000 253:24 0 8G 0 lvm  
│ │ ├─ubuntu--vg-pvc--f8afc5f7--2a83--4ddd--bda6--8158c3f57674_00000 253:25 0 8G 0 lvm  
│ │ └─ubuntu--vg-pvc--0502be2c--6377--47dd--824b--6898ecfc79b2_00000 253:26 0 8G 0 lvm  
│ └─ubuntu--vg-drbdlv01_tdata 253:2 0 1.3T 0 lvm  
│ └─ubuntu--vg-drbdlv01-tpool 253:3 0 1.3T 0 lvm  
│ ├─ubuntu--vg-drbdlv01 253:4 0 1.3T 1 lvm  
│ ├─ubuntu--vg-pvc--48402671--3490--4204--a9db--b1ce1e9e28f3_00000 253:5 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--4eb97c36--0db2--4ec9--8cbb--c859d6ca61c1_00000 253:6 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--1e8ea97e--7bcc--4ce9--ad20--a9bb1b73ac33_00000 253:7 0 2G 0 lvm  
│ ├─ubuntu--vg-pvc--6b64c702--2499--4dd1--9be0--76de0dc34f40_00000 253:8 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--561b15b2--cf1d--443e--8ad5--82ac39bda936_00000 253:9 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--da327e42--1c1f--49f0--b362--07688905577a_00000 253:10 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--ee9675b0--36bd--47f1--a265--d4ba7fff1343_00000 253:11 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--38d2ec2e--40c0--4397--99ce--5a99e3b01c59_00000 253:12 0 500.1G 0 lvm  
│ ├─ubuntu--vg-pvc--25c2c3c6--c449--47d6--8005--e8a4a52d80b5_00000 253:13 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--a1e4b63b--9a2a--472c--811c--3eb4e2426d9d_00000 253:14 0 500.1G 0 lvm  
│ ├─ubuntu--vg-pvc--f481ad78--0202--4349--8861--0fdd466db032_00000 253:15 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--d57e29df--3902--4e68--a519--43eafab2737e_00000 253:16 0 500.1G 0 lvm  
│ ├─ubuntu--vg-pvc--27659e8e--e3eb--4230--83ef--43c0eeb63cc9_00000 253:17 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--df6ae000--5cdf--47ff--b951--9aa606068f1a_00000 253:18 0 100G 0 lvm  
│ ├─ubuntu--vg-pvc--ac0213df--0c64--4546--ac56--725bfef7533b_00000 253:19 0 2G 0 lvm  
│ ├─ubuntu--vg-pvc--52490850--defe--4019--b594--5759c9d053aa_00000 253:20 0 2G 0 lvm  
│ ├─ubuntu--vg-pvc--acd26b2a--ab3d--457f--9ef3--c0f23e86fccc_00000 253:21 0 2G 0 lvm  
│ ├─ubuntu--vg-pvc--f1f727d4--514f--4529--b39a--762bafa8c5e1_00000 253:22 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--52ca1165--750c--405e--a354--6c3c1ae57cbf_00000 253:23 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--c339697b--6069--4acc--bcd1--91647d1eb29b_00000 253:24 0 8G 0 lvm  
│ ├─ubuntu--vg-pvc--f8afc5f7--2a83--4ddd--bda6--8158c3f57674_00000 253:25 0 8G 0 lvm  
│ └─ubuntu--vg-pvc--0502be2c--6377--47dd--824b--6898ecfc79b2_00000 253:26 0 8G 0 lvm  
└─sda4 8:4 0 138.4G 0 part 
  └─ubuntu--vg-drbdlv01_tdata 253:2 0 1.3T 0 lvm  
    └─ubuntu--vg-drbdlv01-tpool 253:3 0 1.3T 0 lvm  
      ├─ubuntu--vg-drbdlv01 253:4 0 1.3T 1 lvm  
      ├─ubuntu--vg-pvc--48402671--3490--4204--a9db--b1ce1e9e28f3_00000 253:5 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--4eb97c36--0db2--4ec9--8cbb--c859d6ca61c1_00000 253:6 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--1e8ea97e--7bcc--4ce9--ad20--a9bb1b73ac33_00000 253:7 0 2G 0 lvm  
      ├─ubuntu--vg-pvc--6b64c702--2499--4dd1--9be0--76de0dc34f40_00000 253:8 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--561b15b2--cf1d--443e--8ad5--82ac39bda936_00000 253:9 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--da327e42--1c1f--49f0--b362--07688905577a_00000 253:10 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--ee9675b0--36bd--47f1--a265--d4ba7fff1343_00000 253:11 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--38d2ec2e--40c0--4397--99ce--5a99e3b01c59_00000 253:12 0 500.1G 0 lvm  
      ├─ubuntu--vg-pvc--25c2c3c6--c449--47d6--8005--e8a4a52d80b5_00000 253:13 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--a1e4b63b--9a2a--472c--811c--3eb4e2426d9d_00000 253:14 0 500.1G 0 lvm  
      ├─ubuntu--vg-pvc--f481ad78--0202--4349--8861--0fdd466db032_00000 253:15 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--d57e29df--3902--4e68--a519--43eafab2737e_00000 253:16 0 500.1G 0 lvm  
      ├─ubuntu--vg-pvc--27659e8e--e3eb--4230--83ef--43c0eeb63cc9_00000 253:17 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--df6ae000--5cdf--47ff--b951--9aa606068f1a_00000 253:18 0 100G 0 lvm  
      ├─ubuntu--vg-pvc--ac0213df--0c64--4546--ac56--725bfef7533b_00000 253:19 0 2G 0 lvm  
      ├─ubuntu--vg-pvc--52490850--defe--4019--b594--5759c9d053aa_00000 253:20 0 2G 0 lvm  
      ├─ubuntu--vg-pvc--acd26b2a--ab3d--457f--9ef3--c0f23e86fccc_00000 253:21 0 2G 0 lvm  
      ├─ubuntu--vg-pvc--f1f727d4--514f--4529--b39a--762bafa8c5e1_00000 253:22 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--52ca1165--750c--405e--a354--6c3c1ae57cbf_00000 253:23 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--c339697b--6069--4acc--bcd1--91647d1eb29b_00000 253:24 0 8G 0 lvm  
      ├─ubuntu--vg-pvc--f8afc5f7--2a83--4ddd--bda6--8158c3f57674_00000 253:25 0 8G 0 lvm  
      └─ubuntu--vg-pvc--0502be2c--6377--47dd--824b--6898ecfc79b2_00000 253:26 0 8G 0 lvm  

```

Using the linstor command in the piraeus-cs-controller pod I can see the following:

linstor resoirce-definition list

```auto
╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Port ┊ ResourceGroup ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-1e8ea97e-7bcc-4ce9-ad20-a9bb1b73ac33 ┊ 7004 ┊ sc-b676293a-803a-5779-b7ae-ee4d5cbccbf9 ┊ ok ┊
┊ pvc-3c824d13-7cef-41aa-843f-179c98da108b ┊ 7027 ┊ sc-b676293a-803a-5779-b7ae-ee4d5cbccbf9 ┊ ok ┊
┊ pvc-4eb97c36-0db2-4ec9-8cbb-c859d6ca61c1 ┊ 7002 ┊ sc-b676293a-803a-5779-b7ae-ee4d5cbccbf9 ┊ ok ┊
┊ pvc-6b07fbdf-0486-4698-b0e0-1a0d148751b0 ┊ 7022 ┊ sc-b676293a-803a-5779-b7ae-ee4d5cbccbf9 ┊ ok ┊
┊ pvc-6b64c702-2499-4dd1-9be0-76de0dc34f40 ┊ 7007 ┊ sc-b676293a-803a-5779-b7ae-ee4d5cbccbf9 ┊ ok ┊
┊ pvc-8ffb5ad7-c945-4fb3-8e68-e7c6da77801d ┊ 7003 ┊ sc-b676293a-803a-5779-b7ae-ee4d5cbccbf9 ┊ ok ┊
┊ pvc-25c2c3c6-c449-47d6-8005-e8a4a52d80b5 ┊ 7010 ┊ sc-b676293a-803a-5779-b7ae-ee4d5cbccbf9 ┊ ok ┊
┊ pvc-83908dac-013d-4b5c-867b-d7b9ac7e0df5 ┊ 7024 ┊ sc-b676293a-803a-5779-b7ae-ee4d5cbccbf9 ┊ ok ┊
...

```

linstor resource list

```auto
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ pvc-1e8ea97e-7bcc-4ce9-ad20-a9bb1b73ac33 ┊ qqd-srv1 ┊ 7004 ┊ ┊ ┊ Unknown ┊ 2023-04-04 16:38:26 ┊
┊ pvc-1e8ea97e-7bcc-4ce9-ad20-a9bb1b73ac33 ┊ qqd-srv2 ┊ 7004 ┊ ┊ ┊ Unknown ┊ 2023-04-04 16:38:23 ┊
┊ pvc-1e8ea97e-7bcc-4ce9-ad20-a9bb1b73ac33 ┊ qqd-srv3 ┊ 7004 ┊ ┊ ┊ Unknown ┊ 2023-04-26 15:35:03 ┊
┊ pvc-3c824d13-7cef-41aa-843f-179c98da108b ┊ qqd-srv4 ┊ 7027 ┊ ┊ ┊ Unknown ┊ 2024-07-11 19:45:48 ┊
┊ pvc-3c824d13-7cef-41aa-843f-179c98da108b ┊ qqd-srv1 ┊ 7027 ┊ ┊ ┊ Unknown ┊ 2024-07-11 19:45:49 ┊
┊ pvc-3c824d13-7cef-41aa-843f-179c98da108b ┊ qqd-srv3 ┊ 7027 ┊ ┊ ┊ Unknown ┊ 2024-07-11 19:45:43 ┊
┊ pvc-4eb97c36-0db2-4ec9-8cbb-c859d6ca61c1 ┊ qqd-srv1 ┊ 7002 ┊ ┊ ┊ Unknown ┊ 2023-03-17 17:17:19 ┊
┊ pvc-4eb97c36-0db2-4ec9-8cbb-c859d6ca61c1 ┊ qqd-srv2 ┊ 7002 ┊ ┊ ┊ Unknown ┊ 2023-02-09 21:05:04 ┊
┊ pvc-83908dac-013d-4b5c-867b-d7b9ac7e0df5 ┊ qqd-srv4 ┊ 7024 ┊ ┊ ┊ Unknown ┊ 2024-07-11 19:45:45 ┊
┊ pvc-83908dac-013d-4b5c-867b-d7b9ac7e0df5 ┊ qqd-srv1 ┊ 7024 ┊ ┊ ┊ Unknown ┊ 2024-07-11 19:45:47 ┊
┊ pvc-83908dac-013d-4b5c-867b-d7b9ac7e0df5 ┊ qqd-srv3 ┊ 7024 ┊ ┊ ┊ Unknown ┊ 2024-07-11 19:45:42 ┊
...

```

There are two types of errors logged: StorageException and NullPointerException (shown below)

```auto
...
┊ 669049AF-F218A-046607 ┊ 2024-07-22 14:35:11 ┊ S|qqd-srv1 ┊ NullPointerException ┊
┊ 669049ED-D9B03-020656 ┊ 2024-07-22 14:35:12 ┊ S|qqd-srv4 ┊ NullPointerException ┊
┊ 669049AF-F218A-046608 ┊ 2024-07-22 14:35:12 ┊ S|qqd-srv1 ┊ NullPointerException ┊
┊ 66914BED-9F848-034603 ┊ 2024-07-22 14:35:12 ┊ S|qqd-srv3 ┊ NullPointerException ┊
┊ 66914BED-9F848-034604 ┊ 2024-07-22 14:35:42 ┊ S|qqd-srv3 ┊ StorageException: Device does not exist. VlmData: Node: 'qqd-srv3', Rsc: 'p... ┊
┊ 669049AF-F218A-046609 ┊ 2024-07-22 14:35:42 ┊ S|qqd-srv1 ┊ StorageException: Device does not exist. VlmData: Node: 'qqd-srv1', Rsc: 'p... ┊
┊ 66914BED-9F848-034605 ┊ 2024-07-22 14:35:42 ┊ S|qqd-srv3 ┊ StorageException: Device does not exist. VlmData: Node: 'qqd-srv3', Rsc: 'p... ┊
┊ 669049AB-203B0-009609 ┊ 2024-07-22 14:35:42 ┊ S|qqd-srv2 ┊ StorageException: Device does not exist. VlmData: Node: 'qqd-srv2', Rsc: 'p... ┊
...

```

This is the StorageException:

```auto
root@piraeus-cs-controller:/# l err s 66914BED-9F848-025559
ERROR REPORT 66914BED-9F848-025559

============================================================

Application: LINBIT�� LINSTOR
Module: Satellite
Version: 1.19.1
Build ID: a758bf07796c374fd2004465b0d8690209b74356
Build time: 2022-07-27T06:36:54+00:00
Error time: 2024-07-19 23:44:42
Node: qqd-srv3

============================================================

Reported error:
===============

Description:
    Device does not exist.
Cause:
    The volume could not be found on the system.

Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'getAllocatedSize', Source file 'StltProviderUtils.java', Line #21

Error message: Device does not exist. VlmData: Node: 'qqd-srv3', Rsc: 'pvc-83908dac-013d-4b5c-867b-d7b9ac7e0df5', VlmNr: '0', suffix: 

Call backtrace:

    Method Native Class:Line number
    getAllocatedSize N com.linbit.linstor.layer.storage.utils.StltProviderUtils:21
    getAllocatedSize N com.linbit.linstor.layer.storage.AbsStorageProvider:1241
    updateAllocatedSize N com.linbit.linstor.layer.storage.AbsStorageProvider:1513
    getVlmAllocatedCapacities N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:171
    executeInScope N com.linbit.linstor.api.protobuf.ReqVlmAllocated:84
    lambda$executeReactive$0 N com.linbit.linstor.api.protobuf.ReqVlmAllocated:69
    doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:150
    lambda$fluxInScope$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:76
    call N reactor.core.publisher.MonoCallable:91
    trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:126
    subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
    subscribe N reactor.core.publisher.Flux:8343
    onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request N reactor.core.publisher.Operators$ScalarSubscription:2344
    onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe N reactor.core.publisher.MonoCurrentContext:35
    subscribe N reactor.core.publisher.Flux:8357
    trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:199
    subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
    subscribe N reactor.core.publisher.Flux:8343
    onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:418
    slowPath N reactor.core.publisher.FluxArray$ArraySubscription:126
    request N reactor.core.publisher.FluxArray$ArraySubscription:99
    onSubscribe N reactor.core.publisher.FluxFlatMap$FlatMapMain:363
    subscribe N reactor.core.publisher.FluxMerge:69
    subscribe N reactor.core.publisher.Flux:8357
    onComplete N reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:207
    subscribe N reactor.core.publisher.FluxConcatArray:80
    subscribe N reactor.core.publisher.InternalFluxOperator:62
    subscribe N reactor.core.publisher.FluxDefer:54
    subscribe N reactor.core.publisher.Flux:8357
    onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:418
    drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:414
    drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:679
    onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:243
    drainFused N reactor.core.publisher.UnicastProcessor:286
    drain N reactor.core.publisher.UnicastProcessor:329
    onNext N reactor.core.publisher.UnicastProcessor:408
    next N reactor.core.publisher.FluxCreate$IgnoreSink:618
    next N reactor.core.publisher.FluxCreate$SerializedSink:153
    processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:383
    doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:218
    lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:164
    onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:177
    runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:439
    run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:526
    call N reactor.core.scheduler.WorkerTask:84
    call N reactor.core.scheduler.WorkerTask:37
    run N java.util.concurrent.FutureTask:264
    run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
    runWorker N java.util.concurrent.ThreadPoolExecutor:1128
    run N java.util.concurrent.ThreadPoolExecutor$Worker:628
    run N java.lang.Thread:829

END OF ERROR REPORT.

```

This is the NullPointer type of error. The pvc instance is different. It’s hard to find the corresponding one to the above error, but there is one similar to this.

```auto
root@piraeus-cs-controller:/# l err s 669049AF-F218A-035304
ERROR REPORT 669049AF-F218A-035304

============================================================

Application: LINBIT�� LINSTOR
Module: Satellite
Version: 1.19.1
Build ID: a758bf07796c374fd2004465b0d8690209b74356
Build time: 2022-07-27T06:36:54+00:00
Error time: 2024-07-19 23:45:50
Node: qqd-srv1

============================================================

Reported error:
===============

Category: RuntimeException
Class name: NullPointerException
Class canonical name: java.lang.NullPointerException
Generated at: <UNKNOWN>

Error context:
    An error occurred while processing resource 'Node: 'qqd-srv1', Rsc: 'pvc-82e74425-2932-40c1-8100-6328b8e8e8c9''

Call backtrace:

    Method Native Class:Line number

END OF ERROR REPORT.

```

Thanks again!

---

<div class="post-metadata">

**Author:** ![Devin](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/devin/32/26_2.png) [@Devin](https://forums.linbit.com/u/Devin)\
**Post date:** [July 24, 2024, 5:49pm UTC](https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173/4 "2024-07-24T17:49:45Z")

</div>

If only DRBD is not returning after a reboot, my first guess would be that the systems might have rebooted into a new kernel, which is incompatible with the installed DRBD kernel module.

Check to see if a DRBD module is already loaded and if it is a 9.x version. You can check this via `cat /proc/drbd` If nothing is loaded try a `modprobe drbd` and see if that returns any useful error.

---

<div class="post-metadata">

**Author:** ![dshiklov](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dshiklov](https://forums.linbit.com/u/dshiklov)\
**Post date:** [July 24, 2024, 7:08pm UTC](https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173/5 "2024-07-24T19:08:21Z")

</div>

This particular system was build quite awhile ago and its version is 8.4.11.  
$ cat /proc/drbd  
version: 8.4.11 (api:1/proto:86-101)  
srcversion: C7B8F7076B8D6DB066D84D9

Is this issue known prior to 9.x?

---

<div class="post-metadata">

**Author:** ![Devin](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/devin/32/26_2.png) [@Devin](https://forums.linbit.com/u/Devin)\
**Post date:** [July 24, 2024, 7:29pm UTC](https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173/6 "2024-07-24T19:29:36Z")

</div>

LINSTOR requires DRBD v9 or later. It simply will not work with DRBD v8. You must have been running a newer version of DRBD before the reboot.

As I had mentioned, perhaps the reboot loaded to a new kernel? I would investigate and see if there are any `drbd.ko` files present for any older installed kernels. To fix this you will need to boot to the previous kernel or install a new DRBD kernel module.

---

<div class="post-metadata">

**Author:** ![dshiklov](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dshiklov](https://forums.linbit.com/u/dshiklov)\
**Post date:** [July 24, 2024, 8:09pm UTC](https://forums.linbit.com/t/recover-from-abrupt-server-shutdown/173/7 "2024-07-24T20:09:42Z")

</div>

Thank you Devin!

I found a mixture of 8.x and 9.x modules in /usr/lib/modules/ under different kernel versions.

Appreciate you help!