Completely lost on what to do after major power outage

General
,
1

Before I say anything, I think this community is great and im learning so much from this forum, but i may have bit off more than I can chew at this moment.

So i have a 3 node incus v(6.18) cluster with a seperate 3 node linstor cluster that i’ve setup for a class at my university and I have just experienced a total power outage. When i turned back my power I am having huge issues with restoring my containers from their respective state with linbit.

I have attached the error reports at the bottom as well.

I keep getting this issue where it says that it cant mount the drbd mount or that its stuck in read-only mode … I have reconnected all my nodes from the linstor controller and they connect fine and all show online, but im afraid that i cant I cant recover from this and its important data on there that i was waiting to backup. Can anyone shed some light on this? My professor is gonna kill me if I cant get this back up :joy:, kinda freaking out.

If i go on the GUI i can only see the disks from the storage pool I setup for incus on one of the nodes, the others look empty. This may be from error-io on no quorum which i read about earlier.

I looked at dmesg and i see the following error for most of the mounts..

[ 5564.527381] drbd incus-volume-353e79f3b8e64416bf18b235dbf92cfd: no UpToDate peer with quorum
[ 5564.527384] drbd incus-volume-353e79f3b8e64416bf18b235dbf92cfd: Auto-promote failed: No quorum (-25)
[ 5564.527740] /dev/drbd1039: Can’t open blockdev

and also here is the output of linstor resource list

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName                                  ┊ Node       ┊ Layers       ┊ Usage  ┊ Conns                                       ┊    State ┊ CreatedOn           ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ incus-volume-0afe72c1ffbf47e882011fe28af797d9 ┊ gigabyte   ┊ DRBD,STORAGE ┊ InUse  ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-09-28 14:06:51 ┊
┊ incus-volume-0afe72c1ffbf47e882011fe28af797d9 ┊ r620       ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-11-10 16:54:59 ┊
┊ incus-volume-0afe72c1ffbf47e882011fe28af797d9 ┊ r730xd-1   ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3,r620)          ┊ UpToDate ┊ 2025-09-28 14:03:13 ┊
┊ incus-volume-0afe72c1ffbf47e882011fe28af797d9 ┊ r730xd-2   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-28 14:03:13 ┊
┊ incus-volume-0afe72c1ffbf47e882011fe28af797d9 ┊ r730xd-3   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-28 14:03:13 ┊
┊ incus-volume-0afe72c1ffbf47e882011fe28af797d9 ┊ supermicro ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-11-10 16:57:31 ┊
┊ incus-volume-0cb1c42e43534509a8b59c96b2d44b49 ┊ gigabyte   ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-11-10 17:38:48 ┊
┊ incus-volume-0cb1c42e43534509a8b59c96b2d44b49 ┊ r620       ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-11-10 17:38:39 ┊
┊ incus-volume-0cb1c42e43534509a8b59c96b2d44b49 ┊ r730xd-1   ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3,r620)          ┊ UpToDate ┊ 2025-09-28 14:01:43 ┊
┊ incus-volume-0cb1c42e43534509a8b59c96b2d44b49 ┊ r730xd-2   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-28 14:01:43 ┊
┊ incus-volume-0cb1c42e43534509a8b59c96b2d44b49 ┊ r730xd-3   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-28 14:01:43 ┊
┊ incus-volume-0cb1c42e43534509a8b59c96b2d44b49 ┊ supermicro ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-10-01 20:05:03 ┊
┊ incus-volume-1cd2c1abd5db4fadbd24d3d7ede8c2ce ┊ gigabyte   ┊ DRBD,STORAGE ┊ InUse  ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-10-01 20:12:30 ┊
┊ incus-volume-1cd2c1abd5db4fadbd24d3d7ede8c2ce ┊ r620       ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-11-10 16:54:59 ┊
┊ incus-volume-1cd2c1abd5db4fadbd24d3d7ede8c2ce ┊ r730xd-1   ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3,r620)          ┊ UpToDate ┊ 2025-09-29 16:18:38 ┊
┊ incus-volume-1cd2c1abd5db4fadbd24d3d7ede8c2ce ┊ r730xd-2   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-29 16:18:38 ┊
┊ incus-volume-1cd2c1abd5db4fadbd24d3d7ede8c2ce ┊ r730xd-3   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-29 16:18:38 ┊
┊ incus-volume-1cd2c1abd5db4fadbd24d3d7ede8c2ce ┊ supermicro ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-11-10 16:57:32 ┊
┊ incus-volume-3eca117a33cf4a1784bcb5d7e40411a8 ┊ gigabyte   ┊ DRBD,STORAGE ┊ InUse  ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-10-19 15:28:51 ┊
┊ incus-volume-3eca117a33cf4a1784bcb5d7e40411a8 ┊ r620       ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-11-10 16:57:32 ┊
┊ incus-volume-3eca117a33cf4a1784bcb5d7e40411a8 ┊ r730xd-1   ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3,r620)          ┊ UpToDate ┊ 2025-10-19 15:28:41 ┊
┊ incus-volume-3eca117a33cf4a1784bcb5d7e40411a8 ┊ r730xd-2   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-10-19 15:28:41 ┊
┊ incus-volume-3eca117a33cf4a1784bcb5d7e40411a8 ┊ r730xd-3   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-10-19 15:28:41 ┊
┊ incus-volume-3eca117a33cf4a1784bcb5d7e40411a8 ┊ supermicro ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-11-10 16:57:41 ┊
┊ incus-volume-5bd43e17dd6b40d8a3a4bb1e5826c8d5 ┊ r620       ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-11-10 17:37:47 ┊
┊ incus-volume-5bd43e17dd6b40d8a3a4bb1e5826c8d5 ┊ r730xd-1   ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3,r620)          ┊ UpToDate ┊ 2025-09-28 14:01:16 ┊
┊ incus-volume-5bd43e17dd6b40d8a3a4bb1e5826c8d5 ┊ r730xd-2   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-28 14:01:15 ┊
┊ incus-volume-5bd43e17dd6b40d8a3a4bb1e5826c8d5 ┊ r730xd-3   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-28 14:01:16 ┊
┊ incus-volume-5bd43e17dd6b40d8a3a4bb1e5826c8d5 ┊ supermicro ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-10-01 20:05:00 ┊
┊ incus-volume-6ced1b2804c24652ad90f8c0cb45aec5 ┊ gigabyte   ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-11-10 16:21:49 ┊
┊ incus-volume-6ced1b2804c24652ad90f8c0cb45aec5 ┊ r620       ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-11-10 16:23:25 ┊
┊ incus-volume-6ced1b2804c24652ad90f8c0cb45aec5 ┊ r730xd-1   ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3,r620)          ┊ UpToDate ┊ 2025-09-28 14:03:20 ┊
┊ incus-volume-6ced1b2804c24652ad90f8c0cb45aec5 ┊ r730xd-2   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-28 14:03:20 ┊
┊ incus-volume-6ced1b2804c24652ad90f8c0cb45aec5 ┊ r730xd-3   ┊ DRBD,STORAGE ┊        ┊                                             ┊  Unknown ┊ 2025-09-28 14:03:20 ┊
┊ incus-volume-6ced1b2804c24652ad90f8c0cb45aec5 ┊ supermicro ┊ DRBD,STORAGE ┊ Unused ┊ Connecting(r730xd-2,r730xd-3)               ┊ Diskless ┊ 2025-11-10 16:12:03 ┊

the r730xd-1 is my linstor controller and the rest are satellites. Also, r730xd-2 and r730xd-3 are the other storage nodes… my incus nodes are gigabyte, r620 and supermicro.

this is the output of drbdadm status…

incus-volume-08c3e1c02f764335a6a874325b549ac6 role:Secondary
  disk:UpToDate open:no
  gigabyte role:Primary
    peer-disk:Diskless
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-0afe72c1ffbf47e882011fe28af797d9 role:Secondary
  disk:UpToDate open:no
  gigabyte role:Primary
    peer-disk:Diskless
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-0cb1c42e43534509a8b59c96b2d44b49 role:Secondary suspended:quorum
  disk:UpToDate quorum:no open:no blocked:upper
  gigabyte role:Secondary
    peer-disk:Diskless
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-1184bdc40c084bf7911779109aabd75a role:Secondary
  disk:UpToDate open:no
  gigabyte role:Primary
    peer-disk:Diskless
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-13554e2ba2834d728327b411dbace40b role:Secondary suspended:quorum
  disk:Outdated quorum:no open:no
  gigabyte role:Secondary
    peer-disk:Diskless
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-15ddb08d4e8b4efe80156ef4ce5ded95 role:Secondary
  disk:UpToDate open:no
  gigabyte role:Primary
    peer-disk:Diskless
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-160bcab0d7e445aa89a05f267ad6eb3b role:Secondary suspended:quorum
  disk:UpToDate quorum:no open:no blocked:upper
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-18eb10b18c1749079efb3e3872b13823 role:Secondary suspended:quorum
  disk:UpToDate quorum:no open:no blocked:upper
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-1cd2c1abd5db4fadbd24d3d7ede8c2ce role:Secondary
  disk:UpToDate open:no
  gigabyte role:Primary
    peer-disk:Diskless
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-26018aef7a47442c9e296d8cccfe173b role:Secondary
  disk:UpToDate open:no
  gigabyte connection:Connecting
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting
  supermicro role:Secondary
    peer-disk:Diskless

incus-volume-27eb7afa83684c56ac8ae2929eb4f492 role:Secondary suspended:quorum
  disk:UpToDate quorum:no open:no blocked:upper
  gigabyte role:Secondary
    peer-disk:Diskless
  r620 connection:Connecting
  r730xd-2 connection:Connecting
  r730xd-3 connection:Connecting

ERROR REPORT (From Controller)

ERROR REPORT 69128364-00000-000110

============================================================

Application:                        LINBIT® LINSTOR
Module:                             Controller
Version:                            1.32.3
Build ID:                           6dac06aed233f2c89ac7cc6b1185d6dce9ec74c4
Build time:                         2025-10-13T06:37:58+00:00
Error time:                         2025-11-10 20:53:32
Node:                               r730xd-1
Thread:                             MainWorkerPool-7
Access context information

Identity:                           PUBLIC
Role:                               PUBLIC
Domain:                             PUBLIC

Peer:                               RestClient(192.0.0.1; 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36')

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         DelayedApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils.DelayedApiRcException
Generated at:                       Method 'lambda$mergeExtractingApiRcExceptions$6', Source file 'CtrlResponseUtils.java', Line #187

Error message:                      Exceptions have been converted to responses

Error context:
        Registration of resource group 'micron_pool' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses
Asynchronous stage backtrace:
        (r730xd-2) No response generated by handler.
        (r730xd-3) No response generated by handler.
    
    Error has been observed at the following site(s):
    	*__checkpoint ⇢ Modify resource-group
    Original Stack Trace:

Call backtrace:

    Method                                   Native Class:Line number
    lambda$mergeExtractingApiRcExceptions$6  N      com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils:187

Suppressed exception 1 of 3:
===============
Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #344

Error message:                      (r730xd-2) No response generated by handler.

Error context:
        Registration of resource group 'micron_pool' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses
Call backtrace:

    Method                                   Native Class:Line number
    handleAnswer                             N      com.linbit.linstor.proto.CommonMessageProcessor:344
    handleDataMessage                        N      com.linbit.linstor.proto.CommonMessageProcessor:297
    doProcessInOrderMessage                  N      com.linbit.linstor.proto.CommonMessageProcessor:245
    lambda$doProcessMessage$4                N      com.linbit.linstor.proto.CommonMessageProcessor:230
    subscribe                                N      reactor.core.publisher.FluxDefer:46
    subscribe                                N      reactor.core.publisher.Flux:8848
    onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:430
    drainAsync                               N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
    drain                                    N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
    onNext                                   N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
    drainFused                               N      reactor.core.publisher.SinkManyUnicast:321
    drain                                    N      reactor.core.publisher.SinkManyUnicast:363
    tryEmitNext                              N      reactor.core.publisher.SinkManyUnicast:239
    tryEmitNext                              N      reactor.core.publisher.SinkManySerialized:100
    processInOrder                           N      com.linbit.linstor.netcom.TcpConnectorPeer:446
    doProcessMessage                         N      com.linbit.linstor.proto.CommonMessageProcessor:228
    lambda$processMessage$2                  N      com.linbit.linstor.proto.CommonMessageProcessor:165
    onNext                                   N      reactor.core.publisher.FluxPeek$PeekSubscriber:185
    runAsync                                 N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
    run                                      N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
    call                                     N      reactor.core.scheduler.WorkerTask:84
    call                                     N      reactor.core.scheduler.WorkerTask:37
    run                                      N      java.util.concurrent.FutureTask:317
    run                                      N      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
    runWorker                                N      java.util.concurrent.ThreadPoolExecutor:1144
    run                                      N      java.util.concurrent.ThreadPoolExecutor$Worker:642
    run                                      N      java.lang.Thread:1583

Suppressed exception 2 of 3:
===============
Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'handleAnswer', Source file 'CommonMessageProcessor.java', Line #344

Error message:                      (r730xd-3) No response generated by handler.

Error context:
        Registration of resource group 'micron_pool' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses
Call backtrace:

    Method                                   Native Class:Line number
    handleAnswer                             N      com.linbit.linstor.proto.CommonMessageProcessor:344
    handleDataMessage                        N      com.linbit.linstor.proto.CommonMessageProcessor:297
    doProcessInOrderMessage                  N      com.linbit.linstor.proto.CommonMessageProcessor:245
    lambda$doProcessMessage$4                N      com.linbit.linstor.proto.CommonMessageProcessor:230
    subscribe                                N      reactor.core.publisher.FluxDefer:46
    subscribe                                N      reactor.core.publisher.Flux:8848
    onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:430
    drainAsync                               N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
    drain                                    N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
    onNext                                   N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
    drainFused                               N      reactor.core.publisher.SinkManyUnicast:321
    drain                                    N      reactor.core.publisher.SinkManyUnicast:363
    tryEmitNext                              N      reactor.core.publisher.SinkManyUnicast:239
    tryEmitNext                              N      reactor.core.publisher.SinkManySerialized:100
    processInOrder                           N      com.linbit.linstor.netcom.TcpConnectorPeer:446
    doProcessMessage                         N      com.linbit.linstor.proto.CommonMessageProcessor:228
    lambda$processMessage$2                  N      com.linbit.linstor.proto.CommonMessageProcessor:165
    onNext                                   N      reactor.core.publisher.FluxPeek$PeekSubscriber:185
    runAsync                                 N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
    run                                      N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
    call                                     N      reactor.core.scheduler.WorkerTask:84
    call                                     N      reactor.core.scheduler.WorkerTask:37
    run                                      N      java.util.concurrent.FutureTask:317
    run                                      N      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
    runWorker                                N      java.util.concurrent.ThreadPoolExecutor:1144
    run                                      N      java.util.concurrent.ThreadPoolExecutor$Worker:642
    run                                      N      java.lang.Thread:1583

Suppressed exception 3 of 3:
===============
Category:                           RuntimeException
Class name:                         OnAssemblyException
Class canonical name:               reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at:                       Method 'lambda$mergeExtractingApiRcExceptions$6', Source file 'CtrlResponseUtils.java', Line #187

Error message:                      
Error has been observed at the following site(s):
	*__checkpoint ⇢ Modify resource-group
Original Stack Trace:

Error context:
        Registration of resource group 'micron_pool' failed due to an unhandled exception of type DelayedApiRcException. Exceptions have been converted to responses
Call backtrace:

    Method                                   Native Class:Line number
    lambda$mergeExtractingApiRcExceptions$6  N      com.linbit.linstor.core.apicallhandler.response.CtrlResponseUtils:187
    subscribe                                N      reactor.core.publisher.FluxDefer:46
    subscribe                                N      reactor.core.publisher.Flux:8848
    onComplete                               N      reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:238
    onComplete                               N      reactor.core.publisher.FluxMap$MapSubscriber:144
    checkTerminated                          N      reactor.core.publisher.FluxFlatMap$FlatMapMain:850
    drainLoop                                N      reactor.core.publisher.FluxFlatMap$FlatMapMain:612
    innerComplete                            N      reactor.core.publisher.FluxFlatMap$FlatMapMain:898
    onComplete                               N      reactor.core.publisher.FluxFlatMap$FlatMapInner:1001
    onComplete                               N      reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2230
    onComplete                               N      reactor.core.publisher.FluxMap$MapSubscriber:144
    onComplete                               N      reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:209
    onComplete                               N      reactor.core.publisher.FluxPeek$PeekSubscriber:260
    onComplete                               N      reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2230
    onComplete                               N      reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber:152
    onComplete                               N      reactor.core.publisher.FluxUsing$UsingSubscriber:236
    onComplete                               N      reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber:85
    complete                                 N      reactor.core.publisher.FluxCreate$BaseSink:465
    drain                                    N      reactor.core.publisher.FluxCreate$BufferAsyncSink:871
    complete                                 N      reactor.core.publisher.FluxCreate$BufferAsyncSink:819
    drainLoop                                N      reactor.core.publisher.FluxCreate$SerializedFluxSink:249
    drain                                    N      reactor.core.publisher.FluxCreate$SerializedFluxSink:215
    complete                                 N      reactor.core.publisher.FluxCreate$SerializedFluxSink:206
    apiCallComplete                          N      com.linbit.linstor.netcom.TcpConnectorPeer:540
    handleComplete                           N      com.linbit.linstor.proto.CommonMessageProcessor:370
    handleDataMessage                        N      com.linbit.linstor.proto.CommonMessageProcessor:300
    doProcessInOrderMessage                  N      com.linbit.linstor.proto.CommonMessageProcessor:245
    lambda$doProcessMessage$4                N      com.linbit.linstor.proto.CommonMessageProcessor:230
    subscribe                                N      reactor.core.publisher.FluxDefer:46
    subscribe                                N      reactor.core.publisher.Flux:8848
    onNext                                   N      reactor.core.publisher.FluxFlatMap$FlatMapMain:430
    drainAsync                               N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
    drain                                    N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
    onNext                                   N      reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
    drainFused                               N      reactor.core.publisher.SinkManyUnicast:321
    drain                                    N      reactor.core.publisher.SinkManyUnicast:363
    tryEmitNext                              N      reactor.core.publisher.SinkManyUnicast:239
    tryEmitNext                              N      reactor.core.publisher.SinkManySerialized:100
    processInOrder                           N      com.linbit.linstor.netcom.TcpConnectorPeer:446
    doProcessMessage                         N      com.linbit.linstor.proto.CommonMessageProcessor:228
    lambda$processMessage$2                  N      com.linbit.linstor.proto.CommonMessageProcessor:165
    onNext                                   N      reactor.core.publisher.FluxPeek$PeekSubscriber:185
    runAsync                                 N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
    run                                      N      reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
    call                                     N      reactor.core.scheduler.WorkerTask:84
    call                                     N      reactor.core.scheduler.WorkerTask:37
    run                                      N      java.util.concurrent.FutureTask:317
    run                                      N      java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
    runWorker                                N      java.util.concurrent.ThreadPoolExecutor:1144
    run                                      N      java.util.concurrent.ThreadPoolExecutor$Worker:642
    run                                      N      java.lang.Thread:1583


END OF ERROR REPORT.
2

That is a good first start if all satellites show as Online in linstor node list.

In the linstor resource list however we see that some of your peers do not want to connect to the other peers.
The reason why you get cannot mount the DRBD device is exactly what the error states: You do not have quorum, or in other words the majority of peers are offline / not connected. You will need to get more peers online / connected before you regain quorum and can mount DRBD.

(Side-note: Even if you can only mount the DRBD device as read-only that should be enough to draw a backup, just in case…)

To get the peers back online I’d suggest you carefully check the ErrorReports (from linstor err list) especially those from the nodes that have unconnected peers (i.e. r730xd-2, r730xd-3, r620) to find the root cause.

It depends on your finding how to further proceed from there. LINSTOR usually needs some LV to put a DRBD on top of it. If for example the LV is not accessible, you’d need to find out why and how to get it back (which might be way outside of LINSTOR/DRBD context).

Let us know if you find more details.

3

ghernadi So if i run linstor err list, I get the following … ( I have greatly shortened the list),

interestingly enough if i run lvdisplay or vgdisplay on r730xd-2 and r730xd-3 they show without issue…

 68DDEB8C-1A0E0-000076 ┊ 2025-10-03 20:31:08 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-7bd9fb8fdb3e4...    ┊
┊ 68DDEC67-2EA29-000066 ┊ 2025-10-03 20:31:08 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-26018aef7a474...    ┊
┊ 68DDEC67-2EA29-000067 ┊ 2025-10-03 20:31:08 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-55a04fdae3904...    ┊
┊ 68DDEC67-2EA29-000068 ┊ 2025-10-03 20:31:08 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-675f66fc0b3e4...    ┊
┊ 68DDEB8C-1A0E0-000077 ┊ 2025-10-03 20:31:23 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-d4c9342bd12e4...    ┊
┊ 68DDEC67-2EA29-000069 ┊ 2025-10-03 20:31:23 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-c12ab01704b04...    ┊
┊ 68DDEB8C-1A0E0-000078 ┊ 2025-10-03 20:31:23 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-b1f7ec0f1e664...    ┊
┊ 68DDEB8C-1A0E0-000079 ┊ 2025-10-03 20:31:23 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-e20245b8cf504...    ┊
┊ 68DDEB8C-1A0E0-000080 ┊ 2025-10-03 20:31:23 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-e4b065b0c60c4...    ┊
┊ 68DDEC67-2EA29-000070 ┊ 2025-10-03 20:31:23 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-6e09e94872ef4...    ┊
┊ 68DDEC67-2EA29-000071 ┊ 2025-10-03 20:31:23 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-b4d37d3540594...    ┊
┊ 68DDEC67-2EA29-000072 ┊ 2025-10-03 20:31:23 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-c8e14a08251f4...    ┊
┊ 68DDEB8C-1A0E0-000081 ┊ 2025-10-03 20:31:38 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-ee73769221ce4...    ┊
┊ 68DDEC67-2EA29-000073 ┊ 2025-10-03 20:31:38 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-e2b0a4ffebf84...    ┊
┊ 68DDEB8C-1A0E0-000082 ┊ 2025-10-03 20:31:38 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-ea74262289e04...    ┊
┊ 68DDEB8C-1A0E0-000083 ┊ 2025-10-03 20:31:38 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-f3acdc7c09444...    ┊
┊ 68DDEC67-2EA29-000074 ┊ 2025-10-03 20:31:38 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-ea8e73dd74eb4...    ┊
┊ 68DDEB8C-1A0E0-000084 ┊ 2025-10-03 20:31:38 ┊ S|supermicro ┊ ResourceException: Failed to adjust DRBD resource incus-volume-f7650070d6db4...    ┊
┊ 68DDEC67-2EA29-000075 ┊ 2025-10-03 20:31:38 ┊ S|gigabyte   ┊ ResourceException: Failed to adjust DRBD resource incus-volume-fba9ac3a7f6f4...    ┊
┊ 68DDED56-00000-000003 ┊ 2025-10-03 20:33:16 ┊ C|r730xd-1   ┊ PeerClosingConnectionException                                                     ┊
┊ 68DDED56-00000-000004 ┊ 2025-10-03 20:33:16 ┊ C|r730xd-1   ┊ PeerClosingConnectionException                                                     ┊
┊ 68DDED56-00000-000005 ┊ 2025-10-03 20:33:16 ┊ C|r730xd-1   ┊ IllegalStateException: The current scope has already been entered                  ┊
┊ 68DDEC67-2EA29-000076 ┊ 2025-10-03 20:38:48 ┊ S|gigabyte   ┊ SocketException: Connection reset                                                  ┊
┊ 68E096BD-052D2-000000 ┊ 2025-10-03 20:39:06 ┊ S|r730xd-1   ┊ StorageException: Failed to delete lvm snapshot                                    ┊
┊ 68DDEE87-4D03B-000000 ┊ 2025-10-03 20:40:02 ┊ S|r620       ┊ SocketException: Connection reset                                                  ┊

┊ 68DDEBB5-AB46A-000008 ┊ 2025-10-03 21:01:11 ┊ S|r730xd-2   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 68E096BD-00000-000000 ┊ 2025-10-03 21:08:13 ┊ C|r730xd-1   ┊ IllegalStateException: The current scope has already been entered                  ┊
┊ 68E09BD0-14C60-000948 ┊ 2025-10-05 17:17:48 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
\
┊ 68E09BD0-14C60-001346 ┊ 2025-10-06 11:34:11 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 68E09BD0-14C60-001347 ┊ 2025-10-06 11:37:11 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 68E09DBB-AB46A-000000 ┊ 2025-10-06 11:53:49 ┊ S|r730xd-2   ┊ IOException: No route to host                                                      ┊
┊ 68E09C8F-1A0E0-000000 ┊ 2025-10-06 11:54:11 ┊ S|supermicro ┊ IOException: No route to host                                                      ┊
┊ 68E09BD0-14C60-001348 ┊ 2025-10-06 11:54:11 ┊ S|r730xd-3   ┊ IOException: No route to host                                                      ┊
┊ 68E09E77-4D03B-000000 ┊ 2025-10-06 11:54:41 ┊ S|r620       ┊ IOException: No route to host                                                      ┊
┊ 68E0980D-2EA29-000000 ┊ 2025-10-06 11:54:48 ┊ S|gigabyte   ┊ IOException: No route to host                                                      ┊
┊ 6912801A-14C60-000000 ┊ 2025-11-10 16:15:33 ┊ S|r730xd-3   ┊ StorageException: Volume group 'linstor_micron_pool' does not exist.               ┊
┊ 6912801A-14C60-000001 ┊ 2025-11-10 16:15:35 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 6912801A-14C60-000002 ┊ 2025-11-10 16:15:36 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000003 ┊ 2025-11-10 16:15:53 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000004 ┊ 2025-11-10 16:15:55 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000005 ┊ 2025-11-10 16:16:23 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000006 ┊ 2025-11-10 16:16:25 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000007 ┊ 2025-11-10 16:17:23 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000008 ┊ 2025-11-10 16:17:25 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000009 ┊ 2025-11-10 16:17:52 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 691280D8-AB46A-000000 ┊ 2025-11-10 16:18:44 ┊ S|r730xd-2   ┊ StorageException: Volume group 'linstor_micron_pool' does not exist.               ┊
┊ 691280D8-AB46A-000001 ┊ 2025-11-10 16:18:46 ┊ S|r730xd-2   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 691280D8-AB46A-000002 ┊ 2025-11-10 16:18:47 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000003 ┊ 2025-11-10 16:19:08 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000004 ┊ 2025-11-10 16:19:10 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000010 ┊ 2025-11-10 16:19:23 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000011 ┊ 2025-11-10 16:19:25 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000005 ┊ 2025-11-10 16:19:38 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000006 ┊ 2025-11-10 16:19:40 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000007 ┊ 2025-11-10 16:20:38 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000008 ┊ 2025-11-10 16:20:40 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000009 ┊ 2025-11-10 16:20:52 ┊ S|r730xd-2   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 6912801A-14C60-000012 ┊ 2025-11-10 16:20:52 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 6912801A-14C60-000013 ┊ 2025-11-10 16:21:51 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000010 ┊ 2025-11-10 16:21:51 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000014 ┊ 2025-11-10 16:21:52 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000011 ┊ 2025-11-10 16:21:52 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000015 ┊ 2025-11-10 16:21:54 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000012 ┊ 2025-11-10 16:21:54 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000000 ┊ 2025-11-10 16:21:54 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 6912801A-14C60-000016 ┊ 2025-11-10 16:21:56 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000013 ┊ 2025-11-10 16:21:56 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000001 ┊ 2025-11-10 16:21:56 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000014 ┊ 2025-11-10 16:21:58 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000017 ┊ 2025-11-10 16:21:58 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000002 ┊ 2025-11-10 16:21:58 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 69127E8F-00000-000003 ┊ 2025-11-10 16:21:58 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 6912801A-14C60-000018 ┊ 2025-11-10 16:22:01 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000004 ┊ 2025-11-10 16:22:01 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000015 ┊ 2025-11-10 16:22:01 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000019 ┊ 2025-11-10 16:22:02 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000005 ┊ 2025-11-10 16:22:02 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000016 ┊ 2025-11-10 16:22:02 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000020 ┊ 2025-11-10 16:22:05 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000006 ┊ 2025-11-10 16:22:05 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000017 ┊ 2025-11-10 16:22:05 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000021 ┊ 2025-11-10 16:22:06 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000007 ┊ 2025-11-10 16:22:06 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000018 ┊ 2025-11-10 16:22:06 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000008 ┊ 2025-11-10 16:22:06 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 6912801A-14C60-000022 ┊ 2025-11-10 16:22:08 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000009 ┊ 2025-11-10 16:22:08 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000019 ┊ 2025-11-10 16:22:08 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000023 ┊ 2025-11-10 16:22:10 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000010 ┊ 2025-11-10 16:22:10 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000020 ┊ 2025-11-10 16:22:10 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000011 ┊ 2025-11-10 16:22:10 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 6912801A-14C60-000024 ┊ 2025-11-10 16:22:11 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000012 ┊ 2025-11-10 16:22:11 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 6912801A-14C60-000025 ┊ 2025-11-10 16:22:14 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000013 ┊ 2025-11-10 16:22:14 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000021 ┊ 2025-11-10 16:22:14 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000026 ┊ 2025-11-10 16:22:15 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000014 ┊ 2025-11-10 16:22:16 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000022 ┊ 2025-11-10 16:22:16 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000027 ┊ 2025-11-10 16:22:17 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 69127E8F-00000-000015 ┊ 2025-11-10 16:22:17 ┊ C|r730xd-1   ┊ DelayedApiRcException: Exceptions have been converted to responses                 ┊
┊ 691280D8-AB46A-000023 ┊ 2025-11-10 16:22:17 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000024 ┊ 2025-11-10 16:22:38 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000025 ┊ 2025-11-10 16:22:40 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000026 ┊ 2025-11-10 16:23:15 ┊ S|r730xd-2   ┊ StorageException: Volume group 'linstor_micron_pool' does not exist.               ┊
┊ 691280D8-AB46A-000027 ┊ 2025-11-10 16:23:16 ┊ S|r730xd-2   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 691280D8-AB46A-000028 ┊ 2025-11-10 16:23:17 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000028 ┊ 2025-11-10 16:23:18 ┊ S|r730xd-3   ┊ StorageException: Volume group 'linstor_micron_pool' does not exist.               ┊
┊ 6912801A-14C60-000029 ┊ 2025-11-10 16:23:19 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 6912801A-14C60-000030 ┊ 2025-11-10 16:23:20 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000031 ┊ 2025-11-10 16:23:27 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000029 ┊ 2025-11-10 16:23:27 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000030 ┊ 2025-11-10 16:23:28 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000032 ┊ 2025-11-10 16:23:28 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000031 ┊ 2025-11-10 16:23:30 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000033 ┊ 2025-11-10 16:23:30 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000032 ┊ 2025-11-10 16:23:32 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000034 ┊ 2025-11-10 16:23:32 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000033 ┊ 2025-11-10 16:23:33 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000035 ┊ 2025-11-10 16:23:34 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000034 ┊ 2025-11-10 16:23:35 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000036 ┊ 2025-11-10 16:23:35 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000035 ┊ 2025-11-10 16:23:37 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000037 ┊ 2025-11-10 16:23:37 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000036 ┊ 2025-11-10 16:23:38 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000038 ┊ 2025-11-10 16:23:39 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000037 ┊ 2025-11-10 16:23:40 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000039 ┊ 2025-11-10 16:23:41 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000038 ┊ 2025-11-10 16:23:42 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000040 ┊ 2025-11-10 16:23:42 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000039 ┊ 2025-11-10 16:23:43 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000041 ┊ 2025-11-10 16:23:44 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000040 ┊ 2025-11-10 16:23:45 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000042 ┊ 2025-11-10 16:23:46 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000041 ┊ 2025-11-10 16:23:47 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000043 ┊ 2025-11-10 16:23:48 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000042 ┊ 2025-11-10 16:23:52 ┊ S|r730xd-2   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 6912801A-14C60-000044 ┊ 2025-11-10 16:23:52 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 6912801A-14C60-000045 ┊ 2025-11-10 16:24:23 ┊ S|r730xd-3   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...

I see that in some places it says the following… does this mean that its not finding the vg’s. ?

Does this also show that r730xd-1 is also the one with the main copy, woudl i be able to recreate the pool on r730xd-2 or r730xd-3 and have it copy from r730xd-1 ? I would like to avoid that if possible, if its possible to fix this without that i would be incredibly humble.

┊ 691280D8-AB46A-000025 ┊ 2025-11-10 16:22:40 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 691280D8-AB46A-000026 ┊ 2025-11-10 16:23:15 ┊ S|r730xd-2   ┊ StorageException: Volume group 'linstor_micron_pool' does not exist.               ┊
┊ 691280D8-AB46A-000027 ┊ 2025-11-10 16:23:16 ┊ S|r730xd-2   ┊ ApiRcException: Failed to query free space from storage pool                       ┊
┊ 691280D8-AB46A-000028 ┊ 2025-11-10 16:23:17 ┊ S|r730xd-2   ┊ StorageException: Failed to activate volume linstor_micron_pool/incus-volum...     ┊
┊ 6912801A-14C60-000028 ┊ 2025-11-10 16:23:18 ┊ S|r730xd-3   ┊ StorageException: Volume group 'linstor_micron_pool' does not exist.               ┊
┊ 6912801A-14C60-000029 ┊ 2025-11-10 16:23:19 ┊ S|r730xd-3   ┊ ApiRcException: Failed to query free space from storage pool
4

Sorry, my bad. I should have also mentioned that besides linstor err list there is also linstor err show <error-id> (whereas <error-id> is the first column, i.e. 6912801A-14C60-000045). In there there is some Java stacktraces that I do not expect to tell you much, but there is also a quite detailed “what external command was executed”, “what was its std-out / std-in”, etc.

And yes a StorageException: Volume group 'linstor_micron_pool' does not exist. usually indicates that the VG is missing. Again, if you open (show) the ErrorReport you should get more information about what failed for LINSTOR.

On the other hand, these ErrorReports are also not exactly new (2025-11-10). In that case try something like linstor rd sp <resource-name> --aux test test. This will only set a harmless Aux/test = test property on your resource-definition. The effect of this is that since the resource-definition “changed” all satellites that have a resource from this resource-definition should re-adjust their local resource. This could either fix your issue or at least re-trigger the problem, thus making it more visible and telling you where to continue to look for the root-cause. It is usually a good idea to also check err list again afterwards even if the rd sp ... command did not show anything useful.