in my linstor cluster I’ve got error reports
in details on host I’ve got :
“Error message: Volume group ‘lvm-2-1’ not found”
the name of volume “lvm-2-1” previously was on that host - but it dosn’t exists anymore.
I’ve remove this LVM a long time ago and replaced my setup with ZFS-thin storage.
on controler I’ve got :
To confirm, is the ‘Error time’ reported in the error report later than when you removed the volume group from that host?
If it is after, could you paste more of the error report output here, so we can get more details from the report about the specific error?
Are you seeing just one error report like this, or are you seeing many of them? Are they continuing to be generated? Are there specific times or during specific actions you are attempting to perform in LINSTOR that you are seeing these error reports generated?
LVM lvm-2-1 was removed from linstor config and later from host a long time ago.
I haven’t noticed this error , but few day ago I’ve problem with one of my host and I was checking all logs and I’ve found this error in linstor.
error report occurs daily. only on one of three nodes in cluster.
both other nodes previously was using lvm pools which was remove at the same time.
but problem occour only on one node : pve-2.
2026-08-18 13:19:07 SATELLITE Unable to list log directory LinStorRuntimeException
2026-08-17 13:19:07 SATELLITE Unable to list log directory LinStorRuntimeException
2026-08-16 13:19:07 SATELLITE Unable to list log directory LinStorRuntimeException
2026-08-15 13:19:07 SATELLITE Unable to list log directory LinStorRuntimeException
2026-08-14 13:19:07 SATELLITE Unable to list log directory LinStorRuntimeException
this error report from GUI linstor-contoler :
ERROR REPORT 6A7DC570-4B367-000004
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 1.34.2
Build ID: 92d55e45f88b4ef96f5ee468677f395513b2eee2
Build time: 2026-07-24T10:18:54+00:00
Error time: 2026-08-18 13:19:07
Node: pve-2
Thread: MainWorkerPool-4
Access context information
Identity: PUBLIC
Role: PUBLIC
Domain: PUBLIC
Peer: 10.10.20.29:56712/1
============================================================
Reported error:
===============
Category: RuntimeException
Class name: LinStorRuntimeException
Class canonical name: com.linbit.linstor.LinStorRuntimeException
Generated at: Method 'archiveLogsOlderThan', Source file 'StdErrorReporter.java', Line #613
Error message: Unable to list log directory
ErrorContext:
Asynchronous stage backtrace:
Error has been observed at the following site(s):
*__checkpoint ⇢ Execute single-stage API ArchiveLogs
|_ checkpoint ⇢ Fallback error handling wrapper
Original Stack Trace:
Call backtrace:
Method Native Class:Line number
archiveLogsOlderThan N com.linbit.linstor.logging.StdErrorReporter:613
Suppressed exception 1 of 1:
===============
Category: RuntimeException
Class name: OnAssemblyException
Class canonical name: reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at: Method 'archiveLogsOlderThan', Source file 'StdErrorReporter.java', Line #613
Error message:
Error has been observed at the following site(s):
*__checkpoint ⇢ Execute single-stage API ArchiveLogs
|_ checkpoint ⇢ Fallback error handling wrapper
Original Stack Trace:
Call backtrace:
Method Native Class:Line number
archiveLogsOlderThan N com.linbit.linstor.logging.StdErrorReporter:613
archiveLogDirectory N com.linbit.linstor.logging.StdErrorReporter:500
execute N com.linbit.linstor.api.protobuf.ArchiveLogs:40
executeNonReactive N com.linbit.linstor.proto.CommonMessageProcessor:546
lambda$execute$15 N com.linbit.linstor.proto.CommonMessageProcessor:517
doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:179
lambda$fluxInScope$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:101
call N reactor.core.publisher.MonoCallable:72
trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:128
subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
subscribe N reactor.core.publisher.Flux:8833
onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
request N reactor.core.publisher.Operators$ScalarSubscription:2570
onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
subscribe N reactor.core.publisher.MonoJust:55
subscribe N reactor.core.publisher.MonoDeferContextual:55
subscribe N reactor.core.publisher.InternalFluxOperator:68
subscribe N reactor.core.publisher.FluxDefer:54
subscribe N reactor.core.publisher.Flux:8848
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:430
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
drainFused N reactor.core.publisher.SinkManyUnicast:321
drain N reactor.core.publisher.SinkManyUnicast:363
tryEmitNext N reactor.core.publisher.SinkManyUnicast:239
tryEmitNext N reactor.core.publisher.SinkManySerialized:100
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:446
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:238
lambda$processMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:175
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:185
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:317
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N java.util.concurrent.ThreadPoolExecutor:1144
run N java.util.concurrent.ThreadPoolExecutor$Worker:642
run N java.lang.Thread:1583
Caused by:
==========
Category: Exception
Class name: DirectoryNotEmptyException
Class canonical name: java.nio.file.DirectoryNotEmptyException
Generated at: Method 'implDelete', Source file 'UnixFileSystemProvider.java', Line #289
Error message: /var/log/linstor-satellite/ErrorReport-68F9CDCB-4B367-000002.log
Call backtrace:
Method Native Class:Line number
implDelete N sun.nio.fs.UnixFileSystemProvider:289
delete N sun.nio.fs.AbstractFileSystemProvider:105
delete N java.nio.file.Files:1153
archiveLogsOlderThan N com.linbit.linstor.logging.StdErrorReporter:592
archiveLogDirectory N com.linbit.linstor.logging.StdErrorReporter:500
execute N com.linbit.linstor.api.protobuf.ArchiveLogs:40
executeNonReactive N com.linbit.linstor.proto.CommonMessageProcessor:546
lambda$execute$15 N com.linbit.linstor.proto.CommonMessageProcessor:517
doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:179
lambda$fluxInScope$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:101
call N reactor.core.publisher.MonoCallable:72
trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:128
subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
subscribe N reactor.core.publisher.Flux:8833
onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
request N reactor.core.publisher.Operators$ScalarSubscription:2570
onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
subscribe N reactor.core.publisher.MonoJust:55
subscribe N reactor.core.publisher.MonoDeferContextual:55
subscribe N reactor.core.publisher.InternalFluxOperator:68
subscribe N reactor.core.publisher.FluxDefer:54
subscribe N reactor.core.publisher.Flux:8848
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:430
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
drainFused N reactor.core.publisher.SinkManyUnicast:321
drain N reactor.core.publisher.SinkManyUnicast:363
tryEmitNext N reactor.core.publisher.SinkManyUnicast:239
tryEmitNext N reactor.core.publisher.SinkManySerialized:100
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:446
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:238
lambda$processMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:175
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:185
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:317
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N java.util.concurrent.ThreadPoolExecutor:1144
run N java.util.concurrent.ThreadPoolExecutor$Worker:642
run N java.lang.Thread:1583
END OF ERROR REPORT.
ERROR REPORT 68F9CDCB-4B367-000020
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 1.32.2
Build ID: 13c4b3c5e394ff6d5c04bcb3841b1d061de7606b
Build time: 2025-10-01T06:17:10+00:00
Error time: 2025-10-23 16:24:31
Node: pve-2
Thread: MainWorkerPool-4
============================================================
Reported error:
===============
Category: RuntimeException
Class name: ApiRcException
Class canonical name: com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at: Method 'getStoragePoolSpaceInfoOrError', Source file 'StltApiCallHandlerUtils.java', Line #268
Error message: Failed to query free space from storage pool
Call backtrace:
Method Native Class:Line number
getStoragePoolSpaceInfoOrError N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:268
getSpaceInfo N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:243
getSpaceInfo N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:228
executeInScope N com.linbit.linstor.api.protobuf.ReqFreeSpace:68
doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:178
lambda$fluxInScope$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:101
call N reactor.core.publisher.MonoCallable:72
trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:128
subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
subscribe N reactor.core.publisher.Flux:8833
onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
request N reactor.core.publisher.Operators$ScalarSubscription:2570
onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
subscribe N reactor.core.publisher.MonoJust:55
subscribe N reactor.core.publisher.MonoDeferContextual:55
subscribe N reactor.core.publisher.Flux:8848
trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:202
subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
subscribe N reactor.core.publisher.Flux:8833
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:430
slowPath N reactor.core.publisher.FluxArray$ArraySubscription:126
request N reactor.core.publisher.FluxArray$ArraySubscription:99
onSubscribe N reactor.core.publisher.FluxFlatMap$FlatMapMain:373
subscribe N reactor.core.publisher.FluxMerge:73
subscribe N reactor.core.publisher.Flux:8848
onComplete N reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:238
subscribe N reactor.core.publisher.FluxConcatArray:79
subscribe N reactor.core.publisher.InternalFluxOperator:68
subscribe N reactor.core.publisher.FluxDefer:54
subscribe N reactor.core.publisher.Flux:8848
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:430
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
drainFused N reactor.core.publisher.SinkManyUnicast:321
drain N reactor.core.publisher.SinkManyUnicast:363
tryEmitNext N reactor.core.publisher.SinkManyUnicast:239
tryEmitNext N reactor.core.publisher.SinkManySerialized:100
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:442
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:228
lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:165
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:185
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:264
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N java.util.concurrent.ThreadPoolExecutor:1136
run N java.util.concurrent.ThreadPoolExecutor$Worker:635
run N java.lang.Thread:840
Caused by:
==========
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'checkVgExists', Source file 'LvmUtils.java', Line #608
Error message: Volume group 'lvm-2-1' not found
ErrorContext:
Call backtrace:
Method Native Class:Line number
checkVgExists N com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:608
checkVolumeGroupEntry N com.linbit.linstor.layer.storage.utils.StorageConfigReader:68
checkThinPoolEntry N com.linbit.linstor.layer.storage.utils.StorageConfigReader:94
checkConfig N com.linbit.linstor.layer.storage.lvm.LvmThinProvider:518
checkStorPool N com.linbit.linstor.layer.storage.StorageLayer:451
getSpaceInfo N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1530
getSpaceInfo N com.linbit.linstor.core.devmgr.DeviceManagerImpl:1783
getStoragePoolSpaceInfo N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:280
getStoragePoolSpaceInfoOrError N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:255
getSpaceInfo N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:243
getSpaceInfo N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:228
executeInScope N com.linbit.linstor.api.protobuf.ReqFreeSpace:68
doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:178
lambda$fluxInScope$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:101
call N reactor.core.publisher.MonoCallable:72
trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:128
subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
subscribe N reactor.core.publisher.Flux:8833
onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:196
request N reactor.core.publisher.Operators$ScalarSubscription:2570
onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
subscribe N reactor.core.publisher.MonoJust:55
subscribe N reactor.core.publisher.MonoDeferContextual:55
subscribe N reactor.core.publisher.Flux:8848
trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:202
subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
subscribe N reactor.core.publisher.Flux:8833
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:430
slowPath N reactor.core.publisher.FluxArray$ArraySubscription:126
request N reactor.core.publisher.FluxArray$ArraySubscription:99
onSubscribe N reactor.core.publisher.FluxFlatMap$FlatMapMain:373
subscribe N reactor.core.publisher.FluxMerge:73
subscribe N reactor.core.publisher.Flux:8848
onComplete N reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:238
subscribe N reactor.core.publisher.FluxConcatArray:79
subscribe N reactor.core.publisher.InternalFluxOperator:68
subscribe N reactor.core.publisher.FluxDefer:54
subscribe N reactor.core.publisher.Flux:8848
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:430
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
drainFused N reactor.core.publisher.SinkManyUnicast:321
drain N reactor.core.publisher.SinkManyUnicast:363
tryEmitNext N reactor.core.publisher.SinkManyUnicast:239
tryEmitNext N reactor.core.publisher.SinkManySerialized:100
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:442
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:228
lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:165
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:185
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:446
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:533
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:264
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N java.util.concurrent.ThreadPoolExecutor:1136
run N java.util.concurrent.ThreadPoolExecutor$Worker:635
run N java.lang.Thread:840
END OF ERROR REPORT.
Just a quick question before we dive in too deep here:
LVM lvm-2-1 was removed from linstor config and later from host a long time ago.
What exactly do you mean with “was removed from linstor config”? The point I am asking is to make sure that you have run at some point a linstor sp d pve-2 lvm-2-1 or not. You can easily verify this by showing linstor sp l. If that still contains your lvm-2-1 entry, LINSTOR still assumes that LVM VG “lvm-2-1” still exists. In this case I’d say that LINSTOR is right to complain about the missing LVM VG.
Also I want to highlight that the two shown ErrorReports do not seem to be related. The Volume group 'lvm-2-1' not found one comes from a 1.32.2 satellite, while the Unable to list log directory comes from a 1.34.2 and have nothing to do with anything LVM related.
hi.
lvm-2-1 was previous added/configured on that node and was added to storage pool in linstor.
but I’ve removed it from storage pool - using GUI.
currently on node pve-2 I’ve got only :
there is no “lvm*” pool or storage pool in my cluster : root@linstor-controller:~# linstor sp l | grep lvm | wc -l 0
regarding version mismatch - you are right. I didn’t notice this.
but.
error report is fresh ( yestarday )
root@linstor-controller:/var/lib/linstor# linstor err l | grep 6A58DF83-4B367-000006 │ 6A58DF83-4B367-000006 │ 2026-07-19 13:34:15 │ S|pve-2 │ LinStorRuntimeException: Unable to list log directory │
root@linstor-controller:/var/lib/linstor# linstor err s 6A58DF83-4B367-000006 | grep "/var"Error message: /var/log/linstor-satellite/ErrorReport-68F9CDCB-4B367-000002.log
but on node pve-2 error details are old : root@pve-2:~# ls -al /var/log/linstor-satellite/ErrorReport-68F9CDCB-4B367-000002.log/agent.4132 -rw-r--r-- 1 root root 12792 2025-10-23 /var/log/linstor-satellite/ErrorReport-68F9CDCB-4B367-000002.log/agent.4132
should I remove that old file :
/var/log/linstor-satellite/ErrorReport-68F9CDCB-4B367-000002.log/agent.4132
?
currently I’ve got linstor with latest version on all nodes and controller.
Thanks and sorry that I previously missed a few details:
The LVM related error is almost a year old. I guess we can safely ignore that.
The second error report is a bit more interesting. I missed that /var/log/linstor-satellite/ErrorReport-68F9CDCB-4B367-000002.log is now a directory, not a plain file. This directory contains a file called agent.4132, with the size of 12792 bytes. This could easily had the same content you showed above based on the file-size.
So what I would be interested in is who/what created this directory + the agent.4132 file? Any ideas?
Btw, in post 3 you said "here are the logs from ...002.log, but the shown ErrorReport has a slightly different ID: -000020. It is likely that 000002 and 000020 had the identical error message, so nothing to worry about, but that only means that even when you posted #3 the ...002.log was no longer a file but rather a directory, in fact the other report you show dates back to 2026-07-19. So whatever created this agent.4132 file was probably doing that at that date or a day earlier.
If you want, you can verify my guess that the agent.4132 file contains the same content as the “Volume group ‘lvm-2-1’ not found” ErrorReport. This way we would at least know that something just copied the actual ErrorReport - for whatever reason.
It could also make sense for you to check if there are other “file looking directories” in /var/log/linstor-satellite/ since those could also create trouble in the future.
Anyway if you do not care what created this file, the solution is simple: Delete manually the directory (including the agent.4132 file) and LINSTOR should deal with the rest.
The issue itself is LINSTORs daily check if old ErrorReports can be archived or not. It apparently did archive old ErrorReports and later tried to deleted them using a simple Files.delete(file) call which deliberately fails if the passed file is not a file but a non-empty directory, which is exactly what happened in your case. So deleting the directory should allow LINSTOR to properly archive old logs the next time.
We will try to make the archiving task in LINSTOR more robust so it does not fail in such cases.
hi.thx for help - I’ve removed old error log.
this error log was indeed directory not a file.
the rest of error reports are files only this one was a directory.