Gateway failure to create iscsi target

General
1

I have been trying to set up a 3 node SDS using linstor-satellite, linstor-controller and linstor-client in containers. This seemed harder to achieve than it should have been, but that may just have been me! Then I came to try and create an iscsi target and got the following errors (ips and nodenames changed); exit code 32 means the devices do not exist. Any suggestions about what is causing this :

node003:~ # linstor-gateway iscsi create iqn.2010-10.x.y.z:main-target 192.168.a.b/24 100M --implementation “lio-t” --resource-group “gateway-grp”

ERRO[0009] failed to create iscsi resource: failed to create linstor resource: failed to autoplace resources:

Message: ‘Successfully set property key(s): StorPoolName’ next error:
Message: ‘Successfully set property key(s): StorPoolName’ next error:
Message: ‘Successfully set property key(s): StorPoolName’ next error:
Message: ‘Resource ‘main-target’ successfully autoplaced on 3 nodes’; Details: ‘Used nodes (storage pool name): ‘node001 (gateway_pool)’, ‘node002 (gateway_pool)’, ‘node003 (gateway_pool)’’ next error:
Message: ‘Updated main-target DRBD auto verify algorithm to ‘crc32c’’ next error:
Message: ‘(node003) Volume number 0 of resource ‘main-target’ [LVM-Thin] created’ next error:
Message: ‘(node003) Volume number 1 of resource ‘main-target’ [LVM-Thin] created’ next error:
Message: ‘(node003) Resource ‘main-target’ [DRBD] adjusted.’ next error:
Message: ‘Created resource ‘main-target’ on ‘node003’’ next error:
Message: ‘(node002) Volume number 0 of resource ‘main-target’ [LVM-Thin] created’ next error:
Message: ‘(node002) Volume number 1 of resource ‘main-target’ [LVM-Thin] created’ next error:
Message: ‘(node002) Resource ‘main-target’ [DRBD] adjusted.’ next error:
Message: ‘Created resource ‘main-target’ on ‘node002’’ next error:
Message: ‘(node001) Volume number 0 of resource ‘main-target’ [LVM-Thin] created’ next error:
Message: ‘(node001) Volume number 1 of resource ‘main-target’ [LVM-Thin] created’ next error:
Message: ‘(node001) Resource ‘main-target’ [DRBD] adjusted.’ next error:
Message: ‘Created resource ‘main-target’ on ‘node001’’ next error:
Message: ‘Resource ‘main-target’ on ‘node003’ ready’; Details: ‘Auto-placing resource: main-target’ next error:
Message: ‘Resource ‘main-target’ on ‘node002’ ready’; Details: ‘Auto-placing resource: main-target’ next error:
Message: ‘Resource ‘main-target’ on ‘node001’ ready’; Details: ‘Auto-placing resource: main-target’ next error:

Message: ‘(Node: ‘node003’) Failed execute lsblk’; Details: 'Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1001’ returned with exitcode 32.

Standard out:

Error message:
lsblk: /dev/drbd1001: not a block device

'; Reports: ‘[669E8925-10C50-000152]’ next error: Message: ‘(Node: ‘node002’) Failed execute lsblk’; Details: 'Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1001’ returned with exitcode 32.*

Standard out:

Error message:
lsblk: /dev/drbd1001: not a block device

'; Reports: ‘[669E89A1-4CFFD-000127]’ next error: Message: ‘(Node: ‘node001’) Failed execute lsblk’; Details: 'Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1001’ returned with exitcode 32.

Standard out:

Error message:
lsblk: /dev/drbd1001: not a block device

'; Reports: ‘[669E8A5A-09CA1-000167]’ next error: Message: ‘Modification of resource definitions for resource definition ‘main-target’ failed due to an unknown exception.’; Details: ‘Resource definitions for Resource definition: main-target’; Reports: ‘[669E9D80-00000-000004]’

Other information:
linstor-client-1.23.0
linstor-server-1.28.0
linstor-gateway-1.5.0

2

Are you able to create resources successfully using the LINSTOR client?

linstor resource-group spawn-resources gateway-grp test-res-0 1G
3

Excellent suggestion and interestingly, no. I get the same error messages and also same result using the default group, so this rules out linstor-gateway as the issue. I was able previously to create resources but have changed some of the container mounts since then. I’ll investigate further.

Thanks for the pointer; sometimes after hours of looking a fresh perspective is invaluable. I need to do some more tests anyway as I’m having issues getting the mounts right in container. I want to have an HA setup for linstor-controller but the linstor_db resource is created in /var/lib/linstor inside the container. I have tried mounting a DRBD replicated volume on /var/lib/linstor and making this available inside the container but I’ve not got this right yet.

4

So I built everything again from scratch. Everything is fine until I tried to spawn resources and then got the same error messages. However, doing this manually (rather than when I was using linstor-gateway) I can see that the resources were actually created and the DRBD LVM is working correctly.

I then promoted node 3 to primary, formatted it, and mounted it successfully, showing that the resource is correctly built from a DRBD point of view.

Frankly I have spent too much time for this already and I’m going to build things manually; the point of tools after all is too speed things up!

Data below if you are interested:

LINSTOR ==> resource-group spawn-resources linstor-db-grp linstor-db 200M
INFO:
Resource-group doesn’t have any volume-groups, automatically assume partial mode.
SUCCESS:
Volume definition with number ‘0’ successfully created in resource definition ‘linstor-db’.
SUCCESS:
Description:
New resource definition ‘linstor-db’ created.
Details:
Resource definition ‘linstor-db’ UUID is: bda67ca0-f1b7-4517-a6f1-3ad238a7f867
SUCCESS:
Successfully set property key(s): StorPoolName
SUCCESS:
Successfully set property key(s): StorPoolName
SUCCESS:
Successfully set property key(s): StorPoolName
SUCCESS:
Description:
Resource ‘linstor-db’ successfully autoplaced on 3 nodes
Details:
Used nodes (storage pool name): ‘node001 (acl_pool)’, ‘node002 (acl_pool)’, ‘node003 (acl_pool)’
INFO:
Updated linstor-db DRBD auto verify algorithm to ‘crc32c’
INFO:
Resource-definition property ‘DrbdOptions/Resource/quorum’ updated from undefined to ‘majority’ by auto-quorum
INFO:
Resource-definition property ‘DrbdOptions/Resource/on-no-quorum’ updated from undefined to ‘io-error’ by auto-quorum
SUCCESS:
(node003) Volume number 0 of resource ‘linstor-db’ [LVM-Thin] created
SUCCESS:
(node003) Resource ‘linstor-db’ [DRBD] adjusted.
SUCCESS:
Created resource ‘linstor-db’ on ‘node003’
SUCCESS:
(node002) Volume number 0 of resource ‘linstor-db’ [LVM-Thin] created
SUCCESS:
(node002) Resource ‘linstor-db’ [DRBD] adjusted.
SUCCESS:
Created resource ‘linstor-db’ on ‘node002’
SUCCESS:
(node001) Volume number 0 of resource ‘linstor-db’ [LVM-Thin] created
SUCCESS:
(node001) Resource ‘linstor-db’ [DRBD] adjusted.
SUCCESS:
Created resource ‘linstor-db’ on ‘node001’
SUCCESS:
Description:
Resource ‘linstor-db’ on ‘node001’ ready
Details:
Resource group: linstor-db-grp
SUCCESS:
Description:
Resource ‘linstor-db’ on ‘node002’ ready
Details:
Resource group: linstor-db-grp
SUCCESS:
Description:
Resource ‘linstor-db’ on ‘node003’ ready
Details:
Resource group: linstor-db-grp
ERROR:
Description:
(Node: ‘node003’) Failed execute lsblk
Details:
Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000’ returned with exitcode 32.

Standard out:


Error message:
lsblk: /dev/drbd1000: not a block device

Show reports:
linstor error-reports show 66A0FCF3-10C50-000001
ERROR:
Description:
(Node: ‘node002’) Failed execute lsblk
Details:
Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000’ returned with exitcode 32.

Standard out:


Error message:
lsblk: /dev/drbd1000: not a block device

Show reports:
linstor error-reports show 66A0F82B-4CFFD-000011
ERROR:
Description:
(Node: ‘node001’) Failed execute lsblk
Details:
Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000’ returned with exitcode 32.

Standard out:


Error message:
lsblk: /dev/drbd1000: not a block device

Show reports:
linstor error-reports show 66A0FCC3-09CA1-000003
ERROR:
Description:
Modification of resource definitions for resource definition ‘linstor-db’ failed due to an unknown exception.
Details:
Resource definitions for Resource definition: linstor-db
Show reports:
linstor error-reports show 66A1015C-00000-000001

LINSTOR ==> resource list
╭────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞════════════════════════════════════════════════════════════════════════════════╡
┊ linstor-db ┊ node001 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊
┊ linstor-db ┊ node002 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊
┊ linstor-db ┊ node003 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊
╰────────────────────────────────────────────────────────────────────────────────╯

node003:~ # drbdsetup status
linstor-db role:Secondary
disk:UpToDate
node001 role:Secondary
peer-disk:UpToDate
node002 role:Secondary
peer-disk:UpToDate

5

How are you running the LINSTOR satellite containers? Are you running them with the --privileged flag?

The containers need privilege to access the host’s block devices:

[root@swarm-0 ~]# docker run --rm -it --name test ubuntu sh -c "lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/vdb"
lsblk: /dev/vdb: not a block device

[root@swarm-0 ~]# docker run --privileged --rm -it --name test ubuntu sh -c "lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/vdb"
NAME="/dev/vdb" SIZE="8589934592" PKNAME="" KNAME="/dev/vdb" ROTA="1" TYPE="disk" FSTYPE="" MAJ:MIN="253:16" MODEL="" SERIAL="" WWN="" STATE="" DISC-GRAN="512"
6

Well that was fascinating. The short answer is yes I run the container in privileged mode:

podman run -d --name=linstor-satellite --net=host --privileged -v=/var/lib/linstor.d:/var/lib/linstor.d -v=/etc/linstor:/etc/linstor localhost/linstor-satellite

HOWEVER…

I did some tests on the first 2 nodes which were very confusing and here is the clean output from the third node:

Last login: Thu Jul 25 10:22:43 2024 from 192.168.2.91
node001:~ # podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c62517d2aad5 localhost/linstor-satellite:latest startSatellite 21 hours ago Up 21 hours linstor-satellite
node001:~ # podman exec linstor-satellite bash -c “lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000”
lsblk: /dev/drbd1000: not a block device
node001:~ # podman restart linstor-satellite
linstor-satellite
node001:~ # podman exec linstor-satellite bash -c “lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000”
NAME=“/dev/drbd1000” SIZE=“213823488” PKNAME=“” KNAME=“/dev/drbd1000” ROTA=“1” TYPE=“disk” FSTYPE=“” MAJ:MIN=“147:1000” MODEL=“” SERIAL=“” WWN=“” STATE=“” DISC-GRAN=“512”
node001:~ #

Do you have any explanation because I don’t? The command failed then a simple restart and it works. It is the same container just restarted. Linstor-controller auto-evicted all the nodes but a restart of linstor-controller fixed that.

The problem is that I still get the same error message even though the resources are created correctly; the command:

LINSTOR ==> resource-group spawn-resources test-grp test-res 10M

still eventually comes up with e.g.:

ERROR:
Description:
(Node: ‘node003’) Failed execute lsblk
Details:
Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1001’ returned with exitcode 32.

Standard out:


Error message:
lsblk: /dev/drbd1001: not a block device

Show reports:
linstor error-reports show 66A2225A-10C50-000006

LINSTOR ==> node list
╭───────────────────────────────────────────────────────╮
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞═══════════════════════════════════════════════════════╡
┊ acl001 ┊ COMBINED ┊ 10.10.0.100:3366 (PLAIN) ┊ Online ┊
┊ acl002 ┊ COMBINED ┊ 10.10.0.200:3366 (PLAIN) ┊ Online ┊
┊ acl003 ┊ COMBINED ┊ 10.10.0.150:3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────╯

LINSTOR ==> resource list
╭────────────────────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞════════════════════════════════════════════════════════════════════════════════╡
┊ linstor-db ┊ acl001 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊
┊ linstor-db ┊ acl002 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊
┊ linstor-db ┊ acl003 ┊ 7000 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊
┊ test-res ┊ acl001 ┊ 7001 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-25 10:10:17 ┊
┊ test-res ┊ acl002 ┊ 7001 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-25 10:10:17 ┊
┊ test-res ┊ acl003 ┊ 7001 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-25 10:10:17 ┊
╰────────────────────────────────────────────────────────────────────────────────╯

I immediately re-ran the test:

LINSTOR ==> exit
node003:~ # podman exec linstor-satellite bash -c “lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000”
NAME=“/dev/drbd1000” SIZE=“213823488” PKNAME=“” KNAME=“/dev/drbd1000” ROTA=“1” TYPE=“disk” FSTYPE=“” MAJ:MIN=“147:1000” MODEL=“” SERIAL=“” WWN=“” STATE=“” DISC-GRAN=“512”
node003:~ #

7

Unfortunately, I don’t have an explanation for the intermittent success/fails. Perhaps try binding -v /dev:/dev to the satellite containers?

If LINSTOR satellites are getting evicted from the cluster, that means they lost connectivity to the controller for >=60 minutes. Are you publishing the LINSTOR controller’s port -p 3370:3370 on the controller container?

8

Yes I’m publishing the port:

podman run -d --name=linstor-controller -p 3370:3370 -v=/var/lib/linstor:/var/lib/linstor -v=/etc/linstor:/etc/linstor -v=/var/lib/linstor.d:/var/lib/linstor.d localhost/linstor-controller

Thanks for all your help but I’ve given up on this. It’s taken far too much time. I got things working by ignoring the errors and then after running an update on my servers I started getting a different error message which prevented creating the containers. Running the above command now gives this error:

Error: runc: runc create failed: unable to start container process: exec: “/bin/bash”: stat /bin/bash: permission denied: OCI permission denied

I don’t have time to spend on this so will go back to my old scripts/templates to create resources.

1 Like