# Gateway failure to create iscsi target **URL:** \ **Category:** General\ **Created:** [July 22, 2024, 11:32pm UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189 "2024-07-22T23:32:07Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![briandlees](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@briandlees](https://forums.linbit.com/u/briandlees)\ **Post date:** [July 22, 2024, 11:32pm UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189/1 "2024-07-22T23:32:07Z")
I have been trying to set up a 3 node SDS using linstor-satellite, linstor-controller and linstor-client in containers. This seemed harder to achieve than it should have been, but that may just have been me! Then I came to try and create an iscsi target and got the following errors (ips and nodenames changed); exit code 32 means the devices do not exist. Any suggestions about what is causing this : _node003:~ # linstor-gateway iscsi create iqn.2010-10.x.y.z:main-target 192.168.a.b/24 100M --implementation “lio-t” --resource-group “gateway-grp”_ _ERRO[0009] failed to create iscsi resource: failed to create linstor resource: failed to autoplace resources:_ Message: ‘Successfully set property key(s): StorPoolName’ next error: _Message: ‘Successfully set property key(s): StorPoolName’ next error:_ Message: ‘Successfully set property key(s): StorPoolName’ next error: Message: ‘Resource ‘main-target’ successfully autoplaced on 3 nodes’; Details: ‘Used nodes (storage pool name): ‘node001 (gateway\_pool)’, ‘node002 (gateway\_pool)’, ‘node003 (gateway\_pool)’’ next error: Message: ‘Updated main-target DRBD auto verify algorithm to ‘crc32c’’ next error: Message: ‘(node003) Volume number 0 of resource ‘main-target’ [LVM-Thin] created’ next error: Message: ‘(node003) Volume number 1 of resource ‘main-target’ [LVM-Thin] created’ next error: Message: ‘(node003) Resource ‘main-target’ [DRBD] adjusted.’ next error: Message: ‘Created resource ‘main-target’ on ‘node003’’ next error: Message: ‘(node002) Volume number 0 of resource ‘main-target’ [LVM-Thin] created’ next error: Message: ‘(node002) Volume number 1 of resource ‘main-target’ [LVM-Thin] created’ next error: Message: ‘(node002) Resource ‘main-target’ [DRBD] adjusted.’ next error: Message: ‘Created resource ‘main-target’ on ‘node002’’ next error: Message: ‘(node001) Volume number 0 of resource ‘main-target’ [LVM-Thin] created’ next error: Message: ‘(node001) Volume number 1 of resource ‘main-target’ [LVM-Thin] created’ next error: Message: ‘(node001) Resource ‘main-target’ [DRBD] adjusted.’ next error: Message: ‘Created resource ‘main-target’ on ‘node001’’ next error: Message: ‘Resource ‘main-target’ on ‘node003’ ready’; Details: ‘Auto-placing resource: main-target’ next error: Message: ‘Resource ‘main-target’ on ‘node002’ ready’; Details: ‘Auto-placing resource: main-target’ next error: Message: ‘Resource ‘main-target’ on ‘node001’ ready’; Details: ‘Auto-placing resource: main-target’ next error: Message: ‘(Node: ‘node003’) Failed execute lsblk’; Details: 'Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1001’ returned with exitcode 32. Standard out: Error message: lsblk: /dev/drbd1001: not a block device '; Reports: ‘[669E8925-10C50-000152]’ next error: Message: ‘(Node: ‘node002’) Failed execute lsblk’; Details: 'Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1001’ returned with exitcode 32.\* Standard out: Error message: lsblk: /dev/drbd1001: not a block device '; Reports: ‘[669E89A1-4CFFD-000127]’ next error: Message: ‘(Node: ‘node001’) Failed execute lsblk’; Details: 'Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1001’ returned with exitcode 32. Standard out: Error message: lsblk: /dev/drbd1001: not a block device '; Reports: ‘[669E8A5A-09CA1-000167]’ next error: Message: ‘Modification of resource definitions for resource definition ‘main-target’ failed due to an unknown exception.’; Details: ‘Resource definitions for Resource definition: main-target’; Reports: ‘[669E9D80-00000-000004]’ Other information: linstor-client-1.23.0 linstor-server-1.28.0 linstor-gateway-1.5.0 ---
**Author:** ![kermat](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/kermat/32/21_2.png) [@kermat](https://forums.linbit.com/u/kermat)\ **Post date:** [July 23, 2024, 9:44pm UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189/2 "2024-07-23T21:44:30Z")
Are you able to create resources successfully using the LINSTOR client? ```bash linstor resource-group spawn-resources gateway-grp test-res-0 1G ``` ---
**Author:** ![briandlees](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@briandlees](https://forums.linbit.com/u/briandlees)\ **Post date:** [July 24, 2024, 9:46am UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189/3 "2024-07-24T09:46:50Z")
Excellent suggestion and interestingly, no. I get the same error messages and also same result using the default group, so this rules out linstor-gateway as the issue. I was able previously to create resources but have changed some of the container mounts since then. I’ll investigate further. Thanks for the pointer; sometimes after hours of looking a fresh perspective is invaluable. I need to do some more tests anyway as I’m having issues getting the mounts right in container. I want to have an HA setup for linstor-controller but the linstor\_db resource is created in /var/lib/linstor _inside_ the container. I have tried mounting a DRBD replicated volume on /var/lib/linstor and making this available inside the container but I’ve not got this right yet. ---
**Author:** ![briandlees](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@briandlees](https://forums.linbit.com/u/briandlees)\ **Post date:** [July 24, 2024, 2:49pm UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189/4 "2024-07-24T14:49:38Z")
So I built everything again from scratch. Everything is fine until I tried to spawn resources and then got the same error messages. However, doing this manually (rather than when I was using linstor-gateway) I can see that the resources were actually created and the DRBD LVM is working correctly. I then promoted node 3 to primary, formatted it, and mounted it successfully, showing that the resource is correctly built from a DRBD point of view. Frankly I have spent too much time for this already and I’m going to build things manually; the point of tools after all is too speed things up! Data below if you are interested: LINSTOR ==\> resource-group spawn-resources linstor-db-grp linstor-db 200M INFO: Resource-group doesn’t have any volume-groups, automatically assume partial mode. SUCCESS: Volume definition with number ‘0’ successfully created in resource definition ‘linstor-db’. SUCCESS: Description: New resource definition ‘linstor-db’ created. Details: Resource definition ‘linstor-db’ UUID is: bda67ca0-f1b7-4517-a6f1-3ad238a7f867 SUCCESS: Successfully set property key(s): StorPoolName SUCCESS: Successfully set property key(s): StorPoolName SUCCESS: Successfully set property key(s): StorPoolName SUCCESS: Description: Resource ‘linstor-db’ successfully autoplaced on 3 nodes Details: Used nodes (storage pool name): ‘node001 (acl\_pool)’, ‘node002 (acl\_pool)’, ‘node003 (acl\_pool)’ INFO: Updated linstor-db DRBD auto verify algorithm to ‘crc32c’ INFO: Resource-definition property ‘DrbdOptions/Resource/quorum’ updated from undefined to ‘majority’ by auto-quorum INFO: Resource-definition property ‘DrbdOptions/Resource/on-no-quorum’ updated from undefined to ‘io-error’ by auto-quorum SUCCESS: (node003) Volume number 0 of resource ‘linstor-db’ [LVM-Thin] created SUCCESS: (node003) Resource ‘linstor-db’ [DRBD] adjusted. SUCCESS: Created resource ‘linstor-db’ on ‘node003’ SUCCESS: (node002) Volume number 0 of resource ‘linstor-db’ [LVM-Thin] created SUCCESS: (node002) Resource ‘linstor-db’ [DRBD] adjusted. SUCCESS: Created resource ‘linstor-db’ on ‘node002’ SUCCESS: (node001) Volume number 0 of resource ‘linstor-db’ [LVM-Thin] created SUCCESS: (node001) Resource ‘linstor-db’ [DRBD] adjusted. SUCCESS: Created resource ‘linstor-db’ on ‘node001’ SUCCESS: Description: Resource ‘linstor-db’ on ‘node001’ ready Details: Resource group: linstor-db-grp SUCCESS: Description: Resource ‘linstor-db’ on ‘node002’ ready Details: Resource group: linstor-db-grp SUCCESS: Description: Resource ‘linstor-db’ on ‘node003’ ready Details: Resource group: linstor-db-grp ERROR: Description: (Node: ‘node003’) Failed execute lsblk Details: Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000’ returned with exitcode 32. ``` Standard out: Error message: lsblk: /dev/drbd1000: not a block device ``` Show reports: linstor error-reports show 66A0FCF3-10C50-000001 ERROR: Description: (Node: ‘node002’) Failed execute lsblk Details: Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000’ returned with exitcode 32. ``` Standard out: Error message: lsblk: /dev/drbd1000: not a block device ``` Show reports: linstor error-reports show 66A0F82B-4CFFD-000011 ERROR: Description: (Node: ‘node001’) Failed execute lsblk Details: Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000’ returned with exitcode 32. ``` Standard out: Error message: lsblk: /dev/drbd1000: not a block device ``` Show reports: linstor error-reports show 66A0FCC3-09CA1-000003 ERROR: Description: Modification of resource definitions for resource definition ‘linstor-db’ failed due to an unknown exception. Details: Resource definitions for Resource definition: linstor-db Show reports: linstor error-reports show 66A1015C-00000-000001 LINSTOR ==\> resource list ╭────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊ ╞════════════════════════════════════════════════════════════════════════════════╡ ┊ linstor-db ┊ node001 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊ ┊ linstor-db ┊ node002 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊ ┊ linstor-db ┊ node003 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊ ╰────────────────────────────────────────────────────────────────────────────────╯ node003:~ # drbdsetup status linstor-db role:Secondary disk:UpToDate node001 role:Secondary peer-disk:UpToDate node002 role:Secondary peer-disk:UpToDate ---
**Author:** ![kermat](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/kermat/32/21_2.png) [@kermat](https://forums.linbit.com/u/kermat)\ **Post date:** [July 24, 2024, 3:34pm UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189/5 "2024-07-24T15:34:23Z")
How are you running the LINSTOR satellite containers? Are you running them with the `--privileged` flag? The containers need privilege to access the host’s block devices: ```bash [root@swarm-0 ~]# docker run --rm -it --name test ubuntu sh -c "lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/vdb" lsblk: /dev/vdb: not a block device [root@swarm-0 ~]# docker run --privileged --rm -it --name test ubuntu sh -c "lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/vdb" NAME="/dev/vdb" SIZE="8589934592" PKNAME="" KNAME="/dev/vdb" ROTA="1" TYPE="disk" FSTYPE="" MAJ:MIN="253:16" MODEL="" SERIAL="" WWN="" STATE="" DISC-GRAN="512" ``` ---
**Author:** ![briandlees](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@briandlees](https://forums.linbit.com/u/briandlees)\ **Post date:** [July 25, 2024, 10:20am UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189/6 "2024-07-25T10:20:09Z")
Well that was fascinating. The short answer is yes I run the container in privileged mode: podman run -d --name=linstor-satellite --net=host --privileged -v=/var/lib/linstor.d:/var/lib/linstor.d -v=/etc/linstor:/etc/linstor localhost/linstor-satellite **HOWEVER…** I did some tests on the first 2 nodes which were very confusing and here is the clean output from the third node: Last login: Thu Jul 25 10:22:43 2024 from 192.168.2.91 node001:~ # podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c62517d2aad5 localhost/linstor-satellite:latest startSatellite 21 hours ago Up 21 hours linstor-satellite node001:~ # podman exec linstor-satellite bash -c “lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000” lsblk: /dev/drbd1000: not a block device node001:~ # podman restart linstor-satellite linstor-satellite node001:~ # podman exec linstor-satellite bash -c “lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000” NAME=“/dev/drbd1000” SIZE=“213823488” PKNAME=“” KNAME=“/dev/drbd1000” ROTA=“1” TYPE=“disk” FSTYPE=“” MAJ:MIN=“147:1000” MODEL=“” SERIAL=“” WWN=“” STATE=“” DISC-GRAN=“512” node001:~ # Do you have any explanation because I don’t? The command failed then a simple restart and it works. It is the same container just restarted. Linstor-controller auto-evicted all the nodes but a restart of linstor-controller fixed that. The problem is that I still get the same error message even though the resources are created correctly; the command: LINSTOR ==\> resource-group spawn-resources test-grp test-res 10M still eventually comes up with e.g.: ERROR: Description: (Node: ‘node003’) Failed execute lsblk Details: Command ‘lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1001’ returned with exitcode 32. ``` Standard out: Error message: lsblk: /dev/drbd1001: not a block device ``` Show reports: linstor error-reports show 66A2225A-10C50-000006 LINSTOR ==\> node list ╭───────────────────────────────────────────────────────╮ ┊ Node ┊ NodeType ┊ Addresses ┊ State ┊ ╞═══════════════════════════════════════════════════════╡ ┊ acl001 ┊ COMBINED ┊ 10.10.0.100:3366 (PLAIN) ┊ Online ┊ ┊ acl002 ┊ COMBINED ┊ 10.10.0.200:3366 (PLAIN) ┊ Online ┊ ┊ acl003 ┊ COMBINED ┊ 10.10.0.150:3366 (PLAIN) ┊ Online ┊ ╰───────────────────────────────────────────────────────╯ LINSTOR ==\> resource list ╭────────────────────────────────────────────────────────────────────────────────╮ ┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊ ╞════════════════════════════════════════════════════════════════════════════════╡ ┊ linstor-db ┊ acl001 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊ ┊ linstor-db ┊ acl002 ┊ 7000 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊ ┊ linstor-db ┊ acl003 ┊ 7000 ┊ InUse ┊ Ok ┊ UpToDate ┊ 2024-07-24 13:41:04 ┊ ┊ test-res ┊ acl001 ┊ 7001 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-25 10:10:17 ┊ ┊ test-res ┊ acl002 ┊ 7001 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-25 10:10:17 ┊ ┊ test-res ┊ acl003 ┊ 7001 ┊ Unused ┊ Ok ┊ UpToDate ┊ 2024-07-25 10:10:17 ┊ ╰────────────────────────────────────────────────────────────────────────────────╯ I immediately re-ran the test: LINSTOR ==\> exit node003:~ # podman exec linstor-satellite bash -c “lsblk -P -b --paths -o NAME,SIZE,PKNAME,KNAME,ROTA,TYPE,FSTYPE,MAJ:MIN,MODEL,SERIAL,WWN,STATE,DISC-GRAN /dev/drbd1000” NAME=“/dev/drbd1000” SIZE=“213823488” PKNAME=“” KNAME=“/dev/drbd1000” ROTA=“1” TYPE=“disk” FSTYPE=“” MAJ:MIN=“147:1000” MODEL=“” SERIAL=“” WWN=“” STATE=“” DISC-GRAN=“512” node003:~ # ---
**Author:** ![kermat](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/kermat/32/21_2.png) [@kermat](https://forums.linbit.com/u/kermat)\ **Post date:** [July 25, 2024, 3:19pm UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189/7 "2024-07-25T15:19:44Z")
> [@briandlees](#): > > Do you have any explanation because I don’t? The command failed then a simple restart and it works. It is the same container just restarted. Linstor-controller auto-evicted all the nodes but a restart of linstor-controller fixed that. Unfortunately, I don’t have an explanation for the intermittent success/fails. Perhaps try binding ` -v /dev:/dev` to the satellite containers? If LINSTOR satellites are getting evicted from the cluster, that means they lost connectivity to the controller for \>=60 minutes. Are you publishing the LINSTOR controller’s port ` -p 3370:3370` on the controller container? ---
**Author:** ![briandlees](https://avatars.discourse-cdn.com/v4/letter/b/258eb7/32.png) [@briandlees](https://forums.linbit.com/u/briandlees)\ **Post date:** [August 1, 2024, 11:07am UTC](https://forums.linbit.com/t/gateway-failure-to-create-iscsi-target/189/8 "2024-08-01T11:07:43Z")
Yes I’m publishing the port: podman run -d --name=linstor-controller -p 3370:3370 -v=/var/lib/linstor:/var/lib/linstor -v=/etc/linstor:/etc/linstor -v=/var/lib/linstor.d:/var/lib/linstor.d localhost/linstor-controller Thanks for all your help but I’ve given up on this. It’s taken far too much time. I got things working by ignoring the errors and then after running an update on my servers I started getting a different error message which prevented creating the containers. Running the above command now gives this error: Error: runc: runc create failed: unable to start container process: exec: “/bin/bash”: stat /bin/bash: permission denied: OCI permission denied I don’t have time to spend on this so will go back to my old scripts/templates to create resources.