# Trying to Move VM to LINSTOR Storage: storage migration failed: Source and target image have different sizes

**URL:** <https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213>\
**Category:** General\
**Created:** [August 1, 2024, 11:58pm UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213 "2024-08-01T23:58:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![fx991](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/fx991/32/61_2.png) [@fx991](https://forums.linbit.com/u/fx991)\
**Post date:** [August 1, 2024, 11:58pm UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/1 "2024-08-01T23:58:43Z")

</div>

When I try to move a storage of the same VM, I get this error:

```auto
create full clone of drive scsi0 (lvmlocal01:vm-100-disk-1)

NOTICE
  Trying to create diskful resource (pm-83eb69d4) on (cln21-blabla).
drive mirror is starting for drive-scsi0
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error: drive-scsi0: Source and target image have different sizes (io-status: ok)

```

Any helpful hint, what is wrong?

This is my Server- and LINSTOR configuration:

```auto
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 3,6T 0 disk 
├─sda1 8:1 0 93,1G 0 part 
│ ├─SYS-root 252:1 0 83,8G 0 lvm /
│ └─SYS-swap 252:2 0 9,3G 0 lvm [SWAP]
├─sda2 8:2 0 99M 0 part 
├─sda3 8:3 0 1,5T 0 part 
│ └─lvmlocal-vm--501--disk--0 252:0 0 100G 0 lvm  
└─sda4 8:4 0 2T 0 part 
  ├─linstor_vg-thinpool_tmeta 252:3 0 108M 0 lvm  
  │ └─linstor_vg-thinpool-tpool 252:5 0 1,6T 0 lvm  
  │ ├─linstor_vg-thinpool 252:6 0 1,6T 1 lvm  
  │ └─linstor_vg-pm--d0e23104_00000 252:8 0 11G 0 lvm  
  │ └─drbd1000 147:1000 0 11G 0 disk 
  └─linstor_vg-thinpool_tdata 252:4 0 1,6T 0 lvm  
    └─linstor_vg-thinpool-tpool 252:5 0 1,6T 0 lvm  
      ├─linstor_vg-thinpool 252:6 0 1,6T 1 lvm  
      └─linstor_vg-pm--d0e23104_00000 252:8 0 11G 0 lvm  
        └─drbd1000 147:1000 0 11G 0 disk 

cat /etc/pve/storage.cfg

...
drbd: linstor01
    content images, rootdir
    controller 192.168.204.4
    resourcegroup pve-rg

linstor sp list

+----------------------------------------------------------------------------------------------------------------------------------------------------------------+
| StoragePool | Node | Driver | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName |
|================================================================================================================================================================|
| DfltDisklessStorPool | cln21-blabla | DISKLESS | | | | False | Ok | cln21-blabla;DfltDisklessStorPool |
| DfltDisklessStorPool | cln22-blabla | DISKLESS | | | | False | Ok | cln22-blabla;DfltDisklessStorPool |
| pve-storage | cln21-blabla | LVM_THIN | linstor_vg/thinpool | 1.64 TiB | 1.64 TiB | True | Ok | cln21-blabla;pve-storage |
| pve-storage | cln22-blabla | LVM_THIN | linstor_vg/thinpool | 1.64 TiB | 1.64 TiB | True | Ok | cln22-blabla;pve-storage |
+----------------------------------------------------------------------------------------------------------------------------------------------------------------+

linstor n l
+----------------------------------------------------------------+
| Node | NodeType | Addresses | State |
|================================================================|
| cln21-blabla | SATELLITE | 192.168.204.3:3366 (PLAIN) | Online |
| cln22-blabla | SATELLITE | 192.168.204.4:3366 (PLAIN) | Online |
+----------------------------------------------------------------+

linstor rg list
╭────────────────────────────────────────────────────────────────────╮
┊ ResourceGroup ┊ SelectFilter ┊ VlmNrs ┊ Description ┊
╞════════════════════════════════════════════════════════════════════╡
┊ DfltRscGrp ┊ PlaceCount: 2 ┊ ┊ ┊
╞┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄╡
┊ pve-rg ┊ PlaceCount: 2 ┊ ┊ ┊
┊ ┊ StoragePool(s): pve-storage ┊ ┊ ┊
╰────────────────────────────────────────────────────────────────────╯

linstor resource list
╭────────────────────────────────────────────────────────────────╮
┊ ResourceName ┊ Node ┊ Port ┊ Usage ┊ Conns ┊ State ┊ CreatedOn ┊
╞════════════════════════════════════════════════════════════════╡
╰────────────────────────────────────────────────────────────────╯

```

---

<div class="post-metadata">

**Author:** ![fx991](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/fx991/32/61_2.png) [@fx991](https://forums.linbit.com/u/fx991)\
**Post date:** [August 4, 2024, 2:06am UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/4 "2024-08-04T02:06:58Z")

</div>

Here are the software versions used:

```auto
8.0.7 pve-cluster
5.1.12 pve-container
8.2.2 pve-docs
4.2023.08-4 pve-edk2-firmware
4.2023.08-4 pve-edk2-firmware-legacy
4.2023.08-4 pve-edk2-firmware-ovmf
0.7.1 pve-esxi-import-tools
5.0.7 pve-firewall
4.0.5 pve-ha-manager
8.2.0 pve-headers
3.2.2 pve-i18n
1.3.0 pve-lxc-syscalld
8.2.4 pve-manager
9.0.2-1 pve-qemu-kvm
5.3.0-3 pve-xtermjs
2.3.1 pve-zsync
1.23.0-1 linstor-client
1.29.0-1 linstor-common
1.29.0-1 linstor-controller
8.0.4-1 linstor-proxmox
1.29.0-1 linstor-satellite
1.23.0-1 python-linstor

```

---

<div class="post-metadata">

**Author:** ![fx991](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/fx991/32/61_2.png) [@fx991](https://forums.linbit.com/u/fx991)\
**Post date:** [August 4, 2024, 12:23pm UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/5 "2024-08-04T12:23:59Z")

</div>

The problem is a known bug. The reason for the message is because of the drbd-metadata, which adds to the guest storage and thus leading to different sizes. A workround exists by using external metadata. This is discouraged (by linbit) because the linstor-proxmox-plugin is only designed for use with internal metadata.

Comment to the bug from a developer:

> tl;dr: the problem is now fully understood, but it will take time until this is fully resolved.

Look here for details:

> <https://github.com/LINBIT/linstor-proxmox/issues/54>
>
> Hi
> I've got few simple installations like 3 ProxMox servers with DRBD pool on t…hem, linstor-satellite and proxmox-linstor plugin. 
> And I've got problem at live migrating disk from local LVM/LVM thin to DRBD's pool based on LVM/LVM thin. But if i turn off VM then I can easily without any problems migrate. And also at the same time when I'm live migrating local LVM/LVM thin to DRBD's pool based on ZFS - all going good. 
> 
> pveversion: 
> pve-manager/7.4-3/9002ab8a (running kernel: 5.15.104-1-pve)
> 
> linstor pkg version:
> ii linstor-common 1.22.1-1 all
> ii linstor-proxmox 7.0.0-1 all
> ii linstor-satellite 1.22.1-1 all
> 
> drbd proc:
> version: 9.2.3 (api:2/proto:86-122)
> GIT-hash: c142ca1280c41aee1330b980544ef276330ff6ef build by root@testiko, 2023-04-10 18:28:06
> Transports (api:18): tcp (9.2.3)
> 
> part of LVM's debug log(maybe at will be helpfull):
> lvcreate --config "devices { filter=\['a|/dev/sdb|','r|.\*|'\] }" --virtualsize 5246976k linstor\_thinpool0 --thinpool thinpool0 --name vm-116-disk-1\_00000
> lvchange --config "devices { filter=\['a|/dev/sdb|','r|.\*|'\] }" -ay -K linstor\_thinpool0/vm-116-disk-1\_00000
> pvdisplay --config "devices { filter=\['r|/dev/drbd.\*|','a|.\*|'\] }" --columns -o pv\_name -S vg\_name=linstor\_thinpool0 --noheadings --nosuffix
> lvremove --config "devices { filter=\['a|/dev/sdb|','r|.\*|'\] }" -f linstor\_thinpool0/vm-116-disk-1\_00000
> 
> Error from GUI proxmox:
> create full clone of drive scsi0 (lvm\_local\_test:vm-116-disk-0)
> 
> NOTICE
> Trying to create diskful resource (vm-116-disk-1) on (testiko).
> drive mirror is starting for drive-scsi0
> drive-scsi0: Cancelling block job
> drive-scsi0: Done.
> TASK ERROR: storage migration failed: block job (mirror) error: drive-scsi0: 'mirror' has been cancelled

---

<div class="post-metadata">

**Author:** ![fx991](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/fx991/32/61_2.png) [@fx991](https://forums.linbit.com/u/fx991)\
**Post date:** [August 5, 2024, 3:40pm UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/6 "2024-08-05T15:40:06Z")

</div>

The problem is fixed with the latest version of the linstor-proxmox plugin. One has to set “exactsize yes” as option for the linstor-storage in /etc/pve/storage.cfg for online storage migration to work.

This is my /etc/pve/storage.cfg:

```auto
drbd: linstor01
        resourcegroup pve-rg
        content rootdir,images
        controller 192.168.204.4
        exactsize yes

```

For details read:

> **[LINSTOR 1.0 en](https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-linstor-proxmox-migrating-storage)**
>
> The LINSTOR User’s Guide 2024-08-01 18:35:20 UTC Please Read This First This guide is intended to serve users of the software-defined storage (SDS) solution LINSTOR® as a definitive reference guide and handbook. This guide assumes, throughout, that...

---

<div class="post-metadata">

**Author:** ![fx991](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/fx991/32/61_2.png) [@fx991](https://forums.linbit.com/u/fx991)\
**Post date:** [August 5, 2024, 4:07pm UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/7 "2024-08-05T16:07:55Z")

</div>

I think it’s obvious, that having to set an additional flag is not a perfect choice and I would love linstor/drbd more, to have a software that _ **just works** _ ™. But so far: Thanks for the fix!

---

<div class="post-metadata">

**Author:** ![fx991](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/fx991/32/61_2.png) [@fx991](https://forums.linbit.com/u/fx991)\
**Post date:** [August 5, 2024, 7:38pm UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/8 "2024-08-05T19:38:01Z")

</div>

I did another online moving disk test and it failed.

I just realized, that there had been an orphaned linstor resource definition of a vm. This command showed me all resource-definitions:

```auto
for rd in $(linstor -m resource-definition list | jq -r '.[][].name');do 
   linstor resource-definition list-properties $rd
done

```

The following was the orphaned resource-definition:

```auto
# linstor resource-definition list-properties pm-d0e23104

╭─────────────────────────────────────────────╮
┊ Key ┊ Value ┊
╞═════════════════════════════════════════════╡
┊ Aux/pm/vmid ┊ 102 ┊
┊ DrbdOptions/ExactSize ┊ false ┊
┊ DrbdOptions/Net/allow-two-primaries ┊ yes ┊
┊ DrbdOptions/Resource/quorum ┊ off ┊
╰─────────────────────────────────────────────╯

```

This avoided a successful test for online moving of storage from lvm to linstor. The value ExactSize yes was overruled by this resource-definition and I again got the error.

I had to delete that resource-definition. After that the live online storage moving worked again.

---

<div class="post-metadata">

**Author:** ![kermat](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/kermat/32/21_2.png) [@kermat](https://forums.linbit.com/u/kermat)\
**Post date:** [August 5, 2024, 8:44pm UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/9 "2024-08-05T20:44:42Z")

</div>

I’m sure someone else will struggle with this before it receives a proper fix. Thank you for noting these issues and workarounds.

---

<div class="post-metadata">

**Author:** ![fx991](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/fx991/32/61_2.png) [@fx991](https://forums.linbit.com/u/fx991)\
**Post date:** [August 6, 2024, 10:44am UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/10 "2024-08-06T10:44:24Z")

</div>

I still have unresolved trouble migrating Storage and VM. There is a VM, I migrated back and forth and it can not be migrated between lvm and linstor because of the different sizes problem, although exactsize is set to yes, although the settings of any resource-definition object has exactsize set to yes.

---

<div class="post-metadata">

**Author:** ![Ryan](https://yyz1.discourse-cdn.com/flex003/user_avatar/forums.linbit.com/ryan/32/42_2.png) [@Ryan](https://forums.linbit.com/u/Ryan)\
**Post date:** [August 9, 2024, 11:16pm UTC](https://forums.linbit.com/t/trying-to-move-vm-to-linstor-storage-storage-migration-failed-source-and-target-image-have-different-sizes/213/11 "2024-08-09T23:16:29Z")

</div>

I have been using a Proxmox cluster for testing lately and I am able to reproduce this.

First, if I create a new disk for a running VM backed by LINSTOR with the `exactsize yes` parameter specified in `/etc/pve/storage.cfg`, I can move this disk image back and forth between LVM ↔ LINSTOR repeatedly without issue while the VM is running. Everything works as expected here.

However, `exactsize yes` is only supposed to be temporarily set for migration…

On the flipside, creating a new disk image for a running VM backed by LINSTOR with `exactsize no` presents an issue. Trying to move a running VM’s disk from LINSTOR → LVM at this stage results in:

```txt
TASK ERROR: storage migration failed: block job (mirror) error: drive-virtio2: Source and target image have different sizes (io-status: ok)

```

So, when prepping for migration and toggling `exactsize yes` in `/etc/pve/storage.cfg`, and also ensuring that the current resource definition is adjusted (`linstor rd set-property <res_id> DrbdOptions/ExactSize yes`), migration still fails in Proxmox:

```txt
TASK ERROR: storage migration failed: block job (mirror) error: drive-virtio2: Source and target image have different sizes (io-status: ok)

```

The workaround is to detach the disk, re-attach and then attempt live migration once more. Power cycling the virtual machine also allows the migration to happen, and of course, offline migration works without issue.

I’m guessing the background DRBD adjust command either needs to do some extra operations for this to work, or DRBD has a technical reason it needs to be demoted to _Secondary_ (inactive) when `DrbdOptions/ExactSize yes` is toggled from no to yes.

Thanks for powering through this one, I’ll comment on the GitHub issue with the findings above.
