I’m getting a strange result when doing some performance testing with Kubevirt on a bare-metal Kubernetes cluster with Piraeus 2.8.1. It’s 3 servers (all control plane nodes) that each have 4x 1.8TB SSDs for Piraeus. I’ve configured the storagecluster with:
I used the linstor CLI to create the data-nic interfaces on a 25Gbps dedicated NIC, before enabling the linstorNodeConnection.
When I start a Windows VM, performance seems to be just fine. CrystalDiskMark shows numbers in line with expectations. But on a Linux VM, performance is terrible for some reason. FIO shows no more than 100 IOPS and dd tops out at 2.4MB/sec.
Nothing I do changes this, so there must be something wrong somewhere.
CrystalDiskMark and dd are probably not testing the same things.
What settings are you using with CrystalDiskMark?
Running dd like you’ve shown is effectively testing an IO depth or queue depth of 1, meaning that each of those 10,000 4k writes has to wait for the previous write to complete before placing another in the queue. It’s also testing sequential writes, as opposed to random writes, which are more commonly what people are looking at with 4k writes.
I’m not super familiar with Windows benchmarking tools, but it’s also possible that CrystalDiskMark’s IO is getting multi-threaded where dd isn’t going to do that on its own.
If you wanted to do more robust benchmarking in Linux I would recommend looking at fio. Then you will be able to more closely replicate real world IO patterns in Linux.
FIO gave me only ±100 IOPS in a 65/35 read write mixed workload. Running the exact same FIO test on the same hardware, but using Portworx Enterprise as the CSI, results in ±24.000 IOPS.
The problem isn’t with the testing methodology, the problem is with Piraeus. I’m looking for some pointers on where the issue could be.
You haven’t shared the testing parameters, only that you’re using CrystalDiskMarks in Windows and some combination of fio and dd in Linux, so making sure you’re comparing at least somewhat similar tests seems important. If you’re sure it’s not methodology, I’ll move on.
If Windows VMs perform as expected but Linux VMs are giving you trouble, both while using the same Piraeus storage class, that would suggest there is an issue above the storage, no? Windows VMs and Linux VMs are going to use different storage drivers and IO scheduling methods.
I would look at these things within the Linux guest:
lspci | grep Virtio # ensure NICs and disks are using virtio, not e1000/IDE
dmesg | grep -i virtio # confirm drivers loaded correctly
/sys/block/sdb/queue/scheduler # ensure not using CFQ
cat /proc/cpuinfo # see if CPU flags (e.g., AES, AVX, etc.) are being exposed
If everything looks okay there, can you share more about the guest OSes you’re testing with?
Earlier testing on other hardware didn’t show any issues, so I’m hoping the issue is something hardware-specific. We’re using Ubuntu for testing the Linux VMs, I tried both 22.04 and 24.04 with the same result.
The linux guest shows:
root@vm1x-ubuntu-fio:~# lspci | grep Virtio
01:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01)
05:00.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI (rev 01)
06:00.0 Communication controller: Red Hat, Inc. Virtio console (rev 01)
07:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
08:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
09:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
0a:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
0b:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)
0c:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon (rev 01)
0d:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG (rev 01)
root@vm1x-ubuntu-fio:~# dmesg | grep -i virtio
[ 2.065290] scsi host0: Virtio SCSI HBA
[ 2.070584] virtio_blk virtio3: [vda] 209719464 512-byte logical blocks (107 GB/100 GiB)
[ 2.123908] virtio_blk virtio4: [vdb] 2048 512-byte logical blocks (1.05 MB/1.00 MiB)
[ 2.135260] virtio_blk virtio5: [vdc] 209719464 512-byte logical blocks (107 GB/100 GiB)
[ 2.135555] virtio_net virtio0 enp1s0: renamed from eth0
[ 2.141024] virtio_blk virtio6: [vdd] 209719464 512-byte logical blocks (107 GB/100 GiB)
[ 2.144539] virtio_blk virtio7: [vde] 209719464 512-byte logical blocks (107 GB/100 GiB)
root@vm1x-ubuntu-fio:~# cat /sys/block/vdb/queue/scheduler
[none] mq-deadline
root@vm1x-ubuntu-fio:~# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel Core i7 9xx (Nehalem Class Core i7)
stepping : 3
microcode : 0x1
cpu MHz : 1995.317
cache size : 16384 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 3990.63
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel Core i7 9xx (Nehalem Class Core i7)
stepping : 3
microcode : 0x1
cpu MHz : 1995.317
cache size : 16384 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 3990.63
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel Core i7 9xx (Nehalem Class Core i7)
stepping : 3
microcode : 0x1
cpu MHz : 1995.317
cache size : 16384 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 3990.63
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel Core i7 9xx (Nehalem Class Core i7)
stepping : 3
microcode : 0x1
cpu MHz : 1995.317
cache size : 16384 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt hypervisor lahf_lm cpuid_fault pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 3990.63
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
You could try running a blkdiscard from within the guest against the affected devices to see if there might be some stale blocks that can be discarded from previous tests: