Poor I/O Performance with QEMU-KVM Native Block Device (Dedicated Disk) on RHEL 9
In this article, we will discuss the poor I/O performance observed when running a VM RHEL 9 (QEMU-KVM 9.0.0, release 10.el9_5) and provide potential solutions to improve it. The performance metrics are collected using the FIO tool.
Understanding QEMU-KVM Native Block Device
The QEMU-KVM Native Block Device is a virtual disk provided to virtual machines running on QEMU-KVM. It allows VMs to access physical disks or disk images as if they were local disks, enabling data persistence and storage for VMs.
Identifying Poor I/O Performance
Poor I/O performance can manifest as slower-than-expected read and write operations, which may impact application performance, system responsiveness, and user experience. To identify poor I/O performance, you can use tools like FIO to collect performance metrics.
Collecting Performance Metrics with FIO
FIO is a versatile I/O tester and benchmarking tool that supports various I/O workloads, including sequential and random reads and writes. To collect performance metrics, follow these steps:
- Install FIO:
sudo dnf install fio - Create a job file, e.g., fio-job.ini, to define your test workload:
[global]
ioengine=libaio
direct=1
rw=randwrite
bs=4k
numjobs=16
size=1G
This configuration defines a random write workload with a 4KB block size, 16 parallel jobs, and a 1GB test size.
- Run the FIO test:
fio fio-job.ini
Analyzing Performance Metrics
Analyze FIO test results to identify bottlenecks and poor performance. Key performance metrics include IOPS (input/output operations per second), throughput (MB/s), latency (response time), and CPU utilization.
Potential Solutions to Improve I/O Performance
Several methods can help improve poor I/O performance in QEMU-KVM:
- Increase the number of CPU cores allocated to the VM.
- Optimize the disk cache policy for the Native Block Device.
- Enable QEMU's asynchronous I/O for improved performance.
- Use a specialized I/O scheduler, such as "noop" or "deadline" for your Native Block Device.
- Use virtio-scsi for disk I/O to leverage paravirtualization benefits.
Implementing Solutions in QEMU-KVM
Implement the discussed solutions in QEMU-KVM using the libvirt XML configuration:
- Adjust CPU cores allocation:
<vcpu placement='static' current='4' total='8'/> - Optimize disk cache policy:
<driver name='qemu' type='raw' cache='unsafe'/> - Enable asynchronous I/O:
<driver name='qemu' type='raw' io='native'/> - Use the "noop" or "deadline" I/O scheduler (example with "noop"):
<driver name='qemu' type='raw' queue='scsi,none'/> - Use virtio-scsi for disk I/O:
<driver name='qemu' type='scsi' if='virtio'/>
Poor I/O performance in QEMU-KVM can be addressed by adjusting the VM's resources, optimizing disk cache policy, enabling QEMU's asynchronous I/O, using a specialized I/O scheduler, and employing virtio-scsi for disk I/O. Regularly monitor and collect performance metrics to evaluate the effectiveness of these optimizations.
References
- FIO tool: https://fio.readthedocs.io/en/latest/
- QEMU documentation: https://qemu.weilnetz.de/doc/qemu-doc.html
- Virtio-scsi: https://wiki.openstack.org/wiki/Virtio-SCSI