MDRAID10 performance degradation on OpenSuSE 15.4 is often caused by default kernel scheduler settings that penalize random write patterns on SATA arrays. By switching the I/O scheduler to none or mq-deadline for the underlying block devices, you can significantly improve throughput and reduce latency during heavy random write operations without altering RAID configuration parameters.
Understanding the Performance Bottleneck
When working with MDRAID10 arrays composed of SATA drives, users frequently encounter unexpected slowdowns specifically during random write workloads. This issue is not necessarily a hardware failure or a misconfiguration of the RAID level itself, but rather a mismatch between how the Linux kernel schedules disk I/O and how modern storage controllers handle requests. The default I/O scheduler in many Linux distributions, particularly older kernels or specific configurations found in OpenSuSE 15.4, may prioritize sequential access patterns over random ones. SATA drives have different performance characteristics compared to SSDs or NVMe drives. They rely heavily on command queuing and efficient ordering of read/write requests. If the operating system sends thousands of small, scattered write commands without proper batching or reordering, the drive heads (in HDDs) or controller queues (in SSDs acting as HDDs) become overwhelmed. This results in high latency and low throughput, which manifests as an "extremely slow" experience for the end user. The problem is exacerbated when multiple drives are grouped into a RAID array, as the software RAID layer adds another layer of abstraction that interacts with the kernel's block layer.Why Default Settings Fail Random Writes
The traditionalcfq (Completely Fair Queuing) scheduler was designed for mixed workloads and fairness among processes, not for raw performance in high-throughput scenarios. It introduces significant overhead when dealing with the rapid-fire nature of random writes. Similarly, while deadline is better than cfq, it may still introduce unnecessary delays if not tuned correctly for multi-device RAID setups. For MDRAID10, which stripes data across multiple disks for both redundancy and performance, the underlying block devices need a scheduler that minimizes decision-making overhead and passes requests to the device driver as quickly as possible.
This is where the choice of scheduler becomes critical. Modern Linux kernels support Multi-Queue (MQ) schedulers which are designed for NVMe and high-performance SATA/SCSI stacks. However, these must be explicitly selected. If the system defaults to a legacy scheduler, the performance penalty is substantial. Users reporting slow random writes are almost always seeing the impact of this scheduling inefficiency rather than a fundamental flaw in the MDRAID10 implementation.
Diagnosing Current Scheduler Configuration
Before applying any changes, it is essential to verify the current I/O scheduler assigned to your block devices. You can check this by examining the sysfs interface for each disk involved in the RAID array. Run the following command to list all block devices and their current schedulers:cat /sys/block/sda/queue/scheduler
cat /sys/block/sdb/queue/scheduler
sda and sdb with the actual device names associated with your RAID members. The output will typically look like [cfq] mq-deadline none kyber, where the value inside brackets indicates the currently active scheduler. If you see cfq or an older variant, this is likely contributing to your performance issues. Note that on some systems, especially those using UEFI and newer kernels, the default might already be mq-deadline or none. If you are already using one of these, the issue may lie elsewhere, such as filesystem mount options or cache settings. However, if you are using cfq, changing it is the most direct path to improvement.
Implementing the Solution
To resolve the slow random write speeds, you should switch the I/O scheduler for the underlying physical disks to eithernone or mq-deadline. The none scheduler removes the kernel-level scheduling logic entirely, relying on the device driver to manage requests. This is often the fastest option for SSDs and modern SATA drives with good internal queuing. The mq-deadline scheduler provides a balance, ensuring that no request waits too long while still allowing some optimization.
You can test this change temporarily by writing to the scheduler file for each device. Use the following commands, replacing sdX with your actual device identifiers:
echo none > /sys/block/sdX/queue/scheduler
echo mq-deadline > /sys/block/sdY/queue/scheduler
iostat from the sysstat package or htop. You should observe a noticeable reduction in wait times and an increase in write throughput during random load tests. If the performance improves, you need to make these changes persistent across reboots.
Making Changes Persistent on OpenSuSE 15.4
On OpenSuSE 15.4, the recommended way to set I/O schedulers persistently is through udev rules. Create a new file in the/etc/udev/rules.d/ directory, for example, 99-io-scheduler.rules. Add a rule that matches your specific block devices and sets the scheduler. Here is an example template:
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/scheduler}="none"
mq-deadline, replace none with mq-deadline in the rule. It is crucial to apply this rule to the physical disks (sda, sdb, etc.) and not the RAID device itself (md0), as the RAID layer handles its own internal logic. After saving the file, reload the udev rules with udevadm control --reload-rules and trigger them with udevadm trigger. Rebooting the system is the safest way to ensure all services start with the correct configuration.
Additional Optimization Considerations
While changing the I/O scheduler addresses the primary bottleneck, other factors can influence MDRAID10 performance. Ensure that your filesystem is mounted with appropriate options. For ext4, thenoatime mount option prevents the system from updating access timestamps on every read, reducing unnecessary write operations. For XFS, similar optimizations apply. Additionally, verify that your kernel parameters related to dirty page caching are not causing excessive flushing. The vfs_cache_pressure and dirty_ratio settings in /proc/sys/vm/ can be tuned, but this requires careful benchmarking to avoid memory pressure issues.
It is also worth checking if any background services are interfering with disk I/O. Indexing services, backup daemons, or log rotation scripts can create sporadic I/O spikes that compound the effects of a poor scheduler. Using iotop can help identify which processes are consuming the most I/O bandwidth, allowing you to schedule non-critical tasks during off-peak hours.
Frequently Asked Questions
Will changing the I/O scheduler affect read performance?
Yes, switching to none or mq-deadline generally improves both read and write performance by reducing kernel overhead. Reads benefit because the scheduler no longer artificially delays requests to balance fairness, allowing the drive to service them immediately. In RAID arrays, this leads to more consistent latency across all operations.
Is it safe to use the 'none' scheduler on mechanical hard drives?
Using none on mechanical hard drives (HDDs) can sometimes lead to increased head movement if the workload is highly random, potentially causing wear. However, in a RAID10 setup with multiple drives, the load is distributed, mitigating this risk. For pure HDD arrays, mq-deadline is often a safer middle ground that still offers significant performance gains over cfq.
How do I revert the changes if performance worsens?
To revert, simply edit the udev rule file you created in /etc/udev/rules.d/ and remove or comment out the lines setting the scheduler. Then run udevadm control --reload-rules and udevadm trigger. Alternatively, you can manually set the scheduler back to cfq via the command line for testing purposes before making permanent changes.
Does this solution apply to NVMe drives as well?
NVMe drives typically use the none scheduler by default on modern kernels because they have built-in hardware queue management. If you are using NVMe, the issue is less likely to be the scheduler and more likely related to CPU affinity or PCIe lane sharing. However, verifying the scheduler remains a good diagnostic step to ensure no legacy overrides are present.
What if my RAID array is still slow after changing the scheduler?
If performance does not improve, check the SMART status of your individual drives to rule out hardware degradation. Also, inspect the RAID metadata health using mdadm --detail /dev/md0 to ensure there are no degraded components or resyncing processes running in the background. Finally, consider adjusting the stripe cache size if you are using large drives, though this is an advanced tuning step.