Introduction
In this article, we will discuss the recent issue faced by a Hadoop cluster installed on Dell machines (R750/R760) with 14 disks, each of size 16TB, running RHEL 8.6 version. Specifically, we will look into the inconsistencies in controller firmware version that has led to bad disk performance.
Understanding the Components
Hadoop
Hadoop is an open-source framework for distributed storage and processing of large datasets. It includes several modules such as HDFS (Hadoop Distributed File System), YARN (Yet Another Resource Negotiator), and MapReduce.
Dell Machines (R750/R760)
Dell EMC PowerEdge R750 and R760 servers are high-performance, scalable, and flexible platforms for modern workloads. They support a variety of storage configurations, including SATA, SAS, and NVMe drives.
RHEL 8.6
Red Hat Enterprise Linux 8.6 (RHEL) is a stable and secure operating system that provides a reliable and scalable platform for running applications. It is the recommended OS for running Hadoop clusters due to its robust storage and network capabilities.
Identifying the Issue
After the installation of the Hadoop cluster, it was observed that there were inconsistencies in the controller firmware version across the disks, leading to bad disk performance. The inconsistencies were caused due to the different firmware versions of the disk controllers, which resulted in compatibility issues and reduced the overall performance of the Hadoop cluster.
Solution
The solution to the issue was to update the controller firmware to a consistent version across all the disks. This involved the following steps:
- Identifying the controller firmware version across all the disks.
- Downloading the latest firmware version from the Dell website.
- Updating the firmware version on each disk using the Dell OpenManage software.
- Verifying that the firmware version was consistent across all the disks.
Code Example
Here is an example of how to query the controller firmware version using the OpenManage software:
# iserver query storage firmware --system <system-name> --detail
Prevention
To prevent this issue from occurring again, it is recommended to perform regular firmware updates on all the disks. This can be automated using tools such as Ansible or Puppet, which can be configured to run firmware update tasks on a scheduled basis.
References
- Hadoop -
- Dell EMC PowerEdge R750 -
- Red Hat Enterprise Linux 8.6 -
- Dell OpenManage -
In this article, we discussed the issues related to inconsistent controller firmware versions on Dell machines in a Hadoop cluster, which led to bad disk performance. By updating the firmware version across all the disks, we were able to resolve the issue. Regular firmware updates are recommended to prevent this issue from recurring.