Diagnosis and replacement of a defective hard drive (Linux Dedicated Servers and Bare Metal Servers with software RAID)
Please use the “Print” function at the bottom of the page to create a PDF.
If you receive a notification about a hard drive error or notice irregularities in the system, quick action is required to restore the redundancy of your RAID array. This article explains how to identify a defective hard drive on a Linux Dedicated Server or a Linux Bare Metal Server with software RAID and how to prepare the server for replacing the defective drive.
Notes
This article assumes basic knowledge of server administration with Linux. If you have questions about replacing a defective hard drive or need support, please contact IONOS Customer Support. You can find the contact information on the following page: IONOS Customer Support
The examples in this article refer primarily to classic SATA/SAS drives in software RAID configurations with device names such as /dev/sda and /dev/sdb. For NVMe drives, the device names, the diagnostic tools, and, depending on the configuration, individual steps differ.
To ensure the highest possible reliability, it is necessary that you monitor your server's software RAID. If you receive a notification email about a defective hard drive or notice a hard drive defect yourself, you must identify the defective hard drive and prepare the server for replacing the defective drive. Then, contact IONOS Customer Support to initiate the hard drive replacement.
Please Note
RAID systems provide greater reliability and/or higher speed. However, they are no substitute for regular backups. To prevent data loss, we recommend creating regular backups. Furthermore, make sure to create a backup before executing the steps listed below to ensure the security of your data.
Further information on creating backups can be found in the following category of the IONOS Help Center: Backup Solutions
Check the status of the software RAID
Establish an SSH connection to your server and log in with your root account. For instructions, see the following articles:
Establishing an SSH connection to your Linux server from a Microsoft Windows computer
Establishing an SSH connection to your Linux server from a Linux computer
To check the status of the software RAID, enter the following command in the shell:
cat /proc/mdstat
Interpretation of the output
Intact RAID: A functioning RAID is indicated by the status [UU] (for RAID 1) or [UUUU] (for RAID 5/6 with 4 hard drives). Each U stands for Up (active).
Defective/Missing Device: A [_U] or [U_] indicates that a hard drive is missing or out of sync.
Faulty Marker: An (F) after a device (for example, sdb1[2](F)) means that the system has already marked the drive as defective (faulty) in software.
In configurations with 2 SSDs (operating system) and additional HDDs (data), you will see multiple md devices:
- md1, md2, md127, or similar (often RAID 1 for boot/root)
- md11 or similar (often RAID 5 or 6 for data)
Check all sections of the output for missing Us or (F) markers. Here is an example of an intact RAID:
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
read_ahead 1024 sectors
md2 : active raid1 sda3[1] sdb3[0]
262016 blocks [2/2] [UU]
md1 : active raid1 sda2[1] sdb2[0]
119684160 blocks [2/2] [UU]
md0 : active raid1 sda1[1] sdb1[0]
102208 blocks [2/2] [UU]
unused devices: <none>
The example above shows three Multiple Devices or RAID arrays (md0, md1, md2). For each of these logical drives, it is specified which partitions it consists of and on which drives these partitions are located. The logical drive md0 consists of the partitions sda1 and sdb1. In the line below the respective logical drive, the state of the individual partitions is shown in square brackets at the end of the line.
If a hard drive is missing or defective, this is often indicated by [_U] or [U_]. An (F) after a partition (for example, sdb3[2](F)) means that it has been marked as faulty.
In the following example, only one partition is integrated into all logical drives, which is located on the hard drive sda. The respective partition located on the second hard drive sdb is not integrated. You can also recognize this by the entry [_U]. The partitions of the hard drive sdb that are not integrated indicate that there is an error or a defect with this hard drive.
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
read_ahead 1024 sectors
md0 : active raid1 sda1[1]
102208 blocks [2/1] [_U]
md1 : active raid1 sda2[1]
119684160 blocks [2/1] [_U]
md2 : active raid1 sda3[1]
262016 blocks [2/1] [_U]
unused devices: <none>
In the following example, a defective hard drive is still integrated into the RAID. This can be recognized by the information (F) displayed for md1.
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2](F)
439553856 blocks super 1.0 [2/1] [U_]
bitmap: 1/4 pages [4KB], 65536KB chunk
md1 : active raid1 sdb1[2](F) sda1[0]
19529600 blocks super 1.0 [2/1] [U_]
unused devices: <none>
Diagnosis of hard drive errors
To detect hard drive errors, complete the following:
Install the smartctl program. smartctl is a command-line utility used to monitor drives using SMART (Self-Monitoring, Analysis and Reporting Technology). You can use this program to check if a hard drive is defective. It is part of the Smartmontools. Smartmontools are available as packages for many Linux distributions. More information can be found on the following page: Smartmontools Packages
Note
In some cases, a hard drive defect might not be detected using the SMART values. Therefore, we recommend additionally analyzing the log file /var/log/messages.
Install smartctl
The installation command differs depending on the operating system you use. Run the appropriate command for your distribution:
To install smartctl, log in to the server as an administrator.
Install the required packages for your distribution:
AlmaLinux 9 and 10, and Rocky Linux 9 and 10:
dnf install smartmontools
Debian 12 and 13, and Ubuntu 24.04 and 26.04:
sudo apt update
sudo apt install smartmontools
Retrieving hard drive information
To retrieve a list of hard drives, enter the following command:
smartctl --scan
Example:
[root@localhost ~]# smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
To retrieve detailed information for error diagnosis, enter the following command:
smartctl -iHAl error [HARD_DRIVE_NAME]
Notes
The device interfaces must be specified in the following format:
SCSI/SATA devices:
smartctl -iHAl error /dev/sd[a-z]
Example:
[root@localhost ~]# smartctl -iHAl error /dev/sda
NVMe drives use different device names, for example, /dev/nvme0 or /dev/nvme0n1, and, depending on the query, require other or additional tools.
After entering the command, information similar to the following will be displayed:
[root@localhost ~]# smartctl -iHAl error /dev/sda
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-862.14.4.el7.x86_64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: HGST HUS722T1TALA604
Serial Number: WMC6N0K2RW66
LU WWN Device Id: 5 0014ee 004722db0
Firmware Version: RAGNWA07
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri May 3 07:45:14 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always 0
3 Spin_Up_Time 0x0027 183 183 021 Pre-fail Always 3833
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always 9
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always 0
9 Power_On_Hours 0x0032 097 097 000 Old_age Always 2560
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always 9
16 Unknown_Attribute 0x0022 000 200 000 Old_age Always 26802171994
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always 0
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always 4
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always 67
194 Temperature_Celsius 0x0022 116 111 000 Old_age Always 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline 0
SMART Error Log Version: 1
No Errors Logged
Interpretation of parameters and error diagnosis
Analyse the detailed information you retrieved using the smartctl -iHAl error [HARD_DRIVE_NAME] command.
The first section lists information you can use to identify the hard drive:
=== START OF INFORMATION SECTION ===
Device Model: HGST HUS722T1TALA604
Serial Number: WMC6N0K2RW66
LU WWN Device Id: 5 0014ee 004722db0
Firmware Version: RAGNWA07
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri May 3 07:45:14 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
This section displays, among other things, the Device Model and the Serial Number of the tested hard drive.
The second section shows how smartctl assesses the current condition of the hard drive. If the value displayed is not PASSED but, for example, FAILED or UNKNOWN, you should arrange for the hard drive to be replaced as soon as possible.
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
The third section details the determined SMART values. Next to each current percentage value (VALUE), the worst ever measured value (WORST) and the respective threshold (THRESH) are listed. If the current percentage value or the worst ever measured value exceeds the threshold, a SMART warning is displayed in the WHEN_FAILED column (e.g., FAILING_NOW).
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always 0
3 Spin_Up_Time 0x0027 183 183 021 Pre-fail Always 3833
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always 9
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always 0
9 Power_On_Hours 0x0032 097 097 000 Old_age Always 2560
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always 9
16 Unknown_Attribute 0x0022 000 200 000 Old_age Always 26802171994
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always 0
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always 4
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always 67
194 Temperature_Celsius 0x0022 116 111 000 Old_age Always 31
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline 0
The following parameters can indicate an impending hard drive failure before a SMART warning is triggered:
Reallocated_Sector_Ct: This parameter specifies the number of sectors that have already been reallocated due to read errors. If a sector can no longer be read, written to, or checked correctly, the controller automatically assigns it a spare sector from the hard drive's reserve area. The original faulty sector is permanently marked as defective and is no longer used.
Note
A value greater than zero is not necessarily a cause for concern as long as it remains stable over a long period. However, a critical indicator of an impending hard drive failure is a steadily growing number of reallocated sectors. To detect a defect early, you should log this value regularly and request a replacement immediately if there is a continuous increase.
Current_Pending_Sector: Indicates the number of unstable sectors waiting to be remapped. If a sector cannot be read and written correctly, it initially receives the status Current Pending Sector. In this state, the sector is not reallocated because the data on it is unknown. Only after several unsuccessful read or write attempts is a replacement sector assigned and the faulty sector permanently marked as unreadable. The Current_Pending_Sector value is an important indicator for a hard drive replacement. If this value is not zero, a hard drive failure is often imminent.
Offline_Uncorrectable: Indicates the number of uncorrectable errors during read and write access to sectors.
The last section deals with the internal hard drive error log. Errors are recorded here if the server's work orders were not processed correctly by the hard drive. If a double-digit error count (at least) is displayed in this section, you should arrange for a hard drive replacement as soon as possible.
SMART Error Log Version: 1
No Errors Logged
Retrieve detailed information for hard drive replacement
To arrange for the defective hard drive to be replaced, the following information is required:
Identifier of the hard drive in the RAID (e.g., sda)
Serial number
Model
Log file (optional)
Create a SMART log
To generate a full SMART log, enter the following command:
smartctl -x [HARD_DRIVE_NAME]
Example:
[root@localhost ~]# smartctl -x /dev/sda
If the hard drive can no longer be accessed via smartctl, you can retrieve the required information using the hdparm program. To install hdparm:
AlmaLinux 9 and 10, and Rocky Linux 9 and 10:
dnf -y install hdparm
Debian 12 and 13, and Ubuntu 24.04 and 26.04:
sudo apt-get update
sudo apt-get install hdparm
Then, enter the following command to retrieve the information required for the hard drive replacement:
hdparm -i /dev/sda
Notes
If the SMART log was created as described above, this is sufficient information. You can then arrange for the defective hard drive to be replaced. To do this, please contact IONOS Customer Support.
If you cannot retrieve the serial number of the defective hard drive using smartctl, you can provide IONOS Customer Support with the serial number(s) of the functioning hard drive(s) instead.
hdparm is primarily intended for classic SATA/ATA hard drives and is not suitable for NVMe drives. If your system uses NVMe drives, the required device information can usually be read with distribution-specific NVMe tools, for example, nvme-cli, instead of hdparm. However, this article describes the replacement of classic hard drives in software RAID configurations with device names such as /dev/sda and /dev/sdb.
Prepare the server for hard drive replacement
In the following example, it is assumed that the second hard drive (sdb) needs to be replaced. As part of the status check, the following software RAID status is displayed, for example:
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2]
439553856 blocks super 1.0 [2/1] [UU]
md1 : active raid1 sdb1[2] sda1[0]
19529600 blocks super 1.0 [2/1] [UU]
unused devices: <none>
In this example, the second hard drive (sdb) is still integrated into the RAID and is therefore still in operation.
Manually mark the RAID device as faulty to remove it from the RAID
To remove the faulty hard drive from the RAID, mark it as faulty. To do this, enter the following command:
[root@localhost ~]# mdadm PATH_TO_RAID_ARRAY -f PATH_TO_HARD_DRIVE
In the examples below, the hard drives sdb3 and sdb1 are marked as faulty:
[root@localhost ~]# mdadm /dev/md3 -f /dev/sdb3
mdadm: set /dev/sdb3 faulty in /dev/md3
[root@localhost ~]# mdadm /dev/md1 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md1
After entering the command, the RAID will have the following status (showing the (F) marker):
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2](F)
439553856 blocks super 1.0 [2/1] [U_]
md1 : active raid1 sdb1[2](F) sda1[0]
19529600 blocks super 1.0 [2/1] [U_]
unused devices: <none>
Removing a partition from the RAID
To remove a partition from the RAID, enter the following command:
[root@localhost ~]# mdadm -r /PATH_TO_RAID_ARRAY /PATH_TO_HARD_DRIVE
In the examples below, the hard drives sdb3 and sdb1 are removed from the RAIDs md3 and md1:
[root@localhost ~]# mdadm -r /dev/md3 /dev/sdb3
mdadm: hot removed /dev/sdb3 from /dev/md3
[root@localhost ~]# mdadm -r /dev/md1 /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md1
Next, check the status of the RAID. In this example, the RAID that has been prepared for the hard drive replacement has the following final status:
[root@localhost ~]# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sda3[0]
439553856 blocks super 1.0 [2/1] [U_]
md1 : active raid1 sda1[0]
19529600 blocks super 1.0 [2/1] [U_]
unused devices: <none>
Check used swap partitions
Check which swap partitions are being used by the operating system. To do this, enter the following command:
[root@localhost ~]# cat /proc/swaps
Filename Type Size Used Priority
/dev/sda2 partition 9765884 0 -1
/dev/sdb2 partition 9765884 0 -2
Alternatively, you can verify which swap partitions are defined in fstab by entering the following command:
[root@localhost ~]# grep swap /etc/fstab
/dev/sda2 none swap sw
/dev/sdb2 none swap sw
Deactivate the swap partition on the defective device
Deactivate the swap partition on the defective hard drive so that it can be safely replaced. To do this, enter the following command:
[root@localhost ~]# swapoff PATH_TO_PARTITION
Example:
[root@localhost ~]# swapoff /dev/sdb2
Note
If the swap partition on the defective hard drive is not deactivated before the hard drive is replaced, the swap partition is given the status deleted in /proc/swaps.
Arrange for a hard drive replacement
You can now arrange for the faulty hard drive to be replaced. To do so, please contact IONOS Customer Support. You will find the contact details on the following page: IONOS Customer Support
Required steps after hard drive replacement
After the defective hard drive has been replaced, you need to rebuild the software RAID. For more information on rebuilding a software RAID, see the following article: Rebuilding software RAID (Linux/Dedicated Server with BIOS)
Content
- Check the status of the software RAID
- Interpretation of the output
- Diagnosis of hard drive errors
- Interpretation of parameters and error diagnosis
- Retrieve detailed information for hard drive replacement
- Create a SMART log
- Prepare the server for hard drive replacement
- Arrange for a hard drive replacement
- Required steps after hard drive replacement
- To top