Diagnose and Replace a Defective Hard Drive (Linux Dedicated Server with Software RAID)

In this article, you will learn how to identify a defective hard drive on a Linux Dedicated Server and how to prepare for the replacement.

Please Note

This article assumes that you have basic knowledge of Linux server administration. If you have any questions or need assistance regarding the replacement of a defective hard drive, please contact 1&1 IONOS Customer Support.

In order to ensure the highest possible reliability, it is necessary that you monitor the software RAID of your Dedicated Servers. If you discover that a hard drive is defective or receive a notification email about a defective hard drive, you must contact customer service to arrange for the hard drive replacement. This requires that you identify the defective hard drive and prepare the server for the replacement.

Please Note

RAID systems enable greater reliability and/or higher speed capabilities. However, they are not a substitute for regular backups. To avoid data loss, we recommend that you back up your data regularly. Also, make sure that you back up your data before performing the steps below to ensure the security of your data. For more information on creating backups, click here:

Back Up Linux Server Data

Checking the Status of the Software RAID

To check the status of the software RAID, enter the following command in the shell:

[root@host~]: cat /proc/mdstat

If both hard drives are present and properly mounted, the following message will be displayed:

[root@localhost ~]# cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
md2 : active raid1 sda3[1] sdb3[0]
262016 blocks [2/2] [UU]

md1 : active raid1 sda2[1] sdb2[0]
119684160 blocks [2/2] [UU]

md0 : active raid1 sda1[1] sdb1[0]
102208 blocks [2/2] [UU]

unused devices: <none>

The example above shows three multiple devices or logical drives (md0, md1, md2). For each of these logical drives, you specify which partitions they are composed of and on which drives these partitions are located.

Example: The logical drive md0 consists of the partitions sda1 and sdb 1.

In the line listed below the logical drive, the state of the individual partitions is displayed in square brackets at the end of the line. A U means that the respective hard drive is integrated into the RAID (up).

In the following example, all logical drives have only one partition mounted which is located on the sda hard drive. The respective partition located on the second hard drive sdb is not included. You can also see this from the entry [_U]. The unmounted partitions of the hard drive sdb indicate that there is an error or a defect on this hard drive.

[root@localhost~]# cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
md0 : active raid1 sda1[1]
102208 blocks [2/1] [_U]

md1 : active raid1 sda2[1]
119684160 blocks [2/1] [_U]

md2 : active raid1 sda3[1]
262016 blocks [2/1] [_U]

unused devices: <none>


In the following example, a defective hard drive is still included in the RAID:

[root@localhost~]# cat /proc/mdstat

Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2](F)
439553856 blocks super 1.0 [2/1] [U_]
bitmap: 1/4 pages [4KB], 65536KB chunk

md1 : active raid1 sdb1[2](F) sda1[0]
19529600 blocks super 1.0 [2/1] [U_]

unused devices: <none>

The entry (F) in this example shows that the partition is marked as faulty.

Diagnose Faults and Find Details Needed for a Hard Drive Replacement

In order to detect hard disk errors, we recommend that you follow these steps:

Install the Smartctl program. Smartctl is a command line program used to monitor volumes using SMART (Self-Monitoring, Analysis and Reporting Technology). With this program, you can check whether a hard drive is defective. It is a component of the Smartmontools. The Smartmontools are available as packages for many Linux distributions.

Please Note

A hard drive defect cannot always be detected by the smart values. Accordingly, we recommend that you always analyze the log file /var/log/messages.

Installing Smartctl

To install Smartctl, type the following command:

CentOS

yum install smartmontools

Ubuntu

sudo apt-get install smartmontools

Viewing Hard Drive Information

To view a list of hard drives, type the following command:

smartctl --scanExample

[root@8E8885C~] # smartctl --scan

/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device

To view detailed information for troubleshooting, type the following command:

smartctl -iHAl error [/dev/sd(a-z)]

Analyzing Results

Analyze the detailed information you received using the smartctl -iHAl error [NAMEOFTHEHARDDRIVE] command. The first section lists information that you can use to identify the hard drive:

=== START OF INFORMATION SECTION ===
Device Model:     HGST HUS722T1TALA604
Serial Number:    WMC6N0K2RW66
LU WWN Device Id: 5 0014ee 004722db0
Firmware Version: RAGNWA07
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri May  3 07:45:14 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

This section displays, among other things, the device model and serial number of the hard drive under test.

In the second section the current state of the hard drive is evaluated by Smartctl. If, for example, the value "Failed" or "UNKNOWN" is displayed instead of "PASSED", you should replace the hard drive as soon as possible.

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

In the third section, the SMART VALUES determined are listed in detail. Next to each current percentage value (VALUE), the worst ever measured value (WORST) and the respective limit value (THRESH) are listed. If the current percentage value (VALUE) or the worst ever measured value (WORST) exceeds the limit value (THRESH), a SMART warning is displayed in the WHEN_FAILED column (e.g. FAILING_NOW).

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED     WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always      0
  3 Spin_Up_Time            0x0027   183   183   021    Pre-fail  Always      3833
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always      9
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always      0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always      0
  9 Power_On_Hours          0x0032   097   097   000    Old_age   Always      2560
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always      0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always      0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always      9
 16 Unknown_Attribute       0x0022   000   200   000    Old_age   Always      26802171994
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always      0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always      4
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always      67
194 Temperature_Celsius     0x0022   116   111   000    Old_age   Always      31
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always      0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always      0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline     0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always      0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline     0

The following details can indicate an imminent hard drive failure before a SMART warning is displayed:

Reallocated_Sector_Ct: Specifies the number of sectors reassigned due to read errors. If a sector can no longer be read, written to or checked correctly, a replacement sector is automatically assigned to it. The faulty sector is permanently marked as unreadable. This is a clear warning sign for incipient surface problems. If this value is not equal to zero, a hard drive failure is often imminent. This value is the most important indicator for a hard drive replacement.

Current_Pending_Sector_Ct: Specifies the number of unstable sectors waiting for remapping. If a sector cannot be read and written correctly, it first receives the status Current Pending Sector. The sector is not reassigned in this state, since the data in the sector are unknown. Only after several unsuccessful read or write attempts is a replacement sector assigned and the faulty sector is permanently marked as unreadable. The value Current_Pending_Sector_Ct is an important indicator for a hard drive replacement. If this value is not equal to zero, a hard drive failure is often imminent.

Offline_Uncorrectable: Specifies the number of uncorrectable write and read sector errors.

The last section deals with the internal hard drive log. Errors are recorded here if the work orders of the Servers have not been processed properly by the hard drive. If the number of errors in this section is at least two digits, you should replace the hard drive as soon as possible.

SMART Error Log Version: 1
No Errors Logged

Required Information for Drive Replacement

The following information is required in order to replace the defective hard drive:

  • Label of the Hard Drive in the RAID (such as sda)

  • Serial Number

  • Model

  • Log File (Optional)

Creating a SMART Log

To create a complete SMART log, type the following command:

smartctl -x [NAMEOFTHEHARDDRIVE]Example:

C:\Program Files\smartmontools\bin>smartctl -x /dev/sda -x

If the hard drive can no longer be accessed using Smartctl, you can pull up the required information using the hdparm program.

How to install hdparm:

CentOS

yum -y install hdparm

Ubuntu/Debian
sudo apt-get update
sudo apt-get install hdparm

Then, type the following command to access the information required for drive replacement:

hdparm -i /dev/sda

Please Note
  • If the SMART log was created as described above, it will contain enough sufficient information. You can use it to have the defective hard drive replaced. Please contact the customer service once you have it.

  • If you cannot find the serial number of the defective hard drive using Smartctl, you can alternatively provide customer service with the serial number of the working hard drive(s).

Preparing for Hard Drive Replacement

The following example assumes that the second hard drive (sdb) needs to be replaced. During the status check, for example, the following status of the software RAID is displayed:

[root@host~]# cat /proc/mdstat

Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2]
439553856 blocks super 1.0 [2/1] [UU]

md1 : active raid1 sdb1[2] sda1[0]
19529600 blocks super 1.0 [2/1] [UU]

unused devices: <none>

In this example, the second hard drive (sdb) is still integrated in the RAID and is therefore still in operation.

Manually Mark Raid Device as "Faulty" to Remove it from the RAID

To mark the defective drive as faulty so that it can be removed from the RAID, enter the following command:

[root@host~]# mdadm PATHm_OFTHE_RAID_ARRAYS -f PATH_OFTHE_HARD_DRIVE

In the examples below, the hard drives sdb3 and sdb1 are marked as faulty:

[root@host~]# mdadm /dev/md3 -f /dev/sdb3
mdadm: set /dev/sdb3 faulty in /dev/md3

[root@host~]# mdadm /dev/md1 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md1

After entering the command, the RAID has the following status:

[root@host~]# cat /proc/mdstat

Personalities : [raid1]
md3 : active raid1 sda3[0] sdb3[2](F)
439553856 blocks super 1.0 [2/1] [U_]

md1 : active raid1 sdb1[2](F) sda1[0]
19529600 blocks super 1.0 [2/1] [U_]

unused devices: <none>

Remove Partition/ from the Multiple Device

To remove a partition from the Multiple Device, type the following command:

[root@host~]# mdadm -r /PATH_OFTHE_RAID_ARRAYS /PATH_OFTHE_HARD_DRIVE

In the examples below, the hard drives sdb3 and sdb1 are removed from the multiple device md3 and md1 respectively:

[root@host~]# mdadm -r /dev/md3 /dev/sdb3
mdadm: hot removed /dev/sdb3 from /dev/md3

[root@host~]# mdadm -r /dev/md1 /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md1

Then check the status of the RAID. In this example, the RAID prepared for drive replacement has the following final state:

[root@host~]# cat /proc/mdstat

Personalities : [raid1]
md3 : active raid1 sda3[0]
439553856 blocks super 1.0 [2/1] [U_]

md1 : active raid1 sda1[0]
19529600 blocks super 1.0 [2/1] [U_]

unused devices: <none>

Checking the Swap Partitions Used

Check which swap partitions are used by the operating system. To do this, enter the following command:

[root@host~]# cat /proc/swaps

Filename Type Size Used Priority
/dev/sda2 partition 9765884 0 -1
/dev/sdb2 partition 9765884 0 -2

Alternatively, you can use the following command to check which swap partitions are defined in fstab:

[root@host~]# grep swap /etc/fstab
/dev/sda2 none swap sw
/dev/sdb2 none swap swap sw

Disable Swap Partition on the Defective Device

Deactivate the swap partition on the defective hard drive so that it can be replaced. To do this, enter the following command:

[root@host~]# swapoff PATH_OFTHE_HARD_DRIVE
Example:

[root@host~]# swapoff /dev/sdb2

Please Note

If the swap partition on the defective hard drive is not deactivated and the hard drive is replaced, the swap partition in /proc/swaps receives the status Deleted.

Requesting the Hard Drive Replacement

The defective hard drive can now be replaced. To request the replacement, please contact 1&1 IONOS Customer Support.

Next Steps

After replacing the defective hard drive, it is necessary to rebuild the software RAID. For more information on how to rebuild a software RAID, click here:

Rebuild Software RAID (Linux)