A failed rebuild, a hung build percentage, or loss of a second drive while rebuilding is probably the worst point in an array’s life, as most users will want to just retry, but trying again is where a single disk problem becomes a total loss. Here’s why rebuilds fail, the three moves that make things worse, and how to keep a wounded array recoverable.
All of your surviving disk(s) will be put under the maximum amount of read load (reading all sectors on each disk). This can occur with disks of the same age and condition as the disk that has just died. The rebuild process itself can also act like a “stress test” but at a time when you have the least amount of resources available.
A read error on a survivor. Rebuilding RAID 5 requires reading everything from every surviving disk. If there is a single bad sector on one of the surviving disks, then many controller systems will either abort the rebuild or worse yet they may drop that disk too. Today’s high capacity disks make the chances of encountering at least one unreadable sector during a complete rebuild of an entire array very significant. Therefore, larger rebuild processes fail significantly more times than smaller rebuild processes used to fail. The batch-twin problem. When an array degrades due to wear and tear on a single disk, the other disks in the array that were purchased at the same time and exposed to the exact same workload are now the next most likely to fail. The rebuild process places an extremely heavy workload on these drives. The wrong-drive accident. In extreme situations, a healthy drive is removed in place of a failed drive, or a hot-spare is automatically started rebuilding against a failing survivor. These types of errors are caused by humans and account for a surprisingly large percentage of the arrays that we see in our lab. There are different reasons behind this type of error; however, all of these errors share one common trait: once the original failure occurs, the data remains intact on the remaining disks unless the next move destroys it.
Force online. A member was removed from an array and then forced back onto the array at some point when it was down. This action combines stale data into a live array. Following this action, the “repair” will create garbage in every stripe. Re-initialise / recreate the array. All controllers offer this option as the “fresh start” option. It creates new metadata (and possibly new parity) across all members of the array, over the existing data in your volume. At this time, the array has been repaired; however, your data is now a recovery job. Serial retries. When rebuilding fails due to a failure of one or more disks, you reboot the system, and attempt to rebuild once again. Each subsequent rebuild will run through the same high intensity reads against drives that have already shown their inability to sustain such reads. If the first rebuild did not succeed, the second rebuild is essentially using up the last chances of your data. Any time you are tempted to use either of these options is the exact moment to turn off your systems.
The protocol fits on a label-maker. Power down your server: a degraded array that is at rest is stable; an identical array under heavy load is failing. Photograph and label every disk in your system with its bay number prior to removing any disks. Member order is crucial to successfully reconstructing your lost data. Change nothing in your controller: do not save any configuration, do not perform any repairs or firmware updates while you are attempting to recover from a failure. Next, we follow a traditional recovery sequence: every member of your array must be imaged read-only (including the “bad” member, which often retains 95% readability on less demanding hardware and contains the missing stripes from the surviving members) and then your array is reassembled virtually from those images allowing different parameters to be tested without losing any data. Once your data is extracted from the virtual copy of your array, the physical disks are never exposed to the risk of further damage. Our RAID recovery work involves recovering data from both rack servers and desktop NAS boxes, and the deeper maths behind why arrays fail is covered in our RAID 10 guide.
If there are files in this RAID you cannot afford to lose and you do not have a valid backup: No. Pause the rebuild if the controller will allow it. Either way, stop placing additional load on the array. This type of rebuild usually indicates that one of the surviving drives is experiencing read errors. Hours of intense reading are all that is required for another drive to fail. A paused/degraded RAID is recoverable. However, a rebuild that caused both drives to fail probably won’t leave enough of the RAID to make it fully recoverable.
Pressing “force” to bring a drive back into service is likely the worst action anyone can take against their RAID. When the controller dropped a drive, that was at some point in time. The array has continued to change since that time. Once you press force to bring that drive back, the controller may use stale data to “fix” other parts of the array using the now-corrupted parity information. The best course of action, if the array matters, is to turn off the system and create a disk image of each drive. The array can then be recreated virtually, and virtual errors will not result in lost data.
The same bench and the same method are used. It’s essentially the same architecture: a four bay Synology or QNAP sitting under a desk and being used as a mini-RAID server (mdadm or a vendor-specific version on standard disks). Send just the labelled drives (you don’t need the enclosures): each drive is imaged, then the logical structure of the array is created from those images and each file within the array is restored one-by-one. More information regarding this can be found on our NAS recovery page.
We’ll provide a free 48-hour diagnostic on all multi-disk RAID systems. Each drive will be imaged prior to running any RAID-related logic, we’ll also write out a quote prior to doing any actual work.