MD raid 5 kicked out disk: rebuilding needs reboot. why?

Toggle sidebar Toggle sidebar
J

Janpieter Sollie

Honorable
Jun 2, 2013
73
0
10,640
system: gentoo linux k8_sse3, linux 3.0.80
cpu: 2* opteron 265, 14GB ram

situation:
I have a 3-disk RAID5 array with a MD 1.2 superblock over /dev/sdb1, /sdd1 and /dev/sdf1.
/dev/sdb1 got kicked out a few weeks ago because the SATA cable apparently needed a replacement. I tested the drive with smartctl -a, and it was fine. I didn't notice it a few weeks ago because it's on my NAS server and the SMTP server was filled incorrectly.
I tried the following things to re-attach /dev/sdb1 to /dev/md0:
-> mdadm --manage --re-add /dev/md0 /dev/sdb1
-> mdadm --manage --remove /dev/md0 2 (the disk number in the array)
mdadm --manage --re-add /dev/md0 /dev/sdb1
in all of those cases, mdadm responded with "sdb1: device or resource busy"
so I continued with:
-> mdadm --stop /dev/md0
mdadm --assemble -f /dev/sdb1 /sdd1 /dev/sdf1
mdadm responded with failed to start array
-> mdadm --stop /dev/md0
mdadm --assemble -f --uuid=(uuid of the raid)
mdadm only included /dev/sdd1 and /dev/sdf1

so, at this point, I gave up re-attaching /dev/sdb1, and cleared the superblock on /dev/sdb1.
But, when I tried to add the drive to /dev/md0, it still gave me the error message "/dev/sdb1: device or resource busy".
However; after a server reboot, I could add /dev/sdb1 back into the array, and it started rebuilding.
what happened here? linux should not need a reboot for a software raid issue!
 
Solution
Ijack
You don't appear to have set the disk as faulty (mdadm --fail) before trying to remove it from the array. The software won't let you remove an active disk.
Sort by date Sort by votes
Ijack

Ijack

Splendid
Jul 30, 2008
7,099
0
28,010
Correct.

Bear in mind that mdadm lets you create RAID arrays from partitions rather than just raw disks; a failure in one array is not necessarily a failure in all arrays that use the disk. It would not be sensible to automatically fail a disk because of a failure in one partition. So you have to specifically fail the disk (or partition) in all arrays that use it. Then you can remove the disk. The reboot was not necessary.

This is sensible behaviour; it gives the administrator control over the failure process. There are many online articles that document the correct procedure.

As for deliberately clearing the superblock on the disk, I would class that as an extremely risky move.
 
Upvote 0 Downvote
J

Janpieter Sollie

Honorable
Jun 2, 2013
73
0
10,640
what you say sounds pretty logical,
and indeed, on that disk there's /dev/sdb1, /dev/sdb2 and /dev/sdb3 which assemble to 3 different raid 5 arrays because I wanted to mount them at different places on the root filesystem. only /dev/md0 (which includes /dev/sdb1) was degraded.
I did not replace the disk! only the sata cable.
and I did not empty the superblock on /dev/sdb either, I just used mdadm --zero-superblock /dev/sdb1
so if I understand everything right, if I would have stopped /dev/md1 and /dev/md2, it would probably have worked?
 
Upvote 0 Downvote
Ijack

Ijack

Splendid
Jul 30, 2008
7,099
0
28,010
Yes. if you had failed all of the partitions on that drive you would have then been able to remove them from their arrays, then replace the disk, add the partitions back to the arrays and let them rebuild.

It all sounds pretty complicated to me. I'm not a big fan of using partitions in RAID arrays rather than raw disks. But then, I'm not a great fan of RAID in home setups anyway; it solves few problems and presents some new problems of its very own. I used to run RAID 5 and RAID 1 arrays on dozens of servers at work but this was because they required 24x7 uptime. This is rarely a serious requirement in a home network. I've never felt the need to use RAID on my own computers.
 
Upvote 0 Downvote
You must log in or register to reply here.

TRENDING THREADS

Latest posts

Moderators online

Share this page

COMPANY
RESOURCES
FOLLOW
Top Bottom