On Sun, 8 Dec 2013 21:04:42 +0100 (CET) Tomas Agartz [off-list ref] wrote:

After booting a server that had been powered off for some time, the 4-disk 
raid5 device was up and running in read-only mode with one disk missing. 
After a, in hindsight, hasty decision, "mdadm --manage --add /dev/md0 
/dev/sdd" was executed to re-add the missing device to the array.

At this time, all hell broke loose :) The first thing that happened was 
that sdd was added as a spare instead of re-added as expected. The second 
thing was that a different disk, sdb, was kicked from the array because of 
read/sata-bus errors. The root disk also bailed and the system had to be 
powercycled.

If you want to re-add, it is safest to ask mdadm to --re-add, not to --add.

The real problem, from the start, was probably that sdb was bad all along, 
but from some reason sdd was the device missing from the array after the 
initial boot.

Trying to read data from sdb gives read errors and timeouts, but I was 
able to do "mdadm --examine" after resetting the sata port.

The current state is that, out of 4 disks two are good (sde and sdf), one 
is (in error) marked as a spare (sdd), and the fourth device is unusable 
(sdb).

What is the correct method do change the spare disk back to a data disk 
and try to restart the array with 3 out of 4 devices (sdd, sde and sdf)?

The only real option at this point is to --create the array.  There isn't
enough information for mdadm to be able  to do anything clever.

The device has never had a spare, so I think that sdd used to be "Active 
device 0" before this happened?

Possibly relevant data from mdadm --examine on the four devices:

sdb          State : clean
sdb         Events : 333560
sdb   Device Role : Active device 3
sdb   Array State : .AAA ('A' == active, '.' == missing)

sdd          State : clean
sdd         Events : 333562
sdd   Device Role : spare
sdd   Array State : .AA. ('A' == active, '.' == missing)

sde          State : clean
sde         Events : 333562
sde   Device Role : Active device 1
sde   Array State : .AA. ('A' == active, '.' == missing)

sdf          State : clean
sdf         Events : 333562
sdf   Device Role : Active device 2
sdf   Array State : .AA. ('A' == active, '.' == missing)

If no one else has any better suggestions, my best guess would be to: 
"mdadm --create /dev/md0 --level=5 --raid-devices=4 --assume-clean 
/dev/sdd /dev/sde /dev/sdf missing" (the device was created with default 
values, metadata 1.2, chunk size 512K, layout left-symmetric).

Check the "Data Offset" of the devices and make sure the newly created array
gets the same "Data Offset" (it can explicitly be set with the latest mdadm).

NeilBrown

(Other crazy ideas involve editing the superblock of sdd and making it 
device 0 and then trying to start the array after that).

Best regards,
Tomas
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html