On 13/11/2013 23:10, Justin Piszcz wrote:

I attempted to test both here:
http://home.comcast.net/~jpiszcz/20131113/joystick_cmds.txt

The --re-add did not work btw.

Unfortunately the --re-add HAD to work for the test to be of any value.

Like you did, with just -a (--add), it resynced completely when you 
added the old device back in, similarly to when you run repair.
After that you had two exactly identical devices on the RAID, so, no 
wonder the two md5sums of the two legs resulted to be identical.
Also mismatch_cnt went to zero again, and will need to grow again to 
some significant value before you can repeat the test.

The reason for which --re-add failed seems to be (I did a test on our 
machines) that you need to also --remove the device after --fail, so it's:

when removing:
     mdadm /dev/md1 --fail /dev/sda2
mdadm: set /dev/sdb2 faulty in /dev/md1
     mdadm /dev/md1 --remove /dev/sda2
         mdadm: hot removed /dev/sdb2 from /dev/md1

.... compute md5sums ...

when re-adding:
     mdadm /dev/md1 --re-add /dev/sda2
         mdadm: re-added /dev/sda2

you actually did remove the drive, but after that you did not retry with 
--re-add, you went straight to --add which fully replicated the content 
of sda2 on sdb2

Also: does the array already have a bitmap? It's not clear from the log. 
If it does not have a bitmap, even --re-add will replicate all content, 
so you really need a bitmap for this test
To add a bitmap you can do:
mdadm /dev/md1 --grow --bitmap=internal
at the beginning of the test.

What kernel version is yours?

Regards
J.