broken raid level 5 array caused by user error

Hi Folks,

I'm running a raid level 5 with 4 devices for some years and tried to grow my array yesterday. I wanted to add two more devices and used the following commands:

mdadm --add /dev/md0 /dev/sdf1 /dev/sdg1
mdadm --grow --raid-devices=6 /dev/md0

So far, so good. Everything seems to work, but after about 2 hours, the reshape progress was still at 0.0% and now, my own stupidity kicked in. I checked the logs via journalctl (I'm running Centos 7) and read something about "main process died" or similar... then I decided to reboot.

After reboot, assembling the array failed:

mdadm: Failed to restore critical section for reshape, sorry. Possibly you needed to specify the --backup-file

But I did not have a backup file and so I panicked and made even worse decisions.

First I tried to assemble the array using --invalid-backup but it did not work. I should stop here and ask but I didn't. I read at some board, that rebuilding the original array with 4 devices will fix my problem. I did not validate this and entered the suggested command:

mdadm -CR /dev/md0 --metadata=1.2 -n4 -l5 -c512 /dev/sd[bcde]1 --assume-clean

But this did not work (the array assembled but I could not access the ext4 filesystem), it seems that I assembled it in the wrong device order, so I also tried different (i.e. all possible) orders, but nothing helped ( I always used --assume-clean).

I guess this is the perfect guide for how _not_ to do it :(

I continued reading and found this:

http://serverfault.com/questions/347606/recover-raid-5-data-after-created-new-array-instead-of-re-using

This gave me some hope and now I wonder, if there is a way to get my data back, maybe the offset is wrong?

Things I know about the array:

metadata: 1.2
Left-symetric
chunk-size: 512

When I run mdadm --detail /dev/md0 it still shows an array size of 6TB, the UUID is also still the same

        Version : 1.2
  Creation Time : Mon Nov  9 00:00:40 2015
     Raid Level : raid5

188 more lines

On Mon, 9 Nov 2015, Mathias Mueller wrote:

Please help me, I know I'm stupid and don't deserve it. I really hope, there is a chance for reovering the array.

It's possible. There are numerous threads in the mailing list archives with people who have been in the same situation as you are in now. Without having a backup of the original superblocks, you're going to have to guess offsets. It might make sense to investigate what the defaults were for the mdadm you originally created the array with.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

On 11/09/2015 06:56 AM, Mikael Abrahamsson wrote:

On Mon, 9 Nov 2015, Mathias Mueller wrote:

quoted

Please help me, I know I'm stupid and don't deserve it. I really hope,
there is a chance for reovering the array.

It's possible. There are numerous threads in the mailing list archives
with people who have been in the same situation as you are in now.
Without having a backup of the original superblocks, you're going to
have to guess offsets. It might make sense to investigate what the
defaults were for the mdadm you originally created the array with.

Chunks size defaults have varied, too.

I would search the known first drive for an ext4 filesystem signature
and work backwards from there.  Show the output of this search:

dd if=/dev/sdd1 bs=1M count=1M |hexdump -C |grep '  53 ef ' |head -n 1000

Phil