Re: IMSM Raid 5 always read only and gone after reboot

On Wed, 24 Aug 2011 19:09:13 +0200 Iwan Zarembo [off-list ref] wrote:

Hi Everyone,
Nothing worked with the last mail, so I tried it again. A different 
approach.
What I tried again:

1. I stopped and deleted the array using:
mdadm --stop /dev/md126
mdadm --stop /dev/md127
mdadm --remove /dev/md127
mdadm --zero-superblock /dev/sdb
mdadm --zero-superblock /dev/sdc
mdadm --zero-superblock /dev/sdd
mdadm --zero-superblock /dev/sde

2. I deleted all data (including partition table) on every HDD:
dd if=/dev/zero of=/dev/sd[b-e] bs=512 count=1

3. Checked if mdadm --assemble --scan can find any arrays, but I did not 
find anything.

4. I created the array again using 
https://raid.wiki.kernel.org/index.php/RAID_setup#External_Metadata
mdadm --create --verbose /dev/md/imsm /dev/sd[b-e] --raid-devices 4 
--metadata=imsm
mdadm --create --verbose /dev/md/raid /dev/md/imsm --raid-devices 4 
--level 5

The new Array did not have any partitions, since I deleted everything. 
So everything looks good.
The details are:
# mdadm -D /dev/md127
/dev/md127:
         Version : imsm
      Raid Level : container
   Total Devices : 4

Working Devices : 4


            UUID : 790217ac:df4a8367:7892aaab:b822d6eb
   Member Arrays :

     Number   Major   Minor   RaidDevice

        0       8       16        -        /dev/sdb
        1       8       32        -        /dev/sdc
        2       8       48        -        /dev/sdd
        3       8       64        -        /dev/sde

# mdadm -D /dev/md126
/dev/md126:
       Container : /dev/md/imsm, member 0
      Raid Level : raid5
      Array Size : 2930280448 (2794.53 GiB 3000.61 GB)
   Used Dev Size : 976760320 (931.51 GiB 1000.20 GB)
    Raid Devices : 4
   Total Devices : 4

           State : clean
  Active Devices : 4
Working Devices : 4
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-asymmetric
      Chunk Size : 128K


            UUID : 4ebb43fd:6327cb4e:2506b1d3:572e774e
     Number   Major   Minor   RaidDevice State
        0       8       48        0      active sync   /dev/sdd
        1       8       32        1      active sync   /dev/sdc
        2       8       16        2      active sync   /dev/sdb
        3       8       64        3      active sync   /dev/sde

# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
[raid4] [raid10]
md126 : active (read-only) raid5 sde[3] sdb[2] sdc[1] sdd[0]
       2930280448 blocks super external:/md127/0 level 5, 128k chunk, 
algorithm 0 [4/4] [UUUU]
           resync=PENDING

md127 : inactive sde[3](S) sdd[2](S) sdc[1](S) sdb[0](S)
       836 blocks super external:imsm

unused devices: <none>

Then I stored the configuration of the array using the command
mdadm --examine --scan >> /etc/mdadm.conf

5. I used dpkg-reconfigure mdadm to make sure mdadm starts properly at 
boot time.

6. I rebooted and checked if the array was created in BIOS of the Intel 
raid.
Yes it is existing, and it looks good there.

7. I still cannot see the created array. But in palimpsest I see that my 
four hard drives are a part of a raid.

I also checked the logs for any strange entries, but no success :S

9. I used mdadm --assemble --scan to see the array in palimpsest

10. Started sync process using command from 
http://linuxmonk.ch/trac/wiki/LinuxMonk/Sysadmin/SoftwareRAID#CheckRAIDstate

#echo active > /sys/block/md126/md/array_state

#cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] 
[raid4] [raid10]
md126 : active raid5 sdd[3] sdc[2] sdb[1] sde[0]
       2930280448 blocks super external:/md127/0 level 5, 128k chunk, 
algorithm 0 [4/4] [UUUU]
       [>....................]  resync =  0.9% (9029760/976760320) 
finish=151.7min speed=106260K/sec

The problem is that the raid was gone after a restart. So I did step 9 
and 10 again.

11. Then I started to create a gtp partition table with parted.
Unfortunately mktable gpt on device /dev/md/raid (or the target of the 
link /dev/md126) never came back. Even after a few hours.

I really do not know what else I need to do to get the raid working. Can 
someone help me? I do not think I am the first person having trouble 
with it :S

It sounds like mdmon is not being started.
mdmon monitors the array and performs any metadata updates required.

The reason mktable is taking more than a second is that it tries to write to
the array, the kernel marks the array as 'write-pending' and waits for mdmon
to notice, update the metadata, and switch the array to 'active'.  But mdmon
never does that.

mdmon should be started by mdadm but just to check you can start it by hand:

 /sbin/mdmon md126
or
 /sbin/mdmon --all

If this makes it work, you need to work out why mdmon isn't being started.

NeilBrown