RAID5 - can't assemble array of 5

Hi All,

I've 5 disks array 3TB each, but one of the disks /dev/sdb can't be recognized.
Using mdadm - v3.4 - 28th January 2016 on Debian GNU/Linux 9.11 (stretch).

So, I tried to assemble the array with 4 disks, but it didn't work, see below:

sudo mdadm --assemble --verbose /dev/md0 /dev/sd[acde]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 4.
mdadm: ignoring /dev/sdc as it reports /dev/sda as failed
mdadm: ignoring /dev/sdd as it reports /dev/sda as failed
mdadm: ignoring /dev/sde as it reports /dev/sda as failed
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: no uptodate device for slot 2 of /dev/md0
mdadm: no uptodate device for slot 3 of /dev/md0
mdadm: no uptodate device for slot 4 of /dev/md0
mdadm: added /dev/sda to /dev/md0 as 0
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.

Also tried to get more details and noticed some inconsistencies
related to update times, etc:
# mdadm --examine /dev/sd[acde]

/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 90dee075:e0f72577:2bc039c0:a6287075
           Name : andromedabr:0  (local to host andromedabr)
  Creation Time : Thu Nov 21 00:37:08 2019
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
     Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=944 sectors
          State : clean
    Device UUID : 4cc19a0b:912f64bb:f51c74bf:abace527

    Update Time : Mon Jul 12 21:16:46 2021
       Checksum : 79ab8eb2 - correct
         Events : 600481

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 90dee075:e0f72577:2bc039c0:a6287075
           Name : andromedabr:0  (local to host andromedabr)
  Creation Time : Thu Nov 21 00:37:08 2019
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
     Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=944 sectors
          State : active
    Device UUID : 35e0f4ea:89dfbd34:051ee297:d22e25b7

    Update Time : Fri Oct 15 09:27:03 2021
       Checksum : 3a8a6dd3 - correct
         Events : 597632

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 90dee075:e0f72577:2bc039c0:a6287075
           Name : andromedabr:0  (local to host andromedabr)
  Creation Time : Thu Nov 21 00:37:08 2019
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
     Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=944 sectors
          State : clean
    Device UUID : 1ac77da2:86d63b86:5ad27b38:7c6555de

    Update Time : Fri Oct 15 11:13:37 2021
       Checksum : b5c5acd - correct
         Events : 600481

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : .A.AA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 90dee075:e0f72577:2bc039c0:a6287075
           Name : andromedabr:0  (local to host andromedabr)
  Creation Time : Thu Nov 21 00:37:08 2019
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 5860271024 (2794.39 GiB 3000.46 GB)
     Array Size : 11720540160 (11177.58 GiB 12001.83 GB)
  Used Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=944 sectors
          State : clean
    Device UUID : 5d468b76:81c27d1b:455adc02:2545ccee

    Update Time : Fri Oct 15 11:13:37 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 4f8b2de7 - correct
         Events : 600481

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : .A.AA ('A' == active, '.' == missing, 'R' == replacing)

Please, let me know if there is anything I can do to fix this issue.

Thanks,
Romulo.

Hi Romulo,

On 10/16/21 9:11 PM, Romulo Albuquerque wrote:

Hi All,

I've 5 disks array 3TB each, but one of the disks /dev/sdb can't be recognized.
Using mdadm - v3.4 - 28th January 2016 on Debian GNU/Linux 9.11 (stretch).

So, I tried to assemble the array with 4 disks, but it didn't work, see below >
sudo mdadm --assemble --verbose /dev/md0 /dev/sd[acde]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 4.
mdadm: ignoring /dev/sdc as it reports /dev/sda as failed
mdadm: ignoring /dev/sdd as it reports /dev/sda as failed
mdadm: ignoring /dev/sde as it reports /dev/sda as failed
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: no uptodate device for slot 2 of /dev/md0
mdadm: no uptodate device for slot 3 of /dev/md0
mdadm: no uptodate device for slot 4 of /dev/md0
mdadm: added /dev/sda to /dev/md0 as 0
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.

[trim /]

This didn't work because /dev/sda dropped out of your array back in the 
middle of July.  You were running degraded for two months before 
/dev/sdb dropped out.

If /dev/sdb is totally dead, you will have to use a two-month-old device 
to revive your array, with data corruption for anything written in the 
past two months.  Add a --force to your assemble command to do so.

I recommend trying harder to recover /dev/sdb onto another device.  Then 
assemble that replacement with the other three, leaving out /dev/sda. 
Also using --force.

Finally, investigate why /dev/sda dropped out in July, and why you 
didn't get an email from mdmon.  (Timeout mismatch for the former, 
perhaps, and incomplete configuration for the latter, likely.)

Phil

Shoot, my brain is fried. Three months, not two. /:

On 10/17/21 9:21 AM, Phil Turmel wrote:

Hi Romulo,

On 10/16/21 9:11 PM, Romulo Albuquerque wrote:

quoted

Hi All,

I've 5 disks array 3TB each, but one of the disks /dev/sdb can't be 
recognized.
Using mdadm - v3.4 - 28th January 2016 on Debian GNU/Linux 9.11 
(stretch).

So, I tried to assemble the array with 4 disks, but it didn't work, 
see below >
sudo mdadm --assemble --verbose /dev/md0 /dev/sd[acde]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sda is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 4.
mdadm: ignoring /dev/sdc as it reports /dev/sda as failed
mdadm: ignoring /dev/sdd as it reports /dev/sda as failed
mdadm: ignoring /dev/sde as it reports /dev/sda as failed
mdadm: no uptodate device for slot 1 of /dev/md0
mdadm: no uptodate device for slot 2 of /dev/md0
mdadm: no uptodate device for slot 3 of /dev/md0
mdadm: no uptodate device for slot 4 of /dev/md0
mdadm: added /dev/sda to /dev/md0 as 0
mdadm: /dev/md0 assembled from 1 drive - not enough to start the array.

[trim /]

This didn't work because /dev/sda dropped out of your array back in the 
middle of July.  You were running degraded for two months before 
/dev/sdb dropped out.

If /dev/sdb is totally dead, you will have to use a two-month-old device 
to revive your array, with data corruption for anything written in the 
past two months.  Add a --force to your assemble command to do so.

I recommend trying harder to recover /dev/sdb onto another device.  Then 
assemble that replacement with the other three, leaving out /dev/sda. 
Also using --force.

Finally, investigate why /dev/sda dropped out in July, and why you 
didn't get an email from mdmon.  (Timeout mismatch for the former, 
perhaps, and incomplete configuration for the latter, likely.)

Phil

On 17/10/2021 14:21, Phil Turmel wrote:

Finally, investigate why /dev/sda dropped out in July, and why you 
didn't get an email from mdmon.  (Timeout mismatch for the former, 
perhaps, and incomplete configuration for the latter, likely.)

Are you running a gui environment? Can you run xosview?

It has options to configure it to display raid status (this wasn't 
working quite right when I reported it ages ago, hopefully it's fixed).

But try and get something like that, that just sits there in the 
background on your desktop, showing machine state. Then if anything goes 
wrong, hopefully your desktop will just "feel wrong" and you'll notice.

Cheers,
Wol