Superblock Missing

Respectfully requesting some help.

Very similar to what MasterPrenium posted last month, I'm struggling
to recover from a lost superblock after a grow operation.

I had a raid 5 array running since 2012 (metadata version .90)   I
recently grew it from 2TB drives to 4TB.(metadata version 1.2)    The
grow was successful and the array ran for approximately 2 weeks.
After that I booted to a LiveCD, changed out a boot drive (not in the
array) reinstalled Grub, etc.
After doing the work I rebooted and attempted to load the array.
With the new boot drive I didn't realize it at the time, but I was
missing the raid456 module.   The array wouldn't load.    I fixed the
kernel & module issues, and attempted to load but it said missing
superblock info.  I was dumbfounded because the array had nothing to
do with the boot/OS drive.

Problem: Superblock not found

The hard drives are all good.  All the logs looked clean, and after
several troubleshooting steps I eventually I ran the command

#mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1

The array immediately went into a sync which completed successfully
the next day.    Still no superblock.


The array starts fine now, but still no superblock.

mdadm --assemble  /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1
mdadm: /dev/md0 has been started with 3 drives.
virtual raid # dmesg
[177604.370207] md: md0 stopped.
[177604.385059] md: bind<sdd1>
[177604.385425] md: bind<sde1>
[177604.385591] md: bind<sdc1>
[177604.415558] md/raid:md0: device sdc1 operational as raid disk 0
[177604.415561] md/raid:md0: device sde1 operational as raid disk 2
[177604.415562] md/raid:md0: device sdd1 operational as raid disk 1
[177604.415850] md/raid:md0: allocated 3316kB
[177604.417538] md/raid:md0: raid level 5 active with 3 out of 3
devices, algorithm 2
[177604.417540] RAID conf printout:
[177604.417541]  --- level:5 rd:3 wd:3
[177604.417543]  disk 0, o:1, dev:sdc1
[177604.417544]  disk 1, o:1, dev:sdd1
[177604.417545]  disk 2, o:1, dev:sde1
[177604.417637] created bitmap (30 pages) for device md0
[177604.418110] md0: bitmap initialized from disk: read 2 pages, set 0
of 59518 bits
[177605.200213] md0: detected capacity change from 0 to 7988370735104


# mdadm --examine /dev/md0
mdadm: No md superblock detected on /dev/md0.


As a troubleshooting step I tried:

mdadm --grow /dev/md0 --size max
mdadm: component size of /dev/md0 unchanged at 3900571648K

I believe this shows the pre-grow size of 4TB, not the current correct
amount of 8TB

Any ideas (other than restore) would be appreciated.



Requested from "When Things Go Wrong"

-------------------------------------
# mdadm --examine /dev/sd[cde]1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
           Name : virtual:0  (local to host virtual)
  Creation Time : Sat Aug 26 16:21:20 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7801143296 (3719.88 GiB 3994.19 GB)
     Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 77580f12:adf88476:d9c1448b:b041443f

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Aug 31 22:21:32 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : adbbd4f2 - correct
         Events : 6261

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
           Name : virtual:0  (local to host virtual)
  Creation Time : Sat Aug 26 16:21:20 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7801143296 (3719.88 GiB 3994.19 GB)
     Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 1cb11ccc:2aeb095d:6be1838b:7d7b33b7

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Aug 31 22:21:32 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : c67c7975 - correct
         Events : 6261

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
           Name : virtual:0  (local to host virtual)
  Creation Time : Sat Aug 26 16:21:20 2017
     Raid Level : raid5
   Raid Devices : 3

 Avail Dev Size : 7801143296 (3719.88 GiB 3994.19 GB)
     Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : c322a4fb:ec99f835:0ce9cb45:ce6b376d

Internal Bitmap : 8 sectors from superblock
    Update Time : Thu Aug 31 22:21:32 2017
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 3f3e91d2 - correct
         Events : 6261

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)

------------------------------------------------

# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sat Aug 26 16:21:20 2017
     Raid Level : raid5
     Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
  Used Dev Size : 3900571648 (3719.88 GiB 3994.19 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Aug 31 22:21:32 2017
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : virtual:0  (local to host virtual)
           UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
         Events : 6261

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       3       8       65        2      active sync   /dev/sde1

---------------------------------------------------------------------------

# ./lsdrv
PCI [ahci] 00:17.0 SATA controller: Intel Corporation Sunrise Point-H
SATA controller [AHCI mode] (rev 31)
├scsi 0:0:0:0 ATA      HGST HTS541010A7 {S0A001SSGRU38K}
│└sda 931.51g [8:0] Partitioned (dos)
│ ├sda1 915.64g [8:1] ext4 {119f7409-bb57-4c82-a526-286a639779d6}
│ │└Mounted as /dev/sda1 @ /
│ ├sda2 1.00k [8:2] Partitioned (dos)
│ └sda5 15.87g [8:5] swap {f2217412-d4d6-4a6e-b699-7f2978db9423}
├scsi 1:0:0:0 ATA      ST500LM012 HN-M5 {S2SVJ9CC509140}
│└sdb 465.76g [8:16] Partitioned (dos)
│ └sdb1 465.76g [8:17] ext4 {f155bd77-8924-400c-bab4-04b641338282}
├scsi 2:0:0:0 ATA      WDC WD40EZRZ-00W {WD-WCC4E0XVVCXH}
│└sdc 3.64t [8:32] Partitioned (gpt)
│ └sdc1 3.63t [8:33] MD raid5 (0/3) (w/ sdd1,sde1) in_sync 'virtual:0'
{964fd303-748b-8f0e-58f3-84ce3fde97cc}
│  └md0 7.27t [9:0] MD v1.2 raid5 (3) clean, 512k Chunk
{964fd303:748b8f0e:58f384ce:3fde97cc}
│                   Empty/Unknown
├scsi 3:x:x:x [Empty]
├scsi 4:0:0:0 ATA      WDC WD40EZRX-00S {WD-WCC4E0992649}
│└sdd 3.64t [8:48] Partitioned (gpt)
│ └sdd1 3.63t [8:49] MD raid5 (1/3) (w/ sdc1,sde1) in_sync 'virtual:0'
{964fd303-748b-8f0e-58f3-84ce3fde97cc}
│  └md0 7.27t [9:0] MD v1.2 raid5 (3) clean, 512k Chunk
{964fd303:748b8f0e:58f384ce:3fde97cc}
│                   Empty/Unknown
└scsi 5:0:0:0 ATA      WDC WD40EZRZ-00W {WD-WCC4E6CV0S5R}
 └sde 3.64t [8:64] Partitioned (gpt)
  └sde1 3.63t [8:65] MD raid5 (2/3) (w/ sdc1,sdd1) in_sync 'virtual:0'
{964fd303-748b-8f0e-58f3-84ce3fde97cc}
   └md0 7.27t [9:0] MD v1.2 raid5 (3) clean, 512k Chunk
{964fd303:748b8f0e:58f384ce:3fde97cc}
                    Empty/Unknown
PCI [pata_pdc2027x] 04:01.0 Mass storage controller: Promise
Technology, Inc. 20269 (rev 02)
├scsi 6:0:0:0 LITE-ON  DVDRW SHW-160P6S {LITE-ON_DVDRW_SHW-160P6S}
│└sr0 1.00g [11:0] Empty/Unknown
└scsi 7:x:x:x [Empty]
Other Block Devices
├loop0 0.00k [7:0] Empty/Unknown
├loop1 0.00k [7:1] Empty/Unknown
├loop2 0.00k [7:2] Empty/Unknown
├loop3 0.00k [7:3] Empty/Unknown
├loop4 0.00k [7:4] Empty/Unknown
├loop5 0.00k [7:5] Empty/Unknown
├loop6 0.00k [7:6] Empty/Unknown
├loop7 0.00k [7:7] Empty/Unknown
├md127 0.00k [9:127] MD vnone  () clear, None (None) None {None}
│                    Empty/Unknown
├ram0 64.00m [1:0] Empty/Unknown
├ram1 64.00m [1:1] Empty/Unknown
├ram2 64.00m [1:2] Empty/Unknown
├ram3 64.00m [1:3] Empty/Unknown
├ram4 64.00m [1:4] Empty/Unknown
├ram5 64.00m [1:5] Empty/Unknown
├ram6 64.00m [1:6] Empty/Unknown
├ram7 64.00m [1:7] Empty/Unknown
├ram8 64.00m [1:8] Empty/Unknown
├ram9 64.00m [1:9] Empty/Unknown
├ram10 64.00m [1:10] Empty/Unknown
├ram11 64.00m [1:11] Empty/Unknown
├ram12 64.00m [1:12] Empty/Unknown
├ram13 64.00m [1:13] Empty/Unknown
├ram14 64.00m [1:14] Empty/Unknown
└ram15 64.00m [1:15] Empty/Unknown

-----------------------------------------------------------------

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md0 : active raid5 sdc1[0] sde1[3] sdd1[1]
      7801143296 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>

---------------------------------------------------------------
# smartctl --xall /dev/sdc
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-21-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD40EZRZ-00WN9B0
Serial Number:    WD-WCC (removed)
LU WWN Device Id: 5 0014ee 261e42d67
Firmware Version: 80.00A80
User Capacity:    4,000,753,476,096 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Aug 31 22:30:51 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test
routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (53280) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 532) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x7035)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   204   181   021    -    6758
  4 Start_Stop_Count        -O--CK   100   100   000    -    120
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   079   079   000    -    15652
 10 Spin_Retry_Count        -O--CK   100   100   000    -    0
 11 Calibration_Retry_Count -O--CK   100   100   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    120
192 Power-Off_Retract_Count -O--CK   200   200   000    -    83
193 Load_Cycle_Count        -O--CK   170   170   000    -    92101
194 Temperature_Celsius     -O---K   123   109   000    -    29
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   200   200   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  SATA NCQ Queued Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      39  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xcf       GPL     VS   65535  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    29 Celsius
Power Cycle Min/Max Temperature:     29/32 Celsius
Lifetime    Min/Max Temperature:     19/43 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (109)

Index    Estimated Time   Temperature Celsius
 110    2017-08-31 14:33    29  **********
 ...    ..(476 skipped).    ..  **********
 109    2017-08-31 22:30    29  **********

SCT Error Recovery Control command not supported

Device Statistics (GP/SMART Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           10  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           10  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       178857  Vendor specific

# smartctl --xall /dev/sdd
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-21-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD40EZRX-00SPEB0
Serial Number:    WD-WCC(removed)
LU WWN Device Id: 5 0014ee 2b47635c2
Firmware Version: 80.00A80
User Capacity:    4,000,753,476,096 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Aug 31 22:30:54 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test
routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (52020) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 520) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x7035)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   202   171   021    -    6900
  4 Start_Stop_Count        -O--CK   092   092   000    -    8907
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   100   253   000    -    0
  9 Power_On_Hours          -O--CK   075   075   000    -    18350
 10 Spin_Retry_Count        -O--CK   100   100   000    -    0
 11 Calibration_Retry_Count -O--CK   100   100   000    -    0
 12 Power_Cycle_Count       -O--CK   099   099   000    -    1075
192 Power-Off_Retract_Count -O--CK   200   200   000    -    88
193 Load_Cycle_Count        -O--CK   185   185   000    -    46420
194 Temperature_Celsius     -O---K   119   098   000    -    33
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   200   200   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  SATA NCQ Queued Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb7  GPL,SL  VS       1  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xcf       GPL     VS   65535  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    33 Celsius
Power Cycle Min/Max Temperature:     33/36 Celsius
Lifetime    Min/Max Temperature:     13/54 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (443)

Index    Estimated Time   Temperature Celsius
 444    2017-08-31 14:33    34  ***************
 ...    ..(426 skipped).    ..  ***************
 393    2017-08-31 21:40    34  ***************
 394    2017-08-31 21:41    33  **************
 ...    ..(  6 skipped).    ..  **************
 401    2017-08-31 21:48    33  **************
 402    2017-08-31 21:49    34  ***************
 403    2017-08-31 21:50    34  ***************
 404    2017-08-31 21:51    33  **************
 ...    ..( 32 skipped).    ..  **************
 437    2017-08-31 22:24    33  **************
 438    2017-08-31 22:25    34  ***************
 ...    ..(  4 skipped).    ..  ***************
 443    2017-08-31 22:30    34  ***************

SCT Error Recovery Control command not supported

Device Statistics (GP/SMART Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           10  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           10  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       178859  Vendor specific

# smartctl --xall /dev/sde
smartctl 6.5 2016-01-24 r4214 [x86_64-linux-4.4.0-21-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD40EZRZ-00WN9B0
Serial Number:    WD-WCC(removed)
LU WWN Device Id: 5 0014ee 2b73a30c5
Firmware Version: 80.00A80
User Capacity:    4,000,753,476,096 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Aug 31 22:30:56 2017 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)    The previous self-test
routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:         (50160) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 502) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x7035)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR-K   200   200   051    -    0
  3 Spin_Up_Time            POS--K   195   169   021    -    7208
  4 Start_Stop_Count        -O--CK   100   100   000    -    114
  5 Reallocated_Sector_Ct   PO--CK   200   200   140    -    0
  7 Seek_Error_Rate         -OSR-K   200   200   000    -    0
  9 Power_On_Hours          -O--CK   079   079   000    -    15651
 10 Spin_Retry_Count        -O--CK   100   100   000    -    0
 11 Calibration_Retry_Count -O--CK   100   100   000    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    114
192 Power-Off_Retract_Count -O--CK   200   200   000    -    77
193 Load_Cycle_Count        -O--CK   170   170   000    -    92696
194 Temperature_Celsius     -O---K   122   107   000    -    30
196 Reallocated_Event_Count -O--CK   200   200   000    -    0
197 Current_Pending_Sector  -O--CK   200   200   000    -    0
198 Offline_Uncorrectable   ----CK   200   200   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   200   000    -    0
200 Multi_Zone_Error_Rate   ---R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      6  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  SATA NCQ Queued Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa0-0xa7  GPL,SL  VS      16  Device vendor specific log
0xa8-0xb6  GPL,SL  VS       1  Device vendor specific log
0xb7       GPL,SL  VS      39  Device vendor specific log
0xbd       GPL,SL  VS       1  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xc1       GPL     VS      93  Device vendor specific log
0xcf       GPL     VS   65535  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (6 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       258 (0x0102)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    30 Celsius
Power Cycle Min/Max Temperature:     30/33 Celsius
Lifetime    Min/Max Temperature:     20/45 Celsius
Under/Over Temperature Limit Count:   0/0
Vendor specific:
01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -41/85 Celsius
Temperature History Size (Index):    478 (52)

Index    Estimated Time   Temperature Celsius
  53    2017-08-31 14:33    31  ************
 ...    ..( 87 skipped).    ..  ************
 141    2017-08-31 16:01    31  ************
 142    2017-08-31 16:02    32  *************
 ...    ..( 18 skipped).    ..  *************
 161    2017-08-31 16:21    32  *************
 162    2017-08-31 16:22    31  ************
 ...    ..( 93 skipped).    ..  ************
 256    2017-08-31 17:56    31  ************
 257    2017-08-31 17:57    30  ***********
 ...    ..(  7 skipped).    ..  ***********
 265    2017-08-31 18:05    30  ***********
 266    2017-08-31 18:06    31  ************
 267    2017-08-31 18:07    31  ************
 268    2017-08-31 18:08    30  ***********
 ...    ..(  5 skipped).    ..  ***********
 274    2017-08-31 18:14    30  ***********
 275    2017-08-31 18:15    31  ************
 ...    ..(254 skipped).    ..  ************
  52    2017-08-31 22:30    31  ************

SCT Error Recovery Control command not supported

Device Statistics (GP/SMART Log 0x04) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2            9  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           10  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000f  2            0  R_ERR response for host-to-device data FIS, CRC
0x0012  2            0  R_ERR response for host-to-device non-data FIS, CRC
0x8000  4       178860  Vendor specific


Sincerely,

David

On Fri, Sep 01, 2017 at 08:52:23PM -0400, David Mitchell wrote:

# mdadm --examine /dev/md0
mdadm: No md superblock detected on /dev/md0.

Everyone will assume user error at this point.
/dev/md0 is not supposed to have a mdadm superblock to examine.
It has a filesystem or LUKS or LVM header or whatever it is you do.

For --examine you have to provide one or more member devices.

You can do mdadm --detail /dev/md* or mdadm --examine /dev/sd*.

Requested from "When Things Go Wrong"

-------------------------------------
# mdadm --examine /dev/sd[cde]1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2

Oh look. There it is. Your superblock...

Other than that I see no problems. Your Load Cycle Counts are high 
and for WD there are tools to finetune that behaviour but it's 
unclear whether that's actually a real problem.

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%         0         -

Consider running smart tests more regularly.

Regards
Andreas Klauer

Andreas, thank you for your response.

Sorry I'm still confused.   While the array seems alive, the
filesystem seems to be missing?

I'm unable to mount the filesystem.


As a troubleshooting step I recoverjpeg and it did find a bunch of files.

#recoverjpeg -s 1m /dev/md0


# dumpe2fs /dev/md0
dumpe2fs 1.42.13 (17-May-2015)
dumpe2fs: Bad magic number in super-block while trying to open /dev/md0
Couldn't find valid filesystem superblock.



------------------
# mount /dev/md0 /mnt/raid/
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

----------------
dmesg:
[307811.909930] EXT4-fs (md0): VFS: Can't find ext4 filesystem




---------------

# mdadm --detail /dev/md*
/dev/md0:
        Version : 1.2
  Creation Time : Sat Aug 26 16:21:20 2017
     Raid Level : raid5
     Array Size : 7801143296 (7439.75 GiB 7988.37 GB)
  Used Dev Size : 3900571648 (3719.88 GiB 3994.19 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Thu Aug 31 22:21:32 2017
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : virtual:0  (local to host virtual)
           UUID : 964fd303:748b8f0e:58f384ce:3fde97cc
         Events : 6261

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       3       8       65        2      active sync   /dev/sde1
/dev/md127:
        Version :
     Raid Level : raid0
  Total Devices : 0

          State : inactive

    Number   Major   Minor   RaidDevice


-----------------------------------------------------------------------


Sincerely,

David Mitchell


On Fri, Sep 1, 2017 at 9:43 PM, Andreas Klauer
[off-list ref] wrote:

On Fri, Sep 01, 2017 at 08:52:23PM -0400, David Mitchell wrote:

quoted

# mdadm --examine /dev/md0
mdadm: No md superblock detected on /dev/md0.

Everyone will assume user error at this point.
/dev/md0 is not supposed to have a mdadm superblock to examine.
It has a filesystem or LUKS or LVM header or whatever it is you do.

For --examine you have to provide one or more member devices.

You can do mdadm --detail /dev/md* or mdadm --examine /dev/sd*.

quoted

Requested from "When Things Go Wrong"

-------------------------------------
# mdadm --examine /dev/sd[cde]1
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2

Oh look. There it is. Your superblock...

Other than that I see no problems. Your Load Cycle Counts are high
and for WD there are tools to finetune that behaviour but it's
unclear whether that's actually a real problem.

quoted

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining
LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%         0         -

Consider running smart tests more regularly.

Regards
Andreas Klauer

On Sat, Sep 02, 2017 at 11:43:49AM -0400, David Mitchell wrote:

Sorry I'm still confused.
While the array seems alive, the filesystem seems to be missing?

Hello David,

I'm the one who is confused here.

You posted to the raid mailing list about a superblock, and you even 
posted the mdadm command and error message to go along with it. 
How did it turn into a filesystem problem now? In your last mail, 
I did not see that at all.

This was what you posted and I thought you were referring to:

# mdadm --examine /dev/md0
mdadm: No md superblock detected on /dev/md0.

And thus that was the part I referred to in my answer.

----

# mdadm --detail /dev/md*
/dev/md0:
        Version : 1.2
  Creation Time : Sat Aug 26 16:21:20 2017
     Raid Level : raid5

So according to this, your RAID was created about a week ago. 

You did write that you just migrated to 4TB drives. There are various ways 
to go about that, making a new RAID and copying stuff over from the 
old one is not unusual. So this creation time might be perfectly normal.

Of course if this migration happened more than one week ago then I can 
only assume you used mdadm --create to fix some RAID issue or other? 

mdadm --create is like mkfs: you don't expect data to exist afterwards.
For mdadm --create to retain data you have to get all settings perfectly 
right and those settings change depending on when you originally created 
it and what you did with it since (e.g. mdadm --grow might change offsets).

I'm not sure how to help you because nothing in your mails explains 
how this issue came about.

There are previous discussions on this list about botched mdadm --create, 
wrong data offsets, and searching for correct filesystem offsets.

Whatever you do, don't write on these drives while looking for lost data. 

Regards
Andreas Klauer

Let me provide a more detailed history:

Original array 4 drive 2TB Raid 5 array created back in 2012. Over the
years as drives failed, I replaced the 2 of the 2 TB drives with 4TB
drives, so I had two 4TB  and two 2TB drives providing a 6TB usable
array.  (2x4 - P)

About a month ago I put a third 4TB drive in, and a "grow"  from four
drives to three 4TB drives still in a Raid 5.

Once down to three drives, I then grew to the max size.  Everything
worked perfectly, I had 8TB of usable raid 5 with 3 drives.  The file
system passed a e2fsck no problem.   Ran for about 2 weeks, with
absolutely no problems.

With a spare 2TB drive sitting on my desk, I decided to remove my 1tb
OS drive and put in the 2TB drive as the OS drive.  Not to mess up the
array I made sure to zap the UUID, repartition, reformat, etc.  Copied
the files from my 1TB OS drive to the "new" 2TB OS drive via a live
USB key.

Rebooted, ran into grub issues.  Tried to reinstall grub from a live
USB, but was unable to make it work.  I rebooted then did a new Mint
install on the 2TB drive using LVM for the first time.   I was
extremely careful NOT to touch any of 4TB drives in the array during
the install.  The system rebooted and the new OS worked fine.   I did
not start up the array.  I commented out the array in fstab and
mdadm.conf

I then copied my old files from the 1TB drive over to the new 2TB
drive with the exception of the kernel and initrd.  Rebooted and
everything seemed to work ok with my old files.   I went to assemble
and mount the array, specifying the drives.  I would not assemble.
I wasn't sure if using LVM for the first time on my boot drive, but
not my array was the problem.    What I forgot is I booted to the new
kernel and the old image, and the raid456 module wasn't loading.   I
updated the kernel and modules, and rebooted.    The raid456 was now
loading fine, but the array would not load and mount.  Complained
about superblock.  To rule out the LVM issue and kernel I reverted
back to the old 1TB OS drive, so there would be no changes.

The array still would not assemble.  Raid456 did load correctly.  I
looked at all the array logs and everything seemed clean.   The
timestamps on each of the three 4TB raid drives all matched.   There
were no apparent errors, but the array would not assemble.  I really
do NOT remember running the  --create command.  What did work was the
force command:

mdadm --stop /dev/md0
mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1

The array instantly loaded and went into a sync.  I let the sync
complete overnight.    The file system on the raid array seems to be
gone.

I'm looking for any hints on how I might get it back.

The array loads fine, but when I attempted to mount the array, it
complains about not finding the file system

 #mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error

As a troubleshooting step I ran testdisk to see if that might recover
anything.  It doesn't find a partition on /dev/md0, but it did find a
few old floppy image files I had on the array leading me to believe
the data is there.

As a troubleshooting step I ran recoverjpeg  which did find picture
files.  The first time I ran it it showed a bunch of smaller pictures
and they seemed to be fine.   I stopped it an searched only for
pictures above 512k.    The pictures over 512k don't display
correctly.  It only shows part of the pictures.  I'm still searching
the array for other pictures.

At this point I'm hoping for help on next steps in
recovery/troubleshooting.    I'm on the raid mailing list rather than
the file system mailing list because I'm assuming that something got
changed when I forced assembly.   That perhaps the array is not
presenting the filesystem correctly.   Not sure if LDM had anything to
do with this?   At some point from the array metadata version changed.
  I think this was necessary as I moved from 2TB to 4TB disks.

Below is some history out of the mdadm.conf.  It may be missing lines
and steps because of the OS drive swaps, etc.

I've commented these out over time.  The /dev/md/1 occurred after the
OS reinstall.   I also see the UUID changed.  I don't know whether or
not when I changed out the hard drives this caused the UUID's to
change?   Or if the OS installation changed the UUID's. Or if
somewhere along the way I ran a create.  I really don't remember doing
a --create.  Is the change in UUID's the smoking gun that a --create
was done and my data is gone?

mdadm.conf:
# This file was auto-generated on Sun, 06 May 2012 22:55:33 -0400
# by mkconf $Id$
#ARRAY /dev/md0 metadata=0.90 UUID=f5a6ec06:b292289b:1fd984af:668516de
  (original)
#ARRAY /dev/md/1 UUID=3b6899af:93e8cd0b:18b63448:ef8ab23c
           (after OS reinstall?)
#ARRAY /dev/md0 UUID=3b6899af:93e8cd0b:18b63448:ef8ab23c
#ARRAY /dev/md0 metadata=1.2 name=virtual:1
UUID=3b6899af:93e8cd0b:18b63448:ef8ab23c



# blkid
/dev/sda1: UUID="119f7409-bb57-4c82-a526-286a639779d6" TYPE="ext4"
PARTUUID="3b337c94-01"
/dev/sda5: UUID="f2217412-d4d6-4a6e-b699-7f2978db9423" TYPE="swap"
PARTUUID="3b337c94-05"
/dev/sdb1: UUID="f155bd77-8924-400c-bab4-04b641338282" TYPE="ext4"
PARTUUID="40303808-01"
/dev/sdc1: UUID="964fd303-748b-8f0e-58f3-84ce3fde97cc"
UUID_SUB="77580f12-adf8-8476-d9c1-448bb041443f" LABEL="virtual:0"
TYPE="linux_raid_member" PARTLABEL="Linux RAID"
PARTUUID="6744510f-2263-4956-8d1f-539edcf598fb"
/dev/sdd1: UUID="964fd303-748b-8f0e-58f3-84ce3fde97cc"
UUID_SUB="1cb11ccc-2aeb-095d-6be1-838b7d7b33b7" LABEL="virtual:0"
TYPE="linux_raid_member" PARTLABEL="Linux RAID"
PARTUUID="6744510f-2263-4956-8d1f-539edcf598fb"
/dev/sde1: UUID="964fd303-748b-8f0e-58f3-84ce3fde97cc"
UUID_SUB="c322a4fb-ec99-f835-0ce9-cb45ce6b376d" LABEL="virtual:0"
TYPE="linux_raid_member" PARTLABEL="Linux RAID"
PARTUUID="6744510f-2263-4956-8d1f-539edcf598fb"

As a further troubleshooting step I recently updated to mdadm 4.0

Sincerely,

David Mitchell


On Sat, Sep 2, 2017 at 12:49 PM, Andreas Klauer
[off-list ref] wrote:

On Sat, Sep 02, 2017 at 11:43:49AM -0400, David Mitchell wrote:

quoted

Sorry I'm still confused.
While the array seems alive, the filesystem seems to be missing?

Hello David,

I'm the one who is confused here.

You posted to the raid mailing list about a superblock, and you even
posted the mdadm command and error message to go along with it.
How did it turn into a filesystem problem now? In your last mail,
I did not see that at all.

This was what you posted and I thought you were referring to:

quoted

# mdadm --examine /dev/md0
mdadm: No md superblock detected on /dev/md0.

And thus that was the part I referred to in my answer.

----

quoted

# mdadm --detail /dev/md*
/dev/md0:
        Version : 1.2
  Creation Time : Sat Aug 26 16:21:20 2017
     Raid Level : raid5

So according to this, your RAID was created about a week ago.

You did write that you just migrated to 4TB drives. There are various ways
to go about that, making a new RAID and copying stuff over from the
old one is not unusual. So this creation time might be perfectly normal.

Of course if this migration happened more than one week ago then I can
only assume you used mdadm --create to fix some RAID issue or other?

mdadm --create is like mkfs: you don't expect data to exist afterwards.
For mdadm --create to retain data you have to get all settings perfectly
right and those settings change depending on when you originally created
it and what you did with it since (e.g. mdadm --grow might change offsets).

I'm not sure how to help you because nothing in your mails explains
how this issue came about.

There are previous discussions on this list about botched mdadm --create,
wrong data offsets, and searching for correct filesystem offsets.

Whatever you do, don't write on these drives while looking for lost data.

Regards
Andreas Klauer

On 04/09/17 04:35, David Mitchell wrote:

mdadm --stop /dev/md0
mdadm --assemble --force /dev/md0 /dev/sdc1 /dev/sdd1 /dev/sde1

The array instantly loaded and went into a sync.  I let the sync
complete overnight.    The file system on the raid array seems to be
gone.

I'm looking for any hints on how I might get it back.

The array loads fine, but when I attempted to mount the array, it
complains about not finding the file system

OoWwww!

I think you've just trashed one of the drives :-( The sync will have
overwritten one of the drives with parity calculated from the other two.

You've got two drives of valid data from a 3-drive raid-5 set, so the
data could be recovered, but it's going to be a long job.

Somebody else is going to have to chime in and tell you what to do, but
now is the time you really want your backup :-(

Cheers,
Wol

On Sun, Sep 03, 2017 at 11:35:25PM -0400, David Mitchell wrote:

I really do NOT remember running the  --create command.

There is no other explanation for it. It has happened somehow.

The pictures over 512k don't display correctly.

So not only/necessarily a wrong data offset, but also wrong drive order.

At this point I'm hoping for help on next steps in recovery/troubleshooting.

1. Use overlays.

https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file

   That way you can safely experiment and create RAID with 
   different settings. (Use --assume-clean and/or missing)

2. Check first 128M of each drive (your current data offset).
   See if you can find a valid filesystem header anywhere.
   That way you could determine the correct data offset.

3. Find a JPEG header (any known file type, like a 2-3M file)
   and look at the other drives for the same offset. You should be 
   able to deduce RAID layout, chunksize, drive order from that.

Instead of 3) you can also simply trial and error with overlays 
until you find a setting that allows photorec to find larger files intact.

The resync might not necessarily have damaged your data. If the offsets 
were the same and the RAID level was the same, and the drives were in 
sync, a resync even with wrong settings would still produce the same data. 
For XOR, a ^ b = c and b ^ a = c so switching the drives does no damage 
provided you don't write anything else...

Regards
Andreas Klauer

@Andreas Klauer a huge THANK YOU for taking the time to review my
issue.   I really appreciate your help.

Sincerely,

David Mitchell


On Mon, Sep 4, 2017 at 11:49 AM, Andreas Klauer
[off-list ref] wrote:

On Sun, Sep 03, 2017 at 11:35:25PM -0400, David Mitchell wrote:

quoted

I really do NOT remember running the  --create command.

There is no other explanation for it. It has happened somehow.

quoted

The pictures over 512k don't display correctly.

So not only/necessarily a wrong data offset, but also wrong drive order.

quoted

At this point I'm hoping for help on next steps in recovery/troubleshooting.

1. Use overlays.

https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file

   That way you can safely experiment and create RAID with
   different settings. (Use --assume-clean and/or missing)

2. Check first 128M of each drive (your current data offset).
   See if you can find a valid filesystem header anywhere.
   That way you could determine the correct data offset.

3. Find a JPEG header (any known file type, like a 2-3M file)
   and look at the other drives for the same offset. You should be
   able to deduce RAID layout, chunksize, drive order from that.

Instead of 3) you can also simply trial and error with overlays
until you find a setting that allows photorec to find larger files intact.

The resync might not necessarily have damaged your data. If the offsets
were the same and the RAID level was the same, and the drives were in
sync, a resync even with wrong settings would still produce the same data.
For XOR, a ^ b = c and b ^ a = c so switching the drives does no damage
provided you don't write anything else...

Regards
Andreas Klauer

SUCCESSFUL RECOVERY!

@Andreas Klauer you are the man!    A huge THANK YOU for taking the
time to review my
issue.   I really appreciate your help.  While I do back up there were
some recent pictures of my children that had not been backed up yet.
Thank you!

Recreating the array with a different drive order fixed my problem,
100% recovered!

After setting up overlays per your instructions:

# mdadm --stop /dev/md1
mdadm: stopped /dev/md1

# mdadm --assemble --force /dev/md1 /dev/mapper/sde1 /dev/mapper/sdc1
/dev/mapper/sdd1
mdadm: /dev/md1 has been started with 3 drives.

# mount /dev/md1 /mnt/raid/
mount: wrong fs type, bad option, bad superblock on /dev/md1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so.

# mdadm --stop /dev/md1
mdadm: stopped /dev/md1

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
unused devices: <none>

# mdadm --create --verbose --assume-clean  /dev/md1 --level=5
--raid-devices=3 /dev/mapper/sde1 /dev/mapper/sdc1 /dev/mapper/sdd1

/ NOTE the order changed from C,D,E (not working) to E, C, D (working)

mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: /dev/mapper/sde1 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Sat Aug 26 16:21:20 2017
mdadm: partition table exists on /dev/mapper/sde1 but will be lost or
       meaningless after creating array
mdadm: /dev/mapper/sdc1 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Sat Aug 26 16:21:20 2017
mdadm: partition table exists on /dev/mapper/sdc1 but will be lost or
       meaningless after creating array
mdadm: /dev/mapper/sdd1 appears to be part of a raid array:
       level=raid5 devices=3 ctime=Sat Aug 26 16:21:20 2017
mdadm: size set to 3900571648K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? Y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
# mount /dev/md1 /mnt/raid/
# ls /mnt/raid/
kvm  lost+found  media  my-docs  test  tmp

# umount /mnt/raid
# fsck.ext4 -f -n /dev/md1
e2fsck 1.42.13 (17-May-2015)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/md1: 224614/487571456 files (0.9% non-contiguous),
690410561/1950285824 blocks
virtual mnt # cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0]
[raid1] [raid10]
md1 : active raid5 dm-1[2] dm-0[1] dm-2[0]
      7801143296 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>

Sincerely,

David Mitchell


On Mon, Sep 4, 2017 at 1:56 PM, David Mitchell
[off-list ref] wrote:

@Andreas Klauer a huge THANK YOU for taking the time to review my
issue.   I really appreciate your help.

Sincerely,

David Mitchell


On Mon, Sep 4, 2017 at 11:49 AM, Andreas Klauer
[off-list ref] wrote:

quoted

On Sun, Sep 03, 2017 at 11:35:25PM -0400, David Mitchell wrote:

quoted

I really do NOT remember running the  --create command.

There is no other explanation for it. It has happened somehow.

quoted

The pictures over 512k don't display correctly.

So not only/necessarily a wrong data offset, but also wrong drive order.

quoted

At this point I'm hoping for help on next steps in recovery/troubleshooting.

1. Use overlays.

https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file

   That way you can safely experiment and create RAID with
   different settings. (Use --assume-clean and/or missing)

2. Check first 128M of each drive (your current data offset).
   See if you can find a valid filesystem header anywhere.
   That way you could determine the correct data offset.

3. Find a JPEG header (any known file type, like a 2-3M file)
   and look at the other drives for the same offset. You should be
   able to deduce RAID layout, chunksize, drive order from that.

Instead of 3) you can also simply trial and error with overlays
until you find a setting that allows photorec to find larger files intact.

The resync might not necessarily have damaged your data. If the offsets
were the same and the RAID level was the same, and the drives were in
sync, a resync even with wrong settings would still produce the same data.
For XOR, a ^ b = c and b ^ a = c so switching the drives does no damage
provided you don't write anything else...

Regards
Andreas Klauer