Hi Matt,
I didn't see this make it to linux-raid, so I'll quote more than normal.
Oh, and convention on kernel.org is to reply-to-all, trim unnecessary
quotes, and avoid top-posting. Please.
On 03/27/2015 11:10 PM, Matt Callaghan wrote:
Just noticed the lsdrv [1] link to git; got it, here's the output
{{{
fermulator@fermmy-mdadm:~/downloads/lsdrv/lsdrv$ ./lsdrv
PCI [ahci] 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 40)
├scsi 0:x:x:x [Empty]
└scsi 1:0:0:0 ATA Maxtor 6Y160M0
└sda 152.67g [8:0] Empty/Unknown
├sda1 512.00m [8:1] Empty/Unknown
│└Mounted as /dev/sda1 @ /boot/efi
├sda2 148.71g [8:2] Empty/Unknown
│└Mounted as /dev/disk/by-uuid/5549ca2f-758a-4e04-8e36-cf4544bef4fb @ /
└sda3 3.46g [8:3] Empty/Unknown
PCI [mptsas] 05:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
├scsi 2:0:0:0 ATA ST2000DL003-9VT1
│└sdb 1.82t [8:16] Empty/Unknown
│ └sdb1 1.82t [8:17] Empty/Unknown
├scsi 2:0:1:0 ATA ST2000DL003-9VT1
│└sdc 1.82t [8:32] Empty/Unknown
│ └sdc1 1.82t [8:33] Empty/Unknown
├scsi 2:0:2:0 ATA ST2000DL003-9VT1
│└sdd 1.82t [8:48] Empty/Unknown
│ └sdd1 1.82t [8:49] Empty/Unknown
├scsi 2:0:3:0 ATA ST2000VN000-1H31
│└sde 1.82t [8:64] Empty/Unknown
│ └sde1 1.82t [8:65] Empty/Unknown
├scsi 2:0:4:0 ATA ST2000DL003-9VT1
│└sdf 1.82t [8:80] Empty/Unknown
│ └sdf1 1.82t [8:81] Empty/Unknown
├scsi 2:0:5:0 ATA ST2000DL003-9VT1
│└sdg 1.82t [8:96] Empty/Unknown
│ └sdg1 1.82t [8:97] Empty/Unknown
├scsi 2:0:6:0 ATA ST2000DL003-9VT1
│└sdh 1.82t [8:112] Empty/Unknown
│ └sdh1 1.82t [8:113] Empty/Unknown
├scsi 2:0:7:0 ATA ST2000VN000-1H31
│└sdi 1.82t [8:128] Empty/Unknown
│ └sdi1 1.82t [8:129] Empty/Unknown
└scsi 2:x:x:x [Empty]
Other Block Devices
├loop0 0.00k [7:0] Empty/Unknown
├loop1 0.00k [7:1] Empty/Unknown
├loop2 0.00k [7:2] Empty/Unknown
├loop3 0.00k [7:3] Empty/Unknown
├loop4 0.00k [7:4] Empty/Unknown
├loop5 0.00k [7:5] Empty/Unknown
├loop6 0.00k [7:6] Empty/Unknown
├loop7 0.00k [7:7] Empty/Unknown
├ram0 64.00m [1:0] Empty/Unknown
├ram1 64.00m [1:1] Empty/Unknown
├ram2 64.00m [1:2] Empty/Unknown
├ram3 64.00m [1:3] Empty/Unknown
├ram4 64.00m [1:4] Empty/Unknown
├ram5 64.00m [1:5] Empty/Unknown
├ram6 64.00m [1:6] Empty/Unknown
├ram7 64.00m [1:7] Empty/Unknown
├ram8 64.00m [1:8] Empty/Unknown
├ram9 64.00m [1:9] Empty/Unknown
├ram10 64.00m [1:10] Empty/Unknown
├ram11 64.00m [1:11] Empty/Unknown
├ram12 64.00m [1:12] Empty/Unknown
├ram13 64.00m [1:13] Empty/Unknown
├ram14 64.00m [1:14] Empty/Unknown
└ram15 64.00m [1:15] Empty/Unknown
}}}
Ok. Not that helpful. I suspect you had error messages about missing
utilities. No serial numbers.
[trim /]
mdadm output as of NOW. But note that the output here is likely useless
since the last thing I was trying to was getting the array back together
as per the forum posting... (it's definitely not in the original state
anymore...)
Yep, useless.
[trim /]
smartctl outputs are:
/dev/sdb:
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda Green (AF)
Device Model: ST2000DL003-9VT166
Serial Number: 5YD0XWHR
LU WWN Device Id: 5 000c50 02f4197f5
Firmware Version: CC32
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Fri Mar 27 22:57:05 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM level is: 0 (vendor specific), recommended: 254
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 113 099 006 - 51859880
3 Spin_Up_Time PO---- 093 092 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 422
5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
7 Seek_Error_Rate POSR-- 072 060 030 - 17185766
9 Power_On_Hours -O--CK 061 061 000 - 34871
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 71
183 Runtime_Bad_Block -O--CK 100 100 000 - 0
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 099 099 000 - 1
188 Command_Timeout -O--CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 094 094 000 - 6
190 Airflow_Temperature_Cel -O---K 059 043 045 Past 41 (5 77 42 35 0)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 420
193 Load_Cycle_Count -O--CK 100 100 000 - 422
194 Temperature_Celsius -O---K 041 057 000 - 41 (0 13 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 017 003 000 - 51859880
197 Current_Pending_Sector -O--C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 16990890657845
241 Total_LBAs_Written ------ 100 253 000 - 731266756
242 Total_LBAs_Read ------ 100 253 000 - 1129016466
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
SCT Error Recovery Control command not supported
Now we know why your array fell apart. Using green and/or desktop
drives without mitigating the timeout mismatch problem.
/dev/sdc:
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda Green (AF)
Device Model: ST2000DL003-9VT166
Serial Number: 5YD1B1ZJ
LU WWN Device Id: 5 000c50 02f361865
Firmware Version: CC32
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Fri Mar 27 22:57:06 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM level is: 0 (vendor specific), recommended: 254
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 112 090 006 - 44947192
3 Spin_Up_Time PO---- 093 092 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 68
5 Reallocated_Sector_Ct PO--CK 078 078 036 - 14728
7 Seek_Error_Rate POSR-- 072 066 030 - 15873942
9 Power_On_Hours -O--CK 061 061 000 - 34875
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 74
183 Runtime_Bad_Block -O--CK 100 100 000 - 0
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 001 001 000 - 823
188 Command_Timeout -O--CK 100 099 000 - 65539
189 High_Fly_Writes -O-RCK 093 093 000 - 7
190 Airflow_Temperature_Cel -O---K 058 044 045 Past 42 (2 158 44 36 0)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 65
193 Load_Cycle_Count -O--CK 100 100 000 - 68
194 Temperature_Celsius -O---K 042 056 000 - 42 (0 13 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 016 003 000 - 44947192
197 Current_Pending_Sector -O--C- 089 089 000 - 952
^^^^^
Wow!
198 Offline_Uncorrectable ----C- 089 089 000 - 952
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 141149805250605
241 Total_LBAs_Written ------ 100 253 000 - 3292940140
242 Total_LBAs_Read ------ 100 253 000 - 496297916
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
SCT Error Recovery Control command not supported
And again.
/dev/sdd:
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda Green (AF)
Device Model: ST2000DL003-9VT166
Serial Number: 5YD15M4K
LU WWN Device Id: 5 000c50 02f386588
Firmware Version: CC32
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Fri Mar 27 22:57:07 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM level is: 0 (vendor specific), recommended: 254
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 117 099 006 - 153485440
3 Spin_Up_Time PO---- 093 092 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 352
5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
7 Seek_Error_Rate POSR-- 076 060 030 - 43819206
9 Power_On_Hours -O--CK 061 061 000 - 35013
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 74
183 Runtime_Bad_Block -O--CK 100 100 000 - 0
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 097 097 000 - 3
188 Command_Timeout -O--CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 099 099 000 - 1
190 Airflow_Temperature_Cel -O---K 057 046 045 - 43 (Min/Max 36/43)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 351
193 Load_Cycle_Count -O--CK 100 100 000 - 353
194 Temperature_Celsius -O---K 043 054 000 - 43 (0 11 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 021 003 000 - 153485440
197 Current_Pending_Sector -O--C- 100 100 000 - 8
More Pending sectors. These are locations where unrecoverable read
errors occurred that the firmware is waiting for an overwrite to decide
if they are fixable.
198 Offline_Uncorrectable ----C- 100 100 000 - 8
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 134501195876534
241 Total_LBAs_Written ------ 100 253 000 - 879538094
242 Total_LBAs_Read ------ 100 253 000 - 1783662156
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
SCT Error Recovery Control command not supported
Sigh.
/dev/sde:
=== START OF INFORMATION SECTION ===
Device Model: ST2000VN000-1H3164
Serial Number: W1H25K77
LU WWN Device Id: 5 000c50 06a40c121
Firmware Version: SC42
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5900 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Mar 27 22:57:07 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM level is: 254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 116 099 006 - 117001736
3 Spin_Up_Time PO---- 096 095 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 21
5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 064 060 030 - 3017660
9 Power_On_Hours -O--CK 085 085 000 - 13146
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 21
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 100 100 000 - 0
188 Command_Timeout -O--CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 058 058 000 - 42
190 Airflow_Temperature_Cel -O---K 065 056 045 - 35 (Min/Max 35/37)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 21
193 Load_Cycle_Count -O--CK 100 100 000 - 21
194 Temperature_Celsius -O---K 035 044 000 - 35 (0 16 0 0 0)
197 Current_Pending_Sector -O--C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
SCT Error Recovery Control:
Read: 1 (0.1 seconds)
Write: 1 (0.1 seconds)
Interesting. Is this the device default? The drives I've seen that
have a default have either 4.0s or 7.0s.
/dev/sdf:
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda Green (AF)
Device Model: ST2000DL003-9VT166
Serial Number: 5YD18S73
LU WWN Device Id: 5 000c50 02f3fab7d
Firmware Version: CC32
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Fri Mar 27 22:57:07 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM level is: 0 (vendor specific), recommended: 254
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 109 099 006 - 23951160
3 Spin_Up_Time PO---- 093 092 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 70
5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
7 Seek_Error_Rate POSR-- 075 060 030 - 39605538
9 Power_On_Hours -O--CK 061 061 000 - 34955
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 75
183 Runtime_Bad_Block -O--CK 100 100 000 - 0
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 100 100 000 - 0
188 Command_Timeout -O--CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 089 089 000 - 11
190 Airflow_Temperature_Cel -O---K 058 048 045 - 42 (Min/Max 34/42)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
193 Load_Cycle_Count -O--CK 100 100 000 - 70
194 Temperature_Celsius -O---K 042 052 000 - 42 (0 10 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 013 003 000 - 23951160
197 Current_Pending_Sector -O--C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 194931385731211
241 Total_LBAs_Written ------ 100 253 000 - 4208935845
242 Total_LBAs_Read ------ 100 253 000 - 3841138908
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
SCT Error Recovery Control command not supported
And again.
/dev/sdg:
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda Green (AF)
Device Model: ST2000DL003-9VT166
Serial Number: 5YD1ACSD
LU WWN Device Id: 5 000c50 02f31ac2f
Firmware Version: CC32
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Fri Mar 27 22:57:08 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM level is: 0 (vendor specific), recommended: 254
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 113 099 006 - 50711848
3 Spin_Up_Time PO---- 093 092 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 70
5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
7 Seek_Error_Rate POSR-- 075 060 030 - 41597886
9 Power_On_Hours -O--CK 061 061 000 - 34955
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 74
183 Runtime_Bad_Block -O--CK 100 100 000 - 0
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 100 100 000 - 0
188 Command_Timeout -O--CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 100 100 000 - 0
190 Airflow_Temperature_Cel -O---K 058 048 045 - 42 (Min/Max 36/43)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
193 Load_Cycle_Count -O--CK 100 100 000 - 70
194 Temperature_Celsius -O---K 042 052 000 - 42 (0 10 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 017 003 000 - 50711848
197 Current_Pending_Sector -O--C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 121040768370827
241 Total_LBAs_Written ------ 100 253 000 - 1173584109
242 Total_LBAs_Read ------ 100 253 000 - 1269612579
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
SCT Error Recovery Control command not supported
And sigh again. Broken record, I know. But this is a big deal.
/dev/sdh:
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda Green (AF)
Device Model: ST2000DL003-9VT166
Serial Number: 5YD18S0M
LU WWN Device Id: 5 000c50 02f3f4ec7
Firmware Version: CC32
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
Local Time is: Fri Mar 27 22:57:08 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM level is: 0 (vendor specific), recommended: 254
APM feature is: Unavailable
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 119 099 006 - 229878536
3 Spin_Up_Time PO---- 093 092 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 70
5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
7 Seek_Error_Rate POSR-- 075 060 030 - 38838566
9 Power_On_Hours -O--CK 061 061 000 - 34957
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 76
183 Runtime_Bad_Block -O--CK 100 100 000 - 0
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 100 100 000 - 0
188 Command_Timeout -O--CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 094 094 000 - 6
190 Airflow_Temperature_Cel -O---K 061 051 045 - 39 (Min/Max 29/40)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
193 Load_Cycle_Count -O--CK 100 100 000 - 70
194 Temperature_Celsius -O---K 039 049 000 - 39 (0 11 0 0 0)
195 Hardware_ECC_Recovered -O-RC- 023 003 000 - 229878536
197 Current_Pending_Sector -O--C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
240 Head_Flying_Hours ------ 100 253 000 - 30356828883085
241 Total_LBAs_Written ------ 100 253 000 - 16063676
242 Total_LBAs_Read ------ 100 253 000 - 2558000514
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
SCT Error Recovery Control command not supported
/dev/sdi:
=== START OF INFORMATION SECTION ===
Device Model: ST2000VN000-1H3164
Serial Number: W1H25JXM
LU WWN Device Id: 5 000c50 06a406dab
Firmware Version: SC42
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5900 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Fri Mar 27 22:57:09 2015 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is: Unavailable
APM level is: 254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is: Enabled
ATA Security is: Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
1 Raw_Read_Error_Rate POSR-- 119 099 006 - 218566352
3 Spin_Up_Time PO---- 096 096 000 - 0
4 Start_Stop_Count -O--CK 100 100 020 - 21
5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
7 Seek_Error_Rate POSR-- 064 060 030 - 3082219
9 Power_On_Hours -O--CK 085 085 000 - 13146
10 Spin_Retry_Count PO--C- 100 100 097 - 0
12 Power_Cycle_Count -O--CK 100 100 020 - 21
184 End-to-End_Error -O--CK 100 100 099 - 0
187 Reported_Uncorrect -O--CK 100 100 000 - 0
188 Command_Timeout -O--CK 100 100 000 - 0
189 High_Fly_Writes -O-RCK 050 050 000 - 50
190 Airflow_Temperature_Cel -O---K 064 052 045 - 36 (Min/Max 36/38)
191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
192 Power-Off_Retract_Count -O--CK 100 100 000 - 21
193 Load_Cycle_Count -O--CK 100 100 000 - 21
194 Temperature_Celsius -O---K 036 048 000 - 36 (0 16 0 0 0)
197 Current_Pending_Sector -O--C- 100 100 000 - 0
198 Offline_Uncorrectable ----C- 100 100 000 - 0
199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
||||||_ K auto-keep
|||||__ C event count
||||___ R error rate
|||____ S speed/performance
||_____ O updated online
|______ P prefailure warning
SCT Error Recovery Control:
Read: 1 (0.1 seconds)
Write: 1 (0.1 seconds)
So. You have eight devices that need to make a raid6, and you have no
order information. You have two devices with pending errors that cannot
help us without role #s.
First, you need to deal with the timeout mismatch problem. Only two of
your devices support ERC, so you will need to set long driver timeouts.
Some reading:
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142504030927143&w=2
As for the latter link, I haven't tested that. When I needed such
features myself, I just put the appropriate commands into rc.local.
Since then, I've retired all of my non-raid-rated drives.
Next, you need to run numerous "mdadm --create --assume-clean" attempts
to figure out your device role order. You have 8-factorial permutations
to try (40,320). /dev/sdc and /dev/sdd have pending errors, so leave
them out (use "missing" in their places).
Your only info from the original post that shows all of the necessary
device characteristics is this:
/dev/sdj1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x2
Array UUID : 15d2158f:5cf74d95:fd7f5607:0e447573
Name : fermmy-server:2000 (local to host fermmy-server)
Creation Time : Fri Apr 22 01:12:07 2011
Raid Level : raid6
Raid Devices : 8
Avail Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
Array Size : 11721080448 (11178.09 GiB 12002.39 GB)
Data Offset : 304 sectors
Super Offset : 0 sectors
Recovery Offset : 2441891840 sectors
State : clean
Device UUID : eee3ae0e:f594fdba:58e19113:bc196464
Update Time : Mon Jan 5 00:30:41 2015
Checksum : 7a5a498d - correct
Events : 42912
Layout : left-symmetric
Chunk Size : 64K
Device Role : Active device 4
Array State : A.AAAAAA ('A' == active, '.' == missing)
Note that the data offset is 304. Some of your devices reported a data
offset of 264. None of the reports were from original undisturbed
devices, so we really don't know what offset is correct. "mdadm --add"
will use that mdadm version's offset if it can.
I suggest you try to re-establish the distro you used at the time (April
2011) in a VM and create some test arrays with its version of mdadm to
get the offset to try first.
You then need to create a script that will perform the necessary "mdadm
--create --assume-clean" operations, followed by an "fsck -n" of the
device each time to see how messed up it is. Each attempt into its own
log file, so you can see (by size) which attempts were "cleanest".
Inspect the "best" log files manually to see what was found. With 40k
permutations, you may need to work out some grepping that will help
identify bad from possibly good.
If none of them come up relatively clean, try again with your next best
guess on the offset.
Good luck!
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html