Re: MD Array Unexpected Kernel Hang
From: Alan Braithwaite <hidden>
Date: 2022-06-09 00:26:19
Appreciated. Only reason I didn't include it initially is because it's a giant wall of text (which the other debugging info was anyway, so I should have just been proactive).
Anyway, it can now be found below, annotated with the failing drives first.
Thanks,
- Alan
First 2 (sdg, sdh) are the "bad" drives which cause the md hangs when added to the array. sdh looks particularly healthy to my reading of the output, but was still causing the issues.
[root@ceres tmp]# sudo smartctl -x /dev/sdg
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.9-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS724020ALS640
Revision: A1C4
Compliance: SPC-4
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
LU is resource provisioned, LBPRZ=0
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca06d287590
Serial number: P5GR7TBV
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Wed Jun 8 17:22:20 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
Read Cache is: Enabled
Writeback Cache is: Disabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 42 C
Drive Trip Temperature: 85 C
Manufactured in week 21 of year 2015
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 32
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 1530
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 12276642395193344
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 110826 19 0 110845 1515087 460659.950 0
write: 0 0 0 0 972273 44976.018 0
verify: 0 0 0 0 58029 0.000 0
Non-medium error count: 0
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 39609 - [- - -]
Long (extended) Self-test duration: 22236 seconds [6.2 hours]
Background scan results log
Status: waiting until BMS interval timer expires
Accumulated power on time, hours:minutes 39704:24 [2382264 minutes]
Number of background scans performed: 62, scan progress: 0.00%
Number of background medium scans performed: 62
Protocol Specific port log page for SAS SSP
relative target port id = 1
generation code = 1
number of phys = 1
phy identifier = 0
attached device type: no device attached
attached reason: unknown
reason: power on
negotiated logical link rate: phy enabled; unknown
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=0
SAS address = 0x5000cca06d287591
attached SAS address = 0x0
attached phy identifier = 0
Invalid DWORD count = 0
Running disparity error count = 0
Loss of DWORD synchronization count = 0
Phy reset problem count = 0
relative target port id = 2
generation code = 1
number of phys = 1
phy identifier = 1
attached device type: expander device
attached reason: SMP phy control function
reason: unknown
negotiated logical link rate: phy enabled; 6 Gbps
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=1
SAS address = 0x5000cca06d287592
attached SAS address = 0x50030480003954bf
attached phy identifier = 14
Invalid DWORD count = 10
Running disparity error count = 10
Loss of DWORD synchronization count = 2
Phy reset problem count = 0
[root@ceres tmp]# sudo smartctl -x /dev/sdh
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.9-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS724020ALS640
Revision: A2C0
Compliance: SPC-4
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca06d3da1bc
Serial number: P5H2WSDV
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Wed Jun 8 17:22:25 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
Read Cache is: Enabled
Writeback Cache is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 41 C
Drive Trip Temperature: 85 C
Manufactured in week 28 of year 2015
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 23
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 1512
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 386113231388672
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 525 2 0 527 222 4203.269 0
write: 0 0 0 0 4103 13423.670 0
verify: 0 0 0 0 11426 0.000 0
Non-medium error count: 1
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 39221 - [- - -]
# 2 Background short Completed - 34658 - [- - -]
# 3 Background short Completed - 32405 - [- - -]
# 4 Background short Completed - 32380 - [- - -]
# 5 Background short Completed - 32357 - [- - -]
# 6 Background short Completed - 32332 - [- - -]
# 7 Background short Completed - 32308 - [- - -]
# 8 Background short Completed - 32285 - [- - -]
# 9 Background short Completed - 32260 - [- - -]
#10 Background short Completed - 32237 - [- - -]
#11 Background short Completed - 32213 - [- - -]
#12 Background short Completed - 32189 - [- - -]
#13 Background short Completed - 32165 - [- - -]
#14 Background short Completed - 32141 - [- - -]
#15 Background short Completed - 32117 - [- - -]
#16 Background short Completed - 32093 - [- - -]
#17 Background short Completed - 32068 - [- - -]
#18 Background short Completed - 32045 - [- - -]
#19 Background short Completed - 32020 - [- - -]
#20 Background short Completed - 31997 - [- - -]
Long (extended) Self-test duration: 22650 seconds [6.3 hours]
Background scan results log
Status: waiting until BMS interval timer expires
Accumulated power on time, hours:minutes 39315:59 [2358959 minutes]
Number of background scans performed: 243, scan progress: 0.00%
Number of background medium scans performed: 243
Protocol Specific port log page for SAS SSP
relative target port id = 1
generation code = 1
number of phys = 1
phy identifier = 0
attached device type: no device attached
attached reason: unknown
reason: power on
negotiated logical link rate: phy enabled; unknown
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=0
SAS address = 0x5000cca06d3da1bd
attached SAS address = 0x0
attached phy identifier = 0
Invalid DWORD count = 0
Running disparity error count = 0
Loss of DWORD synchronization count = 0
Phy reset problem count = 0
relative target port id = 2
generation code = 1
number of phys = 1
phy identifier = 1
attached device type: expander device
attached reason: SMP phy control function
reason: unknown
negotiated logical link rate: phy enabled; 6 Gbps
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=1
SAS address = 0x5000cca06d3da1be
attached SAS address = 0x50030480003954bf
attached phy identifier = 15
Invalid DWORD count = 18
Running disparity error count = 16
Loss of DWORD synchronization count = 3
Phy reset problem count = 0
# --- Good Drives Start Here ---
[root@ceres tmp]# sudo smartctl -x /dev/sdc
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.9-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS724020ALS640
Revision: A1C4
Compliance: SPC-4
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
LU is resource provisioned, LBPRZ=0
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca06d2927fc
Serial number: P5GRMNRV
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Wed Jun 8 17:22:31 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
Read Cache is: Enabled
Writeback Cache is: Disabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 44 C
Drive Trip Temperature: 85 C
Manufactured in week 21 of year 2015
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 33
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 2019
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 12460064392609792
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 125617 4 0 125621 1511670 470203.046 0
write: 0 0 0 0 311583 46916.011 0
verify: 0 0 0 0 1226 0.000 0
Non-medium error count: 0
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 39776 - [- - -]
# 2 Background short Completed - 39480 - [- - -]
Long (extended) Self-test duration: 22236 seconds [6.2 hours]
Background scan results log
Status: waiting until BMS interval timer expires
Accumulated power on time, hours:minutes 39871:02 [2392262 minutes]
Number of background scans performed: 65, scan progress: 0.00%
Number of background medium scans performed: 65
Protocol Specific port log page for SAS SSP
relative target port id = 1
generation code = 1
number of phys = 1
phy identifier = 0
attached device type: no device attached
attached reason: unknown
reason: power on
negotiated logical link rate: phy enabled; unknown
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=0
SAS address = 0x5000cca06d2927fd
attached SAS address = 0x0
attached phy identifier = 0
Invalid DWORD count = 0
Running disparity error count = 0
Loss of DWORD synchronization count = 0
Phy reset problem count = 0
relative target port id = 2
generation code = 1
number of phys = 1
phy identifier = 1
attached device type: expander device
attached reason: SMP phy control function
reason: unknown
negotiated logical link rate: phy enabled; 6 Gbps
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=1
SAS address = 0x5000cca06d2927fe
attached SAS address = 0x50030480003954bf
attached phy identifier = 9
Invalid DWORD count = 11
Running disparity error count = 10
Loss of DWORD synchronization count = 2
Phy reset problem count = 0
[root@ceres tmp]# sudo smartctl -x /dev/sdd
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.9-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS724020ALS640
Revision: A1C4
Compliance: SPC-4
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
LU is resource provisioned, LBPRZ=0
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca06d2924f0
Serial number: P5GRMGEV
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Wed Jun 8 17:22:32 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
Read Cache is: Enabled
Writeback Cache is: Disabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 44 C
Drive Trip Temperature: 85 C
Manufactured in week 21 of year 2015
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 33
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 2031
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 12182041999179776
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 284059 0 0 284059 1507556 472213.429 0
write: 0 0 0 0 378639 45073.915 0
verify: 0 0 0 0 319 0.000 0
Non-medium error count: 0
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 39775 - [- - -]
# 2 Background short Completed - 39479 - [- - -]
Long (extended) Self-test duration: 22236 seconds [6.2 hours]
Background scan results log
Status: waiting until BMS interval timer expires
Accumulated power on time, hours:minutes 39870:03 [2392203 minutes]
Number of background scans performed: 63, scan progress: 0.00%
Number of background medium scans performed: 63
Protocol Specific port log page for SAS SSP
relative target port id = 1
generation code = 1
number of phys = 1
phy identifier = 0
attached device type: no device attached
attached reason: unknown
reason: power on
negotiated logical link rate: phy enabled; unknown
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=0
SAS address = 0x5000cca06d2924f1
attached SAS address = 0x0
attached phy identifier = 0
Invalid DWORD count = 0
Running disparity error count = 0
Loss of DWORD synchronization count = 0
Phy reset problem count = 0
relative target port id = 2
generation code = 1
number of phys = 1
phy identifier = 1
attached device type: expander device
attached reason: SMP phy control function
reason: unknown
negotiated logical link rate: phy enabled; 6 Gbps
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=1
SAS address = 0x5000cca06d2924f2
attached SAS address = 0x50030480003954bf
attached phy identifier = 10
Invalid DWORD count = 10
Running disparity error count = 10
Loss of DWORD synchronization count = 2
Phy reset problem count = 0
[root@ceres tmp]# sudo smartctl -x /dev/sde
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.9-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS724020ALS640
Revision: A1C4
Compliance: SPC-4
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
LU is resource provisioned, LBPRZ=0
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca06d283b08
Serial number: P5GR3WJV
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Wed Jun 8 17:22:34 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
Read Cache is: Enabled
Writeback Cache is: Disabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 41 C
Drive Trip Temperature: 85 C
Manufactured in week 21 of year 2015
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 33
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 2024
Elements in grown defect list: 0
Vendor (Seagate Cache) information
Blocks sent to initiator = 12291100328853504
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 72338 1 0 72339 1454480 468885.996 0
write: 0 0 0 0 283314 44839.275 0
verify: 0 0 0 0 754 0.000 0
Non-medium error count: 0
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 39775 - [- - -]
# 2 Background short Completed - 39479 - [- - -]
Long (extended) Self-test duration: 22236 seconds [6.2 hours]
Background scan results log
Status: waiting until BMS interval timer expires
Accumulated power on time, hours:minutes 39870:31 [2392231 minutes]
Number of background scans performed: 62, scan progress: 0.00%
Number of background medium scans performed: 62
Protocol Specific port log page for SAS SSP
relative target port id = 1
generation code = 1
number of phys = 1
phy identifier = 0
attached device type: no device attached
attached reason: unknown
reason: power on
negotiated logical link rate: phy enabled; unknown
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=0
SAS address = 0x5000cca06d283b09
attached SAS address = 0x0
attached phy identifier = 0
Invalid DWORD count = 0
Running disparity error count = 0
Loss of DWORD synchronization count = 0
Phy reset problem count = 0
relative target port id = 2
generation code = 1
number of phys = 1
phy identifier = 1
attached device type: expander device
attached reason: SMP phy control function
reason: unknown
negotiated logical link rate: phy enabled; 6 Gbps
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=1
SAS address = 0x5000cca06d283b0a
attached SAS address = 0x50030480003954bf
attached phy identifier = 11
Invalid DWORD count = 9
Running disparity error count = 9
Loss of DWORD synchronization count = 2
Phy reset problem count = 0
[root@ceres tmp]# sudo smartctl -x /dev/sdf
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.9-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS724020ALS640
Revision: A2C0
Compliance: SPC-4
User Capacity: 2,000,398,934,016 bytes [2.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x5000cca0284dc068
Serial number: P6HBSKZV
Device type: disk
Transport protocol: SAS (SPL-4)
Local Time is: Wed Jun 8 17:22:36 2022 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
Read Cache is: Enabled
Writeback Cache is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 37 C
Drive Trip Temperature: 85 C
Manufactured in week 06 of year 2014
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 47
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 1871
Elements in grown defect list: 5
Vendor (Seagate Cache) information
Blocks sent to initiator = 498103379558400
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 44471 1 0 44472 36583 12783.706 0
write: 0 0 0 0 11015 19676.167 0
verify: 0 0 0 0 6221 0.000 0
Non-medium error count: 1
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed - 46820 - [- - -]
# 2 Background short Completed - 42138 - [- - -]
# 3 Background short Completed - 42111 - [- - -]
# 4 Background short Completed - 42103 - [- - -]
# 5 Background short Completed - 42099 - [- - -]
# 6 Background short Completed - 41365 - [- - -]
# 7 Background short Completed - 41341 - [- - -]
# 8 Background short Completed - 41317 - [- - -]
# 9 Background short Completed - 41293 - [- - -]
#10 Background short Completed - 41269 - [- - -]
#11 Background short Completed - 41245 - [- - -]
#12 Background short Completed - 41221 - [- - -]
#13 Background short Completed - 41197 - [- - -]
#14 Background short Completed - 41172 - [- - -]
#15 Background short Completed - 41149 - [- - -]
#16 Background short Completed - 41125 - [- - -]
#17 Background short Completed - 41101 - [- - -]
#18 Background short Completed - 41077 - [- - -]
#19 Background short Completed - 41053 - [- - -]
#20 Background short Completed - 41029 - [- - -]
Long (extended) Self-test duration: 22650 seconds [6.3 hours]
Background scan results log
Status: waiting until BMS interval timer expires
Accumulated power on time, hours:minutes 46915:04 [2814904 minutes]
Number of background scans performed: 294, scan progress: 0.00%
Number of background medium scans performed: 294
# when lba(hex) [sk,asc,ascq] reassign_status
1 42149:02 00000000becb3840 [3,11,0] Recovered via rewrite in-place
Protocol Specific port log page for SAS SSP
relative target port id = 1
generation code = 1
number of phys = 1
phy identifier = 0
attached device type: no device attached
attached reason: unknown
reason: power on
negotiated logical link rate: phy enabled; unknown
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=0
SAS address = 0x5000cca0284dc069
attached SAS address = 0x0
attached phy identifier = 0
Invalid DWORD count = 0
Running disparity error count = 0
Loss of DWORD synchronization count = 0
Phy reset problem count = 0
relative target port id = 2
generation code = 1
number of phys = 1
phy identifier = 1
attached device type: expander device
attached reason: SMP phy control function
reason: unknown
negotiated logical link rate: phy enabled; 6 Gbps
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=1
SAS address = 0x5000cca0284dc06a
attached SAS address = 0x50030480003954bf
attached phy identifier = 12
Invalid DWORD count = 10
Running disparity error count = 9
Loss of DWORD synchronization count = 2
Phy reset problem count = 0
On Wed, Jun 8, 2022, at 15:56, o1bigtenor wrote:On Wed, Jun 8, 2022 at 5:22 PM Alan Braithwaite [off-list ref] wrote:quoted
Just someone learning who asked questions in the past herequoted
Please let me know if there's any more information I can provide.smartmontools is very likely something that will give some more information that (likely) will be asked for. If not previous installed you might want to get and then look at what it says about each drive. Often said information is asked for when the low down and dirty work gets a happening. HTH