Thanks, info below.
Linux debian 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1
(2019-09-20) x86_64 GNU/Linux
mdadm - v4.1 - 2018-10-01
/dev/sda
=== START OF INFORMATION SECTION ===
Device Model: HITACHI HUS724040ALE640
Serial Number: PBJBNNET
LU WWN Device Id: 5 000cca 23de17ca4
Firmware Version: MJAONS04
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:45:04 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdb
=== START OF INFORMATION SECTION ===
Device Model: MB4000GCWDC
Serial Number: Z1Z8YWB0
LU WWN Device Id: 5 000c50 07ba725b1
Firmware Version: HPGH
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:46:12 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdc
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCHP96XB
LU WWN Device Id: 5 000cca 24cd7c5eb
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:46:39 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdd
=== START OF INFORMATION SECTION ===
Device Model: WL4000GSA6472E
Serial Number: WOL240331724
LU WWN Device Id: 5 0014ee 0588c0835
Firmware Version: A.01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:46:58 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sde
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCGSLX3B
LU WWN Device Id: 5 000cca 24ccabb82
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:47:15 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdf
=== START OF INFORMATION SECTION ===
Device Model: HITACHI HUS724040ALE640
Serial Number: PAGLKHEW
LU WWN Device Id: 5 000cca 22bc8705d
Firmware Version: MJAONS04
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:47:31 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdg
=== START OF INFORMATION SECTION ===
Device Model: MB4000GCWDC
Serial Number: Z1Z8YWJN
LU WWN Device Id: 5 000c50 07ba717cf
Firmware Version: HPGH
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:47:46 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdh
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCGK79AB
LU WWN Device Id: 5 000cca 24cc7d5b7
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:48:01 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdi
=== START OF INFORMATION SECTION ===
Model Family: Toshiba 3.5" MG03ACAxxx(Y) Enterprise HDD
Device Model: TOSHIBA MG03ACA400
Serial Number: 5443K65YF
LU WWN Device Id: 5 000039 58b8009aa
Firmware Version: FL1A
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:48:15 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdj
=== START OF INFORMATION SECTION ===
Device Model: WL4000GSA6472E
Serial Number: WOL240332040
LU WWN Device Id: 5 0014ee 25ddc3495
Firmware Version: 01.01A01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:48:28 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdk
=== START OF INFORMATION SECTION ===
Device Model: MB4000GCWDC
Serial Number: Z1Z8XQ9J
LU WWN Device Id: 5 000c50 07b9764b6
Firmware Version: HPGH
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 6
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:48:41 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdl
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PAJWABKT
LU WWN Device Id: 5 000cca 22be89bf7
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:48:54 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdm
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCG1JAVB
LU WWN Device Id: 5 000cca 24cc0b1bf
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:49:06 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdn
=== START OF INFORMATION SECTION ===
Device Model: HITACHI HUS724040ALE640
Serial Number: PAJK4VMS
LU WWN Device Id: 5 000cca 22be3fb8e
Firmware Version: MJAONS04
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:49:22 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdo
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCG62VNB
LU WWN Device Id: 5 000cca 24cc2c4f6
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:49:34 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdp
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K4000
Device Model: Hitachi HUS724040ALE641
Serial Number: PCGSNTTB
LU WWN Device Id: 5 000cca 24ccac29d
Firmware Version: MJAOA5F0
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Sat Mar 6 17:49:53 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdq
=== START OF INFORMATION SECTION ===
Device Model: HITACHI HUS724040ALE640
Serial Number: PAGTSZES
LU WWN Device Id: 5 000cca 22bcb42ee
Firmware Version: MJAONS04
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:50:05 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sdr
=== START OF INFORMATION SECTION ===
Device Model: WL4000GSA6472E
Serial Number: WOL240331985
LU WWN Device Id: 5 0014ee 003311232
Firmware Version: 01.01A01
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 6 17:50:18 2021 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SCT Error Recovery Control:
Read: 70 (7.0 seconds)
Write: 70 (7.0 seconds)
/dev/sda:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sda1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 220e5a63:3311984e:f28d4b05:593c684e
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : 6c62a047 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 3
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdb:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : c9b754df:84118329:c4e1a36b:bd4c7f83
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 5eb7ae75 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 10
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 0b7f323c:5bcbd1c3:89ef2cd2:fa501410
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 49024193 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 17
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 8c0905ef:8e190062:0a5135ab:5ca3cfa0
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 3c6ce1f - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 1
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sde:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sde1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 896fcc0c:f8b4b6ab:d8056a44:558d0093
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : f6aa6e5a - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 14
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdf:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdf1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : a9bbd48d:24117e4b:f0b0c881:4765bc46
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 89499a5 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdg:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdg1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 01be0b84:bc5ab713:b1352502:de23ddfa
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : fb8228f1 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 9
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdh:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdh1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 32f81397:6f0115a4:c3fa2d02:c078170e
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : b22b23ce - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 16
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdi:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdi1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 0afa52a2:0f7f4f1f:14e924da:d06d9015
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 181486a1 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdj:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdj1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : db1d31c6:cde52a67:9e4c48cc:18bc5dcb
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 2bbec2fc - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 0
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdk:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdk1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 3d9fab1f:172871cc:02215dee:f6c525be
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : ff5c655d - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 11
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdl:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdl1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 114ed6e6:9ab8d767:1fcfeacb:4c408814
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : 95de0ba8 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdm:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdm1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : c8daba50:67137919:9bead1ab:88d3d498
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 159762fe - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 15
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdn:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdn1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : b0893bc1:40c9b3ad:fdf4f722:3acef4c6
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : bf98ccc6 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 2
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdo:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdo1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : c07db065:7316a61b:aac57004:2641fc5f
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 4c8051b0 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 13
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdp:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdp1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x47
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
Recovery Offset : 4561114112 sectors
State : active
Device UUID : 5843bab9:17b0f8ae:f7712bee:41a23f6c
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 39b7d25a - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 12
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdq:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdq1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x45
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : 45b14d19:6d4b9afc:e51c3747:c46230b8
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors
Checksum : 7c0c3300 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 8
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
/dev/sdr:
MBR Magic : aa55
Partition[0] : 4294967295 sectors at 1 (type ee)
/dev/sdr1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x4d
Array UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Name : debian:0 (local to host debian)
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Raid Devices : 18
Avail Dev Size : 7813770895 (3725.90 GiB 4000.65 GB)
Array Size : 62510161920 (59614.34 GiB 64010.41 GB)
Used Dev Size : 7813770240 (3725.90 GiB 4000.65 GB)
Data Offset : 264192 sectors
New Offset : 247808 sectors
Super Offset : 8 sectors
State : active
Device UUID : f27bbac9:00333c9b:c68170ea:a3507ccd
Internal Bitmap : 8 sectors from superblock
Reshape pos'n : 36488912896 (34798.54 GiB 37364.65 GB)
Delta Devices : 5 (13->18)
New Layout : left-symmetric
Update Time : Sat Mar 6 17:30:59 2021
Bad Block Log : 512 entries available at offset 24 sectors - bad
blocks present.
Checksum : 83a03807 - correct
Events : 303248
Layout : left-symmetric-6
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAAAAAAAAAAAA ('A' == active, '.' == missing,
'R' == replacing)
lsdrv
**Warning** The following utility(ies) failed to execute:
sginfo
lsusb
Some information may be missing.
PCI [mpt3sas] 05:00.0 Serial Attached SCSI controller: LSI Logic /
Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
└scsi 0:x:x:x [Empty]
PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10
Family) 4 port SATA IDE Controller #1
└scsi 1:x:x:x [Empty]
PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10
Family) 2 port SATA IDE Controller #2
└scsi 3:x:x:x [Empty]
PCI [pata_marvell] 09:00.0 IDE interface: Marvell Technology Group
Ltd. 88SE6121 SATA II / PATA Controller (rev b2)
└scsi 5:x:x:x [Empty]
USB [usb-storage] {13618102022400B0}
└scsi 8:0:0:0 ADATA USB Flash Drive
└sds 7.24g [65:32] Partitioned (dos)
├sds1 1.24g [65:33] vfat 'D-LIVE_10_2' {3E9E-7588}
│├Mounted as /dev/sds1 @ /run/live/persistence/sds1
│└Mounted as /dev/sds1 @ /usr/lib/live/mount/persistence/sds1
└sds2 6.00g [65:34] ext3 'persistence' {bfde904d-48e4-3741-969a-5ad1f0033243}
├Mounted as /dev/sds2 @ /run/live/persistence/sds2
└Mounted as /dev/sds2 @ /usr/lib/live/mount/persistence/sds2
PCI [mpt3sas] 07:00.0 Serial Attached SCSI controller: LSI Logic /
Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
├phy-7:0:12 scsi 7:0:0:0 ATA HITACHI HUS72404 {PBJBNNET}
│└sda 3.64t [8:0] Partitioned (gpt)
│ └sda1 3.64t [8:1] MD raid6 (3/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ │ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
│ └VG super 40.02t 0 free {EqBsuH-weZl-urAI-aO2f-PiO6-rAO3-J8m1F9}
│ ├dm-1 39.73t [253:1] LV nova ext4 {d1359e47-522e-41f5-bb38-8d8d40826e9e}
│ └dm-0 300.00g [253:0] LV star ext4 {dde10d6e-5e4d-4196-8abe-d36d863c97cd}
├phy-7:0:14 scsi 7:0:1:0 ATA MB4000GCWDC {Z1Z8YWB0}
│└sdb 3.64t [8:16] Partitioned (gpt)
│ └sdb1 3.64t [8:17] MD raid6 (10/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:15 scsi 7:0:2:0 ATA Hitachi HUS72404 {PCHP96XB}
│└sdc 3.64t [8:32] Partitioned (gpt)
│ └sdc1 3.64t [8:33] MD raid6 (17/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:18 scsi 7:0:3:0 ATA WL4000GSA6472E {WOL240331724}
│└sdd 3.64t [8:48] Partitioned (gpt)
│ └sdd1 3.64t [8:49] MD raid6 (1/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:19 scsi 7:0:4:0 ATA HITACHI HUS72404 {PAGLKHEW}
│└sdf 3.64t [8:80] Partitioned (gpt)
│ └sdf1 3.64t [8:81] MD raid6 (4/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:20 scsi 7:0:5:0 ATA MB4000GCWDC {Z1Z8YWJN}
│└sdg 3.64t [8:96] Partitioned (gpt)
│ └sdg1 3.64t [8:97] MD raid6 (9/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:21 scsi 7:0:6:0 ATA Hitachi HUS72404 {PCGK79AB}
│└sdh 3.64t [8:112] Partitioned (gpt)
│ └sdh1 3.64t [8:113] MD raid6 (16/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:23 scsi 7:0:7:0 ATA TOSHIBA MG03ACA4 {5443K65YF}
│└sdi 3.64t [8:128] Partitioned (gpt)
│ └sdi1 3.64t [8:129] MD raid6 (7/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:25 scsi 7:0:8:0 ATA WL4000GSA6472E {WOL240332040}
│└sdj 3.64t [8:144] Partitioned (gpt)
│ └sdj1 3.64t [8:145] MD raid6 (0/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:26 scsi 7:0:9:0 ATA Hitachi HUS72404 {PAJWABKT}
│└sdl 3.64t [8:176] Partitioned (gpt)
│ └sdl1 3.64t [8:177] MD raid6 (5/18 (13)) (w/
sdo1,sdn1,sdm1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:27 scsi 7:0:10:0 ATA Hitachi HUS72404 {PCG1JAVB}
│└sdm 3.64t [8:192] Partitioned (gpt)
│ └sdm1 3.64t [8:193] MD raid6 (15/18 (13)) (w/
sdo1,sdn1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:28 scsi 7:0:11:0 ATA Hitachi HUS72404 {PCGSLX3B}
│└sde 3.64t [8:64] Partitioned (gpt)
│ └sde1 3.64t [8:65] MD raid6 (14/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:29 scsi 7:0:12:0 ATA MB4000GCWDC {Z1Z8XQ9J}
│└sdk 3.64t [8:160] Partitioned (gpt)
│ └sdk1 3.64t [8:161] MD raid6 (11/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:30 scsi 7:0:13:0 ATA HITACHI HUS72404 {PAJK4VMS}
│└sdn 3.64t [8:208] Partitioned (gpt)
│ └sdn1 3.64t [8:209] MD raid6 (2/18 (13)) (w/
sdo1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:31 scsi 7:0:14:0 ATA Hitachi HUS72404 {PCG62VNB}
│└sdo 3.64t [8:224] Partitioned (gpt)
│ └sdo1 3.64t [8:225] MD raid6 (13/18 (13)) (w/
sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:32 scsi 7:0:15:0 ATA Hitachi HUS72404 {PCGSNTTB}
│└sdp 3.64t [8:240] Partitioned (gpt)
│ └sdp1 3.64t [8:241] MD raid6 (12/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sdq1,sda1)
spare 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
├phy-7:0:33 scsi 7:0:16:0 ATA HITACHI HUS72404 {PAGTSZES}
│└sdq 3.64t [65:0] Partitioned (gpt)
│ └sdq1 3.64t [65:1] MD raid6 (8/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdr1,sdb1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
│ └md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
│ PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
└phy-7:0:34 scsi 7:0:17:0 ATA WL4000GSA6472E {WOL240331985}
└sdr 3.64t [65:16] Partitioned (gpt)
└sdr1 3.64t [65:17] MD raid6 (6/18 (13)) (w/
sdo1,sdn1,sdm1,sdl1,sdk1,sdj1,sdi1,sdh1,sdg1,sdf1,sde1,sdd1,sdc1,sdb1,sdq1,sda1,sdp1)
in_sync 'debian:0' {2a0d5568-ea53-b429-30df-79c9e7559668}
└md0 40.02t [9:0] MD v1.2 raid6 (18 (13)) read-auto DEGRADED, 512k
Chunk {2a0d5568:ea53b429:30df79c9:e7559668}
PV LVM2_member 40.02t used, 0 free
{HUYjPl-6dU0-53un-71iH-1MAx-dEMU-HtvuV0}
Other Block Devices
├loop0 610.27m [7:0] squashfs
│├Mounted as /dev/loop0 @ /run/live/rootfs/filesystem.squashfs
│└Mounted as /dev/loop0 @ /usr/lib/live/mount/rootfs/filesystem.squashfs
├loop1 0.00k [7:1] Empty/Unknown
├loop2 0.00k [7:2] Empty/Unknown
├loop3 0.00k [7:3] Empty/Unknown
├loop4 0.00k [7:4] Empty/Unknown
├loop5 0.00k [7:5] Empty/Unknown
├loop6 0.00k [7:6] Empty/Unknown
└loop7 0.00k [7:7] Empty/Unknown
mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid6 sde1[16] sda1[3] sdj1[0] sdr1[6]
sdl1[5] sdi1[7] sdf1[4] sdo1[17] sdh1[14] sdq1[8] sdc1[13] sdn1[2]
sdm1[15] sdd1[1] sdp1[18] sdg1[9] sdk1[12] sdb1[10]
42975736320 blocks super 1.2 level 6, 512k chunk, algorithm 18
[18/17] [UUUUUUUUUUUU_UUUUU]
resync=PENDING
bitmap: 17/30 pages [68KB], 65536KB chunk
unused devices: <none>
Half the drives actually had SCT/ECR disabled. I ran the script which
set all the drives to 7s and tried reshaping again but still stalling.
On Sat, Mar 6, 2021 at 1:20 PM antlists [off-list ref] wrote:
https://raid.wiki.kernel.org/index.php/Linux_Raid#When_Things_Go_Wrogn
You've obviously covered some of this - glitches with Ubuntu and old
mdadm are common but this clearly isn't that ...
Please give us the information that page asks for - it probably won't
help much, but some things sometimes jump out ...
Cheers,
Wol
On 06/03/2021 11:36, apfc123@gmail.com wrote:
quoted
Hello,
Reshape is currently stalled at 58%. If I reboot the system, the array
is started in auto-read-only mode with resync pending. I can freeze
the sync and mount the filesystem read only to access the data.
When it first stalled and rebooted, I ran extended smart tests on all
18 drives and one came back with read error. I ddrescued it to a new
one (99.99% rescued) and swapped it out, but the reshape still stalls
immediately when setting the array to --readwrite.
The raid5 array started out with 12 drives. I added 6 more drives then
grew it to 18 drives and raid level 6 at the same time. This migration
was started with mdadm 4.1 running on kernel 4.19. I've tried booting
with debian testing running kernel 5.10 (didn't check mdadm version),
and also archlinux running 5.11 but with the same results.
/dev/md0:
Version : 1.2
Creation Time : Tue Dec 17 01:51:38 2019
Raid Level : raid6
Array Size : 42975736320 (40984.86 GiB 44007.15 GB)
Used Dev Size : 3906885120 (3725.90 GiB 4000.65 GB)
Raid Devices : 18
Total Devices : 18
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Sat Mar 6 09:41:58 2021
State : clean, degraded, resyncing (PENDING)
Active Devices : 17
Working Devices : 18
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric-6
Chunk Size : 512K
Consistency Policy : bitmap
Delta Devices : 5, (13->18)
New Layout : left-symmetric
Name : debian:0 (local to host debian)
UUID : 2a0d5568:ea53b429:30df79c9:e7559668
Events : 303246
Number Major Minor RaidDevice State
0 8 129 0 active sync /dev/sdi1
1 8 49 1 active sync /dev/sdd1
2 8 209 2 active sync /dev/sdn1
3 8 1 3 active sync /dev/sda1
4 8 65 4 active sync /dev/sde1
5 8 145 5 active sync /dev/sdj1
6 65 17 6 active sync /dev/sdr1
7 8 113 7 active sync /dev/sdh1
8 65 1 8 active sync /dev/sdq1
9 8 81 9 active sync /dev/sdf1
10 8 17 10 active sync /dev/sdb1
12 8 193 11 active sync /dev/sdm1
18 8 241 12 spare rebuilding /dev/sdp1
17 8 225 13 active sync /dev/sdo1
16 8 177 14 active sync /dev/sdl1
15 8 161 15 active sync /dev/sdk1
14 8 97 16 active sync /dev/sdg1
13 8 33 17 active sync /dev/sdc1
One of the entries from dmesg the first time reshaping stalled:
[105003.994653] INFO: task md0_reshape:3296 blocked for more than 120 seconds.
[105003.994916] Tainted: G I 4.19.0-6-amd64 #1
Debian 4.19.67-2+deb10u1
[105003.995169] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[105003.995434] md0_reshape D 0 3296 2 0x80000000
[105003.995436] Call Trace:
[105003.995441] ? __schedule+0x2a2/0x870
[105003.995442] schedule+0x28/0x80
[105003.995448] reshape_request+0x862/0x940 [raid456]
[105003.995451] ? finish_wait+0x80/0x80
[105003.995454] raid5_sync_request+0x34a/0x3b0 [raid456]
[105003.995460] md_do_sync.cold.86+0x3f4/0x911 [md_mod]
[105003.995461] ? finish_wait+0x80/0x80
[105003.995464] ? __switch_to_asm+0x35/0x70
[105003.995467] ? md_rdev_init+0xb0/0xb0 [md_mod]
[105003.995471] md_thread+0x94/0x150 [md_mod]
[105003.995473] kthread+0x112/0x130
[105003.995475] ? kthread_bind+0x30/0x30
[105003.995476] ret_from_fork+0x35/0x40
I'm not sure where to continue troubleshooting. Been moving drives
around in the storage appliance in case there are bad ports on the
backplane but doesn't seem to make any difference. I only have one
4-port HBA right now so can't even try another. The storage appliance
has 2 controllers and already tried swapping them as well as the SAS
cables. Only thing left I can try hardware wise is to install the
interposers that go in between the drives and the backplane but not
very hopeful.
Any help would be greatly appreciated. I don't have the extra space
right now to copy everything out and create a new array so really
hoping to get past this stalling issue. Thanks.