Re: RAID5 Phantom Drive Appeared while Reshaping Four Drive Array (HARDLOCK)
From: raid <hidden>
Date: 2023-05-22 07:23:18
Hi,
Thanks for the guidance as the current state has at least changed somewhat.
BTW Sorry about Life getting in the way of tech. =) Reason for my delayed response.
-sudo mdadm -I /dev/sdc1
mdadm: /dev/sdc1 attached to /dev/md480, not enough to start (1).
-sudo mdadm -D /dev/md480
/dev/md480:
Version : 1.2
Raid Level : raid0
Total Devices : 1
Persistence : Superblock is persistent
State : inactive
Working Devices : 1
Delta Devices : 1, (-1->0)
New Level : raid5
New Layout : left-symmetric
New Chunksize : 512K
Name : GRANDSLAM:480
UUID : 20211025:02005a7a:5a7abeef:cafebabe
Events : 78714
Number Major Minor RaidDevice
- 8 33 - /dev/sdc1
-sudo mdadm -I /dev/sdd1
mdadm: /dev/sdd1 attached to /dev/md480, not enough to start (2).
-sudo mdadm -D /dev/md480
/dev/md480:
Version : 1.2
Raid Level : raid0
Total Devices : 2
Persistence : Superblock is persistent
State : inactive
Working Devices : 2
Delta Devices : 1, (-1->0)
New Level : raid5
New Layout : left-symmetric
New Chunksize : 512K
Name : GRANDSLAM:480
UUID : 20211025:02005a7a:5a7abeef:cafebabe
Events : 78714
Number Major Minor RaidDevice
- 8 49 - /dev/sdd1
- 8 33 - /dev/sdc1
-sudo mdadm -I /dev/sde1
mdadm: /dev/sde1 attached to /dev/md480, not enough to start (2).
-sudo mdadm -D /dev/md480
/dev/md480:
Version : 1.2
Raid Level : raid0
Total Devices : 3
Persistence : Superblock is persistent
State : inactive
Working Devices : 3
Delta Devices : 1, (-1->0)
New Level : raid5
New Layout : left-symmetric
New Chunksize : 512K
Name : GRANDSLAM:480
UUID : 20211025:02005a7a:5a7abeef:cafebabe
Events : 78712
Number Major Minor RaidDevice
- 8 65 - /dev/sde1
- 8 49 - /dev/sdd1
- 8 33 - /dev/sdc1
-sudo mdadm -I /dev/sdf1
mdadm: /dev/sdf1 attached to /dev/md480, not enough to start (3).
-sudo mdadm -D /dev/md480
/dev/md480:
Version : 1.2
Raid Level : raid0
Total Devices : 4
Persistence : Superblock is persistent
State : inactive
Working Devices : 4
Delta Devices : 1, (-1->0)
New Level : raid5
New Layout : left-symmetric
New Chunksize : 512K
Name : GRANDSLAM:480
UUID : 20211025:02005a7a:5a7abeef:cafebabe
Events : 78714
Number Major Minor RaidDevice
- 8 81 - /dev/sdf1
- 8 65 - /dev/sde1
- 8 49 - /dev/sdd1
- 8 33 - /dev/sdc1
-sudo mdadm -R /dev/md480
mdadm: failed to start array /dev/md480: Input/output error
---
NOTE: Of additional interest...
---
-sudo mdadm -D /dev/md480
/dev/md480:
Version : 1.2
Creation Time : Tue Oct 26 14:06:53 2021
Raid Level : raid5
Used Dev Size : 18446744073709551615
Raid Devices : 5
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Thu May 4 14:39:03 2023
State : active, FAILED, Not Started
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Consistency Policy : unknown
Delta Devices : 1, (4->5)
Name : GRANDSLAM:480
UUID : 20211025:02005a7a:5a7abeef:cafebabe
Events : 78714
Number Major Minor RaidDevice State
- 0 0 0 removed
- 0 0 1 removed
- 0 0 2 removed
- 0 0 3 removed
- 0 0 4 removed
- 8 81 3 sync /dev/sdf1
- 8 49 1 sync /dev/sdd1
- 8 33 0 sync /dev/sdc1
---
-watch -c -d -n 1 cat /proc/mdstat
---
Every 1.0s: cat /proc/mdstat OAK2023: Mon May 22 01:48:24 2023
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md480 : inactive sdf1[4] sdd1[1] sdc1[0]
46877239294 blocks super 1.2
unused devices: <none>
---
Hopeful that is some progress towards an array start? It's definately unexpected output to me.
I/O Error starting md480
Thanks!
SA
On Thu, 2023-05-18 at 11:15 +0800, Yu Kuai wrote:Hi, 在 2023/05/18 7:45, Wol 写道:quoted
Hmmm. Firstly, what command did you give to grow the array? Secondly, take a look at the thread "Raid5 to raid6 grow interrupted, mdadm hangs on assemble command". There's a problem there with rebuilds locking up, which is not fatal, and will be fixed, but might not have rippled through yet ... That raid0 thing is almost certainly nothing to be worried about - it seems to be normal for any array that doesn't assemble completely. The only things that bother me slightly are I believe mdadm 4.2 has been released? Don't quote me on that. And scterc is disabled by default? Weird. I've cc'd a few people who I hope can help further ...Hi, please cc yukuai3@huawei.com for me, huaweicloud email is just for send, I don't receive emails from this...quoted
Cheers, Wol On 17/05/2023 14:26, raid wrote:quoted
RAID5 Phantom Drive Appeared while Reshaping Four Drive Array (HARDLOCK) I've been struggling with this for about two weeks now, realizing that I need some expert help. My original 18 month old RAID5 consists of three newer TOSHIBA drives. /dev/sdc :: TOSHIBA MG08ACA16TE (4303) :: 16 TB (16,000,900,661,248 bytes) /dev/sdd :: TOSHIBA MG08ACA16TE (4303) :: 16 TB (16,000,900,661,248 bytes) /dev/sde :: TOSHIBA MG08ACA16TE (4002) :: 16 TB (16,000,900,661,248 bytes) Recently added... /dev/sdf :: TOSHIBA MG08ACA16TE (4303) :: 16 TB (16,000,900,661,248 bytes) In a nutshell, I've added a fourth drive to my RAID5 and executed -- grow & mdadm estimated completion in 3-5 days. At about 30-50% of reshaping, the computer hard locked. Pushing the reset button was the agonizing requirement. After first reboot mdadm assembled & continued. But it displayed a fifth physical disk. The phantom FIFTH drive appeared as failed, while the other four continued reshaping, temporarily. The reshaping speed dropped to 0 after another day or so. It was near 80%, I think. So, I used mdadm -S then mdadm --assemble --scan it couldn't start (because phantom drive?) not enough drives to start the array. The Array State on each member shows the fifth drive with varying status. File system (ext4) appears damaged and won't mount. Unrecognized filesystem. 20TB are backed up, there are, however, about 7000 newly scanned documents that aren't. I've done a cursory examination of data using R-Linux. Abit of in depth peeking using Active Disk Editor. Life goes on. I've researched and read way more than I ever thought I would about mdadm RAID. Not any closer on how to proceed. I'm a hardware technician with some software skills. I'm stumped. Also trying to be cautious not to damage whats left of the RAID. ANY help with what commands I can attempt to at least get the RAID to assemble WITHOUT the phantom fifth drive would be immensely appreciated. All four drives now appear as spares. --- watch -c -d -n 1 cat /proc/mdstat md480 : inactive sdc1[0](S) sdd1[1](S) sdf1[4](S) sde1[3](S) 62502985709 blocks super 1.2 --- uname -a Linux OAK2023 4.19.0-24-amd64 #1 SMP Debian 4.19.282-1 (2023-04-29) x86_64 GNU/Linux --- mdadm --version mdadm - v4.1 - 2018-10-01 --- mdadm -E /dev/sd[c-f]1 /dev/sdc1: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 20211025:02005a7a:5a7abeef:cafebabe Name : GRANDSLAM:480 Creation Time : Tue Oct 26 14:06:53 2021 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 31251492831 (14901.87 GiB 16000.76 GB) Array Size : 62502983680 (59607.49 GiB 64003.06 GB) Used Dev Size : 31251491840 (14901.87 GiB 16000.76 GB) Data Offset : 264192 sectors New Offset : 261120 sectors Super Offset : 8 sectors State : clean Device UUID : 8f0835db:3ea24540:2ab4232d:6203d1b7 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 51850891264 (49448.86 GiB 53095.31 GB) Delta Devices : 1 (4->5) Update Time : Thu May 4 14:39:03 2023 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 37ac3c04 - correct Events : 78714 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 0 Array State : AA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sdd1: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 20211025:02005a7a:5a7abeef:cafebabe Name : GRANDSLAM:480 Creation Time : Tue Oct 26 14:06:53 2021 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 31251492831 (14901.87 GiB 16000.76 GB) Array Size : 62502983680 (59607.49 GiB 64003.06 GB) Used Dev Size : 31251491840 (14901.87 GiB 16000.76 GB) Data Offset : 264192 sectors New Offset : 261120 sectors Super Offset : 8 sectors State : clean Device UUID : b4660f49:867b9f1e:ecad0ace:c7119c37 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 51850891264 (49448.86 GiB 53095.31 GB) Delta Devices : 1 (4->5) Update Time : Thu May 4 14:39:03 2023 Bad Block Log : 512 entries available at offset 72 sectors Checksum : a4927b98 - correct Events : 78714 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 1 Array State : AA.A. ('A' == active, '.' == missing, 'R' == replacing) /dev/sde1: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 20211025:02005a7a:5a7abeef:cafebabe Name : GRANDSLAM:480 Creation Time : Tue Oct 26 14:06:53 2021 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 31251492831 (14901.87 GiB 16000.76 GB) Array Size : 62502983680 (59607.49 GiB 64003.06 GB) Used Dev Size : 31251491840 (14901.87 GiB 16000.76 GB) Data Offset : 264192 sectors New Offset : 261120 sectors Super Offset : 8 sectors State : clean Device UUID : 79a3dff4:c53f9071:f9c1c262:403fbc10 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 51850891264 (49448.86 GiB 53095.31 GB) Delta Devices : 1 (4->5) Update Time : Thu May 4 14:38:38 2023 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 112fbe09 - correct Events : 78712 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 2 Array State : AAAA. ('A' == active, '.' == missing, 'R' == replacing)I have no idle why other disk shows that device 2 is missing, and what is device 4. Anyway, can you try the following? mdadm -I /dev/sdc1 mdadm -D /dev/mdxxx mdadm -I /dev/sdd1 mdadm -D /dev/mdxxx mdadm -I /dev/sde1 mdadm -D /dev/mdxxx mdadm -I /dev/sdf1 mdadm -D /dev/mdxxx If above works well, you can try: mdadm -R /dev/mdxxx, and see if the array can be started. Thanks, Kuaiquoted
quoted
/dev/sdf1: Magic : a92b4efc Version : 1.2 Feature Map : 0x45 Array UUID : 20211025:02005a7a:5a7abeef:cafebabe Name : GRANDSLAM:480 Creation Time : Tue Oct 26 14:06:53 2021 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 31251492926 (14901.87 GiB 16000.76 GB) Array Size : 62502983680 (59607.49 GiB 64003.06 GB) Used Dev Size : 31251491840 (14901.87 GiB 16000.76 GB) Data Offset : 264192 sectors New Offset : 261120 sectors Super Offset : 8 sectors State : clean Device UUID : 9d9c1c0d:030844a7:f365ace6:5e568930 Internal Bitmap : 8 sectors from superblock Reshape pos'n : 51850891264 (49448.86 GiB 53095.31 GB) Delta Devices : 1 (4->5) Update Time : Thu May 4 14:39:03 2023 Bad Block Log : 512 entries available at offset 72 sectors Checksum : 2d33aff - correct Events : 78714 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 3 Array State : AA.A. ('A' == active, '.' == missing, 'R' == replacing) --- mdadm -E /dev/sd[c-f]1 | grep -E '^/dev/sd|Update' /dev/sdc1: Update Time : Thu May 4 14:39:03 2023 /dev/sdd1: Update Time : Thu May 4 14:39:03 2023 /dev/sde1: Update Time : Thu May 4 14:38:38 2023 /dev/sdf1: Update Time : Thu May 4 14:39:03 2023 --- mdadm --assemble --scan mdadm: /dev/md/GRANDSLAM:480 assembled from 3 drives - not enough to start the array. --- /etc/mdadm/mdadm.conf # This configuration was auto-generated on Tue, 26 Oct 2021 12:52:33 -0500 by mkconf ARRAY /dev/md480 metadata=1.2 name=GRANDSLAM:480 UUID=20211025:02005a7a:5a7abeef:cafebabe --- NOTE: Raid Level is now shown below to be raid0. This is a RAID5. Delta Devices are munged? NOW;mdadm -D /dev/md480 2023.05.17 02:44:06 AM /dev/md480: Version : 1.2 Raid Level : raid0 Total Devices : 4 Persistence : Superblock is persistent State : inactive Working Devices : 4 Delta Devices : 1, (-1->0) New Level : raid5 New Layout : left-symmetric New Chunksize : 512K Name : GRANDSLAM:480 UUID : 20211025:02005a7a:5a7abeef:cafebabe Events : 78714 Number Major Minor RaidDevice - 8 81 - /dev/sdf1 - 8 65 - /dev/sde1 - 8 49 - /dev/sdd1 - 8 33 - /dev/sdc1 --- NOTE: The HITACHI MG08ACA16TE drives default to DISABLED I've since enabled the setting if this helps. smartctl -l scterc /dev/sdc; smartctl -l scterc /dev/sdd; smartctl -l scterc /dev/sde; smartctl -l scterc /dev/sdf smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-24-amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-24-amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-24-amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.19.0-24-amd64] (local build) Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org SCT Error Recovery Control: Read: 70 (7.0 seconds) Write: 70 (7.0 seconds) --- Exhausted and maybe I'm just looking for someone to suggest running the command that I really don't want to run yet. Enabling Loss Of Confusion flag hasn't worked either..