Started by Alexander Afonyashin on Aug 25, 2015 · 13 messages from 3 authors · Last reply Sep 1, 2015
open the first message on its own page
A Alexander Afonyashin #
Hi,
Recently I had to run fsck on 47TB ext4 partition backed by hardware
RAID6 (LSI MegaRAID SAS 2108). Right now over 2 weeks passed but fsck
is not finished yet. It occupies 30GB RSS, almost 35GB VSS and eats
100% of single CPU. It detected errors (and fixed them) but doesn't
finish yet.
Rescue disc is based on Debian 7.8.
kernel: 4.1.4-5
e2fsprogs: 1.42.5-1.1+deb7u1
Any suggestions?
Regards,
Alexander AfonyashinA Andreas Dilger replied to Alexander Afonyashin #
On Aug 25, 2015, at 9:30 AM, Alexander Afonyashin [off-list ref] wrote:
Hi, Recently I had to run fsck on 47TB ext4 partition backed by hardware RAID6 (LSI MegaRAID SAS 2108). Right now over 2 weeks passed but fsck is not finished yet. It occupies 30GB RSS, almost 35GB VSS and eats 100% of single CPU. It detected errors (and fixed them) but doesn't finish yet. Rescue disc is based on Debian 7.8. kernel: 4.1.4-5 e2fsprogs: 1.42.5-1.1+deb7u1 Any suggestions?
Usually the only reason for e2fsck to run so long is because of duplicate block pass 1b/1c. Having some of the actual output of e2fsck would allow us to give some useful advice. The only thing I can offer is for you to run "strace -p <e2fsck_pid>" and/or "ltrace -p <e2fsck_pid>" to see what it is doing. Cheers, Andreas
A Alexander Afonyashin replied to Andreas Dilger #
Hi,
The last output (2 days ago) from fsck:
[skipped]
Block #524296 (1235508688) causes directory to be too big. CLEARED.
Block #524297 (4003498426) causes directory to be too big. CLEARED.
Block #524298 (3113378389) causes directory to be too big. CLEARED.
Block #524299 (1368545889) causes directory to be too big. CLEARED.
Too many illegal blocks in inode 4425477.
Clear inode? yes
---------------------------
iostat output:
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 14.52 0.00 85.48
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
loop0 0.00 0.00 2.00 0.00 12.00 0.00
12.00 0.09 46.00 46.00 0.00 46.00 9.20
sda 0.00 0.00 87.00 0.00 348.00 0.00
8.00 1.00 11.86 11.86 0.00 11.45 99.60
---------------------------
strace ouput:
root@rescue ~ # strace -f -t -p 4779
Process 4779 attached - interrupt to quit
07:26:54 lseek(4, 14154266963968, SEEK_SET) = 14154266963968
07:26:54 read(4,
"\277\224\312\371\302\356\tJC{P\244#3\"2P\327*2Q5\372\206\262\20\\\373\226\262\21\316"...,
4096) = 4096
07:27:02 lseek(4, 1408506736640, SEEK_SET) = 1408506736640
07:27:02 read(4,
"\352\3041\345\1\337p\263l;\354\377E[\17\350\235\260\r\344\265\337\3655\223E\216\226\376\263!\n"...,
4096) = 4096
07:27:08 lseek(4, 5948177264640, SEEK_SET) = 5948177264640
07:27:08 read(4,On Aug 25, 2015, at 9:30 AM, Alexander Afonyashin [off-list ref] wrote:quoted
Hi, Recently I had to run fsck on 47TB ext4 partition backed by hardware RAID6 (LSI MegaRAID SAS 2108). Right now over 2 weeks passed but fsck is not finished yet. It occupies 30GB RSS, almost 35GB VSS and eats 100% of single CPU. It detected errors (and fixed them) but doesn't finish yet. Rescue disc is based on Debian 7.8. kernel: 4.1.4-5 e2fsprogs: 1.42.5-1.1+deb7u1 Any suggestions?Usually the only reason for e2fsck to run so long is because of duplicate block pass 1b/1c. Having some of the actual output of e2fsck would allow us to give some useful advice. The only thing I can offer is for you to run "strace -p <e2fsck_pid>" and/or "ltrace -p <e2fsck_pid>" to see what it is doing. Cheers, Andreas
A Alexander Afonyashin replied to Alexander Afonyashin #
Hi, I've restarted fsck ~6 hours ago. It's again occupied ~30GB RAM and strace shows that number of syscalls per second becomes fewer and fewer. Regards, Alexander On Thu, Aug 27, 2015 at 8:28 AM, Alexander Afonyashin [off-list ref] wrote:
Hi,
The last output (2 days ago) from fsck:
[skipped]
Block #524296 (1235508688) causes directory to be too big. CLEARED.
Block #524297 (4003498426) causes directory to be too big. CLEARED.
Block #524298 (3113378389) causes directory to be too big. CLEARED.
Block #524299 (1368545889) causes directory to be too big. CLEARED.
Too many illegal blocks in inode 4425477.
Clear inode? yes
---------------------------
iostat output:
avg-cpu: %user %nice %system %iowait %steal %idle
0.00 0.00 0.00 14.52 0.00 85.48
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
avgrq-sz avgqu-sz await r_await w_await svctm %util
loop0 0.00 0.00 2.00 0.00 12.00 0.00
12.00 0.09 46.00 46.00 0.00 46.00 9.20
sda 0.00 0.00 87.00 0.00 348.00 0.00
8.00 1.00 11.86 11.86 0.00 11.45 99.60
---------------------------
strace ouput:
root@rescue ~ # strace -f -t -p 4779
Process 4779 attached - interrupt to quit
07:26:54 lseek(4, 14154266963968, SEEK_SET) = 14154266963968
07:26:54 read(4,
"\277\224\312\371\302\356\tJC{P\244#3\"2P\327*2Q5\372\206\262\20\\\373\226\262\21\316"...,
4096) = 4096
07:27:02 lseek(4, 1408506736640, SEEK_SET) = 1408506736640
07:27:02 read(4,
"\352\3041\345\1\337p\263l;\354\377E[\17\350\235\260\r\344\265\337\3655\223E\216\226\376\263!\n"...,
4096) = 4096
07:27:08 lseek(4, 5948177264640, SEEK_SET) = 5948177264640
07:27:08 read(4,A Andreas Dilger replied to Alexander Afonyashin #
On Aug 27, 2015, at 8:23 AM, Alexander Afonyashin [off-list ref] wrote:
Hi, I've restarted fsck ~6 hours ago. It's again occupied ~30GB RAM and strace shows that number of syscalls per second becomes fewer and fewer.
My first suggestion would be to upgrade e2fsprogs to the latest stable version - 1.42.13 so that you are not hitting any older bugs. What was the original problem reported that caused the e2fsck to be run? Next, please include the full output from the start of e2fsck, unless it is just a lot of the same lines repeated. There are a lot of Lustre users with 32TB or 48TB ext4 filesystems that can finish a full e2fsck in a few hours, unless there is some kind of major corruption. It may be possible to fix some of the corruption manually with debugfs to avoid a lengthy e2fsck run. If you can run "ltrace -p <e2fsck_pid>" on the e2fsck then it would tell us what code it is running. It doesn't seem to be IO bound (only one seek+read per 6 seconds). Are there any special formatting options that were used for the filesystem originally? What does "debugfs -c -R stats <dev>" report about the filesystem? Cheers, Andreas
Regards, Alexander On Thu, Aug 27, 2015 at 8:28 AM, Alexander Afonyashin [off-list ref] wrote:quoted
Hi, The last output (2 days ago) from fsck: [skipped] Block #524296 (1235508688) causes directory to be too big. CLEARED. Block #524297 (4003498426) causes directory to be too big. CLEARED. Block #524298 (3113378389) causes directory to be too big. CLEARED. Block #524299 (1368545889) causes directory to be too big. CLEARED. Too many illegal blocks in inode 4425477. Clear inode? yes --------------------------- iostat output: avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 14.52 0.00 85.48 Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util loop0 0.00 0.00 2.00 0.00 12.00 0.00 12.00 0.09 46.00 46.00 0.00 46.00 9.20 sda 0.00 0.00 87.00 0.00 348.00 0.00 8.00 1.00 11.86 11.86 0.00 11.45 99.60 --------------------------- strace ouput: root@rescue ~ # strace -f -t -p 4779 Process 4779 attached - interrupt to quit 07:26:54 lseek(4, 14154266963968, SEEK_SET) = 14154266963968 07:26:54 read(4, "\277\224\312\371\302\356\tJC{P\244#3\"2P\327*2Q5\372\206\262\20\\\373\226\262\21\316"..., 4096) = 4096 07:27:02 lseek(4, 1408506736640, SEEK_SET) = 1408506736640
Cheers, Andreas
T Theodore Ts'o replied to Alexander Afonyashin #
On Thu, Aug 27, 2015 at 05:23:58PM +0300, Alexander Afonyashin wrote:
Hi, I've restarted fsck ~6 hours ago. It's again occupied ~30GB RAM and strace shows that number of syscalls per second becomes fewer and fewer.
Can you run it under "script" so we can get a transcript of the run?
It sounds like your file system has gotten very badly damaged, so the
question is figuring out what happened so we can advise you about how
to recover.
Can you also send the output of dumpe2fs?
Thanks,
- TedA Alexander Afonyashin replied to Theodore Ts'o #
Hi Ted,
Since the moment when fsck has been restarted yesterday (under screen)
it found only a few errors:
-----------------------------
root@rescue ~ # fsck -y -v /dev/sda3
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/dev/sda3 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 4425472 is too big. Truncate? yes
Block #593549637 (1359559318) causes file to be too big. CLEARED.
Block #593549638 (782521882) causes file to be too big. CLEARED.
Block #593549639 (1184464312) causes file to be too big. CLEARED.
Block #593549640 (1283655900) causes file to be too big. CLEARED.
Block #593549641 (2241107660) causes file to be too big. CLEARED.
Block #593549642 (117581257) causes file to be too big. CLEARED.
Block #593549643 (957560556) causes file to be too big. CLEARED.
Block #593549644 (1037291178) causes file to be too big. CLEARED.
Block #593549645 (2099220496) causes file to be too big. CLEARED.
Block #593549646 (539062498) causes file to be too big. CLEARED.
Block #593549647 (96880969) causes file to be too big. CLEARED.
Too many illegal blocks in inode 4425472.
Clear inode? yes
Inode 4425549, i_size is 18119674890280087280, should be 0. Fix? yes
Inode 4425549, i_blocks is 27717127168711, should be 0. Fix? yes
Inode 4425554 has compression flag set on filesystem without
compression support. Clear? yes
Inode 4425554, i_size is 12927291416082045099, should be 0. Fix? yes
Inode 4425554, i_blocks is 207957728820169, should be 0. Fix? yes
Inode 4425667, i_size is 3071282098034816027, should be 0. Fix? yes
Inode 4425667, i_blocks is 225290567745721, should be 0. Fix? yesOn Thu, Aug 27, 2015 at 05:23:58PM +0300, Alexander Afonyashin wrote:quoted
Hi, I've restarted fsck ~6 hours ago. It's again occupied ~30GB RAM and strace shows that number of syscalls per second becomes fewer and fewer.Can you run it under "script" so we can get a transcript of the run? It sounds like your file system has gotten very badly damaged, so the question is figuring out what happened so we can advise you about how to recover. Can you also send the output of dumpe2fs? Thanks, - Ted
A Alexander Afonyashin replied to Andreas Dilger #
Hi Andreas, Here's the ltrace output (it's definitely running infinitive loop)" root@rescue ~ # ltrace -p 31435 2>&1|head -n 30 ext2fs_mark_generic_bmap(0x797480, 0xf49f8345, 0x63e000, 0x64af40, 0x7ffe7148e8d0) = 0 ext2fs_blocks_count(0x641b00, 0x7ffe7148e998, 18, 0xffffffff, 0xf49f8346) = 0x2f47bfdfb ext2fs_test_generic_bmap(0x64ae80, 0xe4dcb10f, 0x63e000, 0, 0x7ffe7148e8d0) = 0 ext2fs_mark_generic_bmap(0x64ae80, 0xe4dcb10f, 0x63e000, 0x64af40, 0x7ffe7148e8d0) = 0 ext2fs_blocks_count(0x641b00, 0x7ffe7148e998, 18, 0xffffffff, 0xe4dcb110) = 0x2f47bfdfb ext2fs_test_generic_bmap(0x64ae80, 0x2c4ceefd, 0x63e000, 0, 0x7ffe7148e8d0) = 0 ext2fs_mark_generic_bmap(0x64ae80, 0x2c4ceefd, 0x63e000, 0x64af40, 0x7ffe7148e8d0) = 0 ext2fs_blocks_count(0x641b00, 0x7ffe7148e998, 18, 0xffffffff, 0x2c4ceefe) = 0x2f47bfdfb ext2fs_test_generic_bmap(0x64ae80, 0x27a62eff, 0x63e000, 0, 0x7ffe7148e8d0) = 0 ext2fs_mark_generic_bmap(0x64ae80, 0x27a62eff, 0x63e000, 0x64af40, 0x7ffe7148e8d0) = 0 ext2fs_blocks_count(0x641b00, 0x7ffe7148e998, 18, 0xffffffff, 0x27a62f00) = 0x2f47bfdfb ext2fs_test_generic_bmap(0x64ae80, 0x7887810d, 0x63e000, 0, 0x7ffe7148e8d0) = 1 ext2fs_mark_generic_bmap(0x797480, 0x7887810d, 0x63e000, 0x64af40, 0x7ffe7148e8d0) = 0 ext2fs_blocks_count(0x641b00, 0x7ffe7148e998, 18, 0xffffffff, 0x7887810e) = 0x2f47bfdfb [skipped] Right after running ltrace, fsck exited with the message: fsck: Warning... fsck.ext4 for device /dev/sda3 exited with signal 5. Will try to install latest version of e2fsprogs. root@rescue ~ # debugfs -c -R stats /dev/sda3 debugfs 1.42.5 (29-Jul-2012) /dev/sda3: catastrophic mode - not reading inode or group bitmaps
On Aug 27, 2015, at 8:23 AM, Alexander Afonyashin [off-list ref] wrote:quoted
Hi, I've restarted fsck ~6 hours ago. It's again occupied ~30GB RAM and strace shows that number of syscalls per second becomes fewer and fewer.My first suggestion would be to upgrade e2fsprogs to the latest stable version - 1.42.13 so that you are not hitting any older bugs. What was the original problem reported that caused the e2fsck to be run? Next, please include the full output from the start of e2fsck, unless it is just a lot of the same lines repeated. There are a lot of Lustre users with 32TB or 48TB ext4 filesystems that can finish a full e2fsck in a few hours, unless there is some kind of major corruption. It may be possible to fix some of the corruption manually with debugfs to avoid a lengthy e2fsck run. If you can run "ltrace -p <e2fsck_pid>" on the e2fsck then it would tell us what code it is running. It doesn't seem to be IO bound (only one seek+read per 6 seconds). Are there any special formatting options that were used for the filesystem originally? What does "debugfs -c -R stats <dev>" report about the filesystem? Cheers, Andreasquoted
Regards, Alexander On Thu, Aug 27, 2015 at 8:28 AM, Alexander Afonyashin [off-list ref] wrote:quoted
Hi, The last output (2 days ago) from fsck:
A Alexander Afonyashin replied to Alexander Afonyashin #
Hi, A brief story of problem. - hardware raid6 became partially degraded because one of disks failed (slot0 - mention it) as controller said - provider was asked to replace the failed drive (hot-swap) - while it was performing the task, the 2nd disk (slot4) has failed and raid became degraded (fully) - so provider was asked to replace 2nd disk too - I don't know what exactly happened (and how) but they replace disk in slot4 with disk from slot0 (see below - it's really looks like this) and inserted new disk into slot0 - system not booted due to 'no partitions' found (gpt) - I booted from rescue disk and found the cool thing: 1st LBA sector (GPT master sector) of LD0 (there was only one logical disk configured on controller) moved 1MB from start of logical disk. Paying attention that the strip size is 256K - this looks logical. In fact, controller holds raid metadata info on drives so the order which they are inserted into slots should not be a difference.I had experience with LSI controllers and it was so all the time. But this time it failed to recognize that disk was simply moved from one slot to another (may be due to the fact it has marked disk as failed - but suddenly it returned to life). I don't know if there's a bug in firmware or something else happened but when disk was placed back into original slot0 (keeping slot4 open) - GPT partition map has returned. But ... It seems that automatic rebuild had been started since first disk's replacement. And did its job. So I have partially broken ext4 that wish to fix. P.S. Raid hardware (performed by controller) rebuild process has been completed without errors. Regards, Alexander On Tue, Aug 25, 2015 at 6:30 PM, Alexander Afonyashin
Hi, Recently I had to run fsck on 47TB ext4 partition backed by hardware RAID6 (LSI MegaRAID SAS 2108). Right now over 2 weeks passed but fsck is not finished yet. It occupies 30GB RSS, almost 35GB VSS and eats 100% of single CPU. It detected errors (and fixed them) but doesn't finish yet. Rescue disc is based on Debian 7.8. kernel: 4.1.4-5 e2fsprogs: 1.42.5-1.1+deb7u1 Any suggestions? Regards, Alexander Afonyashin
A Andreas Dilger replied to Alexander Afonyashin #
On Aug 28, 2015, at 1:56 AM, Alexander Afonyashin [off-list ref] wrote:
Hi, A brief story of problem. - hardware raid6 became partially degraded because one of disks failed (slot0 - mention it) as controller said - provider was asked to replace the failed drive (hot-swap) - while it was performing the task, the 2nd disk (slot4) has failed and raid became degraded (fully) - so provider was asked to replace 2nd disk too - I don't know what exactly happened (and how) but they replace disk in slot4 with disk from slot0 (see below - it's really looks like this) and inserted new disk into slot0
I would suspect that the slot0 disk started to be overwritten by
the RAID rebuild while it was installed in slot4? It seems like
there is likely "confusing" corruption such as valid inode blocks
written to the wrong offset rather than just "random" corruption.
- system not booted due to 'no partitions' found (gpt) - I booted from rescue disk and found the cool thing: 1st LBA sector (GPT master sector) of LD0 (there was only one logical disk configured on controller) moved 1MB from start of logical disk. Paying attention that the strip size is 256K - this looks logical. In fact, controller holds raid metadata info on drives so the order which they are inserted into slots should not be a difference.I had experience with LSI controllers and it was so all the time. But this time it failed to recognize that disk was simply moved from one slot to another (may be due to the fact it has marked disk as failed - but suddenly it returned to life). I don't know if there's a bug in firmware or something else happened but when disk was placed back into original slot0 (keeping slot4 open) - GPT partition map has returned. But ... It seems that automatic rebuild had been started since first disk's replacement. And did its job. So I have partially broken ext4 that wish to fix. P.S. Raid hardware (performed by controller) rebuild process has been completed without errors.
In my experience, just because the RAID rebuild doesn't report any errors, doesn't mean that it didn't write random corruption across the disk. Depending on how badly the filesystem is corrupted, you might be able to recover some data, or you might have corruption every 8th or 10th block in your filesystem. Running e2fsck in such a case will make the metadata "valid" but it will not make the data correct. Definitely, if you have a filesystem over 16TB you should be running e2fsprogs-1.42.13 to get the latest fixes. Cheers, Andreas
Regards, Alexander On Tue, Aug 25, 2015 at 6:30 PM, Alexander Afonyashin [off-list ref] wrote:quoted
Hi, Recently I had to run fsck on 47TB ext4 partition backed by hardware RAID6 (LSI MegaRAID SAS 2108). Right now over 2 weeks passed but fsck is not finished yet. It occupies 30GB RSS, almost 35GB VSS and eats 100% of single CPU. It detected errors (and fixed them) but doesn't finish yet. Rescue disc is based on Debian 7.8. kernel: 4.1.4-5 e2fsprogs: 1.42.5-1.1+deb7u1 Any suggestions? Regards, Alexander Afonyashin-- To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Cheers, Andreas
T Theodore Ts'o replied to Alexander Afonyashin #
If dumpe2fs is hanging as well, it's likely that the problem may be at the hardware level. You might want to check dmesg or the kernel log to see if there are any I/O errors being reported from hard drive. What might be happening is that when a program (such as e2fsck or dumpe2fs) tries to read from a specific part of the hard drive, the hard drive is retrying a large number of times because the hard drive head or platter surface has gotten damaged in some way. It might also be a good idea to check the S.M.A.R.T. status using the smartctl program. Cheers, - Ted
A Alexander Afonyashin replied to Theodore Ts'o #
Hi, Running fsck from e2fsprogs-1.42.13 results in SIGKILL: Inode 4425496 is too big. Truncate? yes Block #524289 (103743230) causes directory to be too big. CLEARED. Block #524290 (3236857350) causes directory to be too big. CLEARED. Block #524291 (3625464338) causes directory to be too big. CLEARED. Block #524292 (1370882069) causes directory to be too big. CLEARED. Block #524293 (3868016883) causes directory to be too big. CLEARED. Block #524294 (3919147116) causes directory to be too big. CLEARED. Block #524295 (279419478) causes directory to be too big. CLEARED. Block #524296 (194746972) causes directory to be too big. CLEARED. Block #524297 (1695856868) causes directory to be too big. CLEARED. Block #524298 (587425254) causes directory to be too big. CLEARED. Block #524299 (142614537) causes directory to be too big. CLEARED. Too many illegal blocks in inode 4425496. Clear inode? yes Inode 4425357 has compression flag set on filesystem without compression support. Clear? yes Warning... fsck.ext4 for device /dev/sda3 exited with signal 9. root@rescue ~ # e2fsprogs-1.42.13/build/misc/fsck -v -y /dev/sda3 Regards, Alexander On Fri, Aug 28, 2015 at 8:53 PM, Theodore Ts'o [off-list ref] wrote:
If dumpe2fs is hanging as well, it's likely that the problem may be at
the hardware level. You might want to check dmesg or the kernel log
to see if there are any I/O errors being reported from hard drive.
What might be happening is that when a program (such as e2fsck or
dumpe2fs) tries to read from a specific part of the hard drive, the
hard drive is retrying a large number of times because the hard drive
head or platter surface has gotten damaged in some way.
It might also be a good idea to check the S.M.A.R.T. status using the
smartctl program.
Cheers,
- TedA Andreas Dilger replied to Alexander Afonyashin #
On Aug 31, 2015, at 1:20 AM, Alexander Afonyashin [off-list ref] wrote:
Hi, Running fsck from e2fsprogs-1.42.13 results in SIGKILL: Inode 4425496 is too big. Truncate? yes Block #524289 (103743230) causes directory to be too big. CLEARED. Block #524290 (3236857350) causes directory to be too big. CLEARED. Block #524291 (3625464338) causes directory to be too big. CLEARED. Block #524292 (1370882069) causes directory to be too big. CLEARED. Block #524293 (3868016883) causes directory to be too big. CLEARED. Block #524294 (3919147116) causes directory to be too big. CLEARED. Block #524295 (279419478) causes directory to be too big. CLEARED. Block #524296 (194746972) causes directory to be too big. CLEARED. Block #524297 (1695856868) causes directory to be too big. CLEARED. Block #524298 (587425254) causes directory to be too big. CLEARED. Block #524299 (142614537) causes directory to be too big. CLEARED. Too many illegal blocks in inode 4425496. Clear inode? yes Inode 4425357 has compression flag set on filesystem without compression support. Clear? yes Warning... fsck.ext4 for device /dev/sda3 exited with signal 9. root@rescue ~ # e2fsprogs-1.42.13/build/misc/fsck -v -y /dev/sda3
Hmm, the "fsck" command is just a wrapper, and it is not necessarily calling the e2fsck command from your build tree. You should run: e2fsprogs-1.42.13/build/e2fsck/e2fsck -fy /dev/sda3 That said, if you are having problems with the e2fsck, could you run it under gdb to see where it is failing? Signal 9 is SIGKILL which means that the process was killed by some external signal?
Regards, Alexander On Fri, Aug 28, 2015 at 8:53 PM, Theodore Ts'o [off-list ref] wrote:quoted
If dumpe2fs is hanging as well, it's likely that the problem may be at the hardware level. You might want to check dmesg or the kernel log to see if there are any I/O errors being reported from hard drive. What might be happening is that when a program (such as e2fsck or dumpe2fs) tries to read from a specific part of the hard drive, the hard drive is retrying a large number of times because the hard drive head or platter surface has gotten damaged in some way. It might also be a good idea to check the S.M.A.R.T. status using the smartctl program. Cheers, - Ted
Cheers, Andreas