From: Stan Hoeppner <hidden> Date: 2012-10-10 11:54:19
On 10/9/2012 10:14 PM, GuoZhong Han wrote:
Recently, a problem has troubled me for a long time.
I created a 4*2T (sda, sdb, sdc, sdd) raid5 with XFS file system, 128K
chuck size and 2048 strip_cache_size. The mdadm 3.2.2, kernel 2.6.38
and mkfs.xfs 3.1.1 were used. When the raid5 was in recovery and the
schedule reached 47%, I/O errors occurred in sdb. The following was
the output:
ata2: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x41 { DriveReady Error }
ata2: error=0x04 { DriveStatusError }
<snip repeated log entries>
end_request: I/O error, dev sdb, sector 1867304064
Run smartctl and post this section:
"Vendor Specific SMART Attributes with Thresholds"
The drive that is sdb may or may not be bad. smartctl may tell you
(us). If the drive is not bad you'll need to force relocation of this
bad sector to a spare. If you don't know how we can assist.
INFO: task xfssyncd/md127:1058 blocked for more than 120 seconds.
The output said “INFO: task xfssyncd/md127:1058 blocked for more than
120 seconds”. What did that mean?
Precisely what it says. It doesn't tell you WHY it was blocked, as it
can't know. The fact that your md array was in recovery and having
problems with one of the member drives is a good reason for xfssyncd to
block.
The state of the raid5 was “PENDING”. I had never seen such a
state of raid5 before. After that, I wrote a program to access the
raid5, there was no response any more. Then I used “ps aux| task
xfssyncd” to see the state of “xfssyncd”. Unfortunately, there was no
response yet. Then I tried “ps aux”. There were outputs, but the
program could exit with “Ctrl+d” or “Ctrl+z”. And when I tested the
write performance for raid5, I/O errors often occurred. I did not know
why this I/O errors occurred so frequently.
What was the problem? Can any one help me?
It looks like drive sdb is bad or going bad. smartctl output or
additional testing should confirm this.
Also, your "XFS...blocked for 120s" error reminds me there are some
known bugs in XFS kernel 2.6.38 which cause a similar error, but are not
the cause of your error. Yours is a drive problem. Nonetheless, there
have been dozens of XFS bugs fixed since 2.6.38 and I recommend you
upgrade to kernel 3.2.31 or 3.4.13 if you roll your own kernels. If you
use distro kernels get the latest 3.x series in the repos.
--
Stan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Stan Hoeppner
Date: 2012-10-10 19:54
To: GuoZhong Han
CC: linux-raid
Subject: Re: task xfssyncd blocked while raid5 was in recovery
On 10/9/2012 10:14 PM, GuoZhong Han wrote:
Recently, a problem has troubled me for a long time.
I created a 4*2T (sda, sdb, sdc, sdd) raid5 with XFS file system, 128K
chuck size and 2048 strip_cache_size. The mdadm 3.2.2, kernel 2.6.38
and mkfs.xfs 3.1.1 were used. When the raid5 was in recovery and the
schedule reached 47%, I/O errors occurred in sdb. The following was
the output:
ata2: translated ATA stat/err 0x41/04 to SCSI SK/ASC/ASCQ 0xb/00/00
ata2: status=0x41 { DriveReady Error }
ata2: error=0x04 { DriveStatusError }
end_request: I/O error, dev sdb, sector 1867304064
quoted
Run smartctl and post this section:
"Vendor Specific SMART Attributes with Thresholds"
quoted
The drive that is sdb may or may not be bad. smartctl may tell you
(us). If the drive is not bad you'll need to force relocation of this
bad sector to a spare. If you don't know how we can assist.
I did not save the outputs of smartctl, but I remember that the “RAW_VALUE”
of the attribute whose name was“Current_Pending_Sector” of sdb was 1. Did that
indicate that the drive was bad? If the drive was not bad, what is the best way to relocate
these bad sectors to spare? I have been using the tool "HDD_Regenerator" running in windows,
which is too slow. Each relocation took dozens of hours. It takes a long time to find the bad sector.
If you have any better idea, please let me know.
INFO: task xfssyncd/md127:1058 blocked for more than 120 seconds.
The output said "INFO: task xfssyncd/md127:1058 blocked for more than
120 seconds" What did that mean?
quoted
Precisely what it says. It doesn't tell you WHY it was blocked, as it
can't know. The fact that your md array was in recovery and having
problems with one of the member drives is a good reason for xfssyncd to
block.
The state of the raid5 was "PENDING". I had never seen such a
state of raid5 before. After that, I wrote a program to access the
raid5, there was no response any more. Then I used "ps aux| grep
xfssyncd" to see the state of xfssyncd? Unfortunately, there was no
response yet. Then I tried "ps aux". There were outputs, but the
program could exit with "ctrl+d" or "ctrl+z". And when I tested the
write performance for raid5, I/O errors often occurred. I did not know
why this I/O errors occurred so frequently.
What was the problem? Can any one help me?
quoted
It looks like drive sdb is bad or going bad. smartctl output or
additional testing should confirm this.
quoted
Also, your "XFS...blocked for 120s" error reminds me there are some
known bugs in XFS kernel 2.6.38 which cause a similar error, but are not
the cause of your error. Yours is a drive problem. Nonetheless, there
have been dozens of XFS bugs fixed since 2.6.38 and I recommend you
upgrade to kernel 3.2.31 or 3.4.13 if you roll your own kernels. If you
use distro kernels get the latest 3.x series in the repos.
Hm, I searched "XFS.. blocked for 120s" on google. There was a same problem
with me, the flowing is the link:
http://lists.debian.org/debian-kernel/2012/01/msg00274.html
I use the tilera platform, so it is hard to upgrade to kernel 3.2.31 or 3.4.13.
From: Chris Murphy <hidden> Date: 2012-10-11 03:47:15
On Oct 10, 2012, at 8:42 PM, hanguozhong wrote:
I did not save the outputs of smartctl, but I remember that the “RAW_VALUE”
of the attribute whose name was“Current_Pending_Sector” of sdb was 1. Did that
indicate that the drive was bad?
Just one sector is bad. But there are other attributes, maybe the disk isn't spinning up, maybe there are a lot of other read errors or uncorrected ECC errors. So I think we need more data.
smartctl -A /dev/sdb
If the drive was not bad, what is the best way to relocate
these bad sectors to spare? I have been using the tool "HDD_Regenerator" running in windows,
which is too slow. Each relocation took dozens of hours. It takes a long time to find the bad sector.
If you have any better idea, please let me know.
Bad sectors can be like mice. Chances are, you've got more than one. I would obliterate the disk with Secure Erase. It's faster than writing zeros with dd. But I question why normal usage isn't causing the sectors to be reallocated and you continue to have errors. I personally would replace the disk, and get the array back to normal health, and then futz with the misbehaving drive on a separate system - if this is an important array.
Chris Murphy
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Mikael Abrahamsson <hidden> Date: 2012-10-11 06:12:32
On Wed, 10 Oct 2012, Stan Hoeppner wrote:
Precisely what it says. It doesn't tell you WHY it was blocked, as it
can't know. The fact that your md array was in recovery and having
problems with one of the member drives is a good reason for xfssyncd to
block.
Doesn't he still have 3 good drives? So since sdb was failed, there would
be no reason for sdb to cause blocking or writes to the (now degraded)
raid5? OP said he saw write IO errors to the array (?), which I thought
was strange.
--
Mikael Abrahamsson email: swmike@swm.pp.se
From: Stan Hoeppner <hidden> Date: 2012-10-11 11:01:58
On 10/11/2012 1:12 AM, Mikael Abrahamsson wrote:
On Wed, 10 Oct 2012, Stan Hoeppner wrote:
quoted
Precisely what it says. It doesn't tell you WHY it was blocked, as it
can't know. The fact that your md array was in recovery and having
problems with one of the member drives is a good reason for xfssyncd
to block.
Doesn't he still have 3 good drives? So since sdb was failed, there
would be no reason for sdb to cause blocking or writes to the (now
degraded) raid5? OP said he saw write IO errors to the array (?), which
I thought was strange.
I think the more important question is, why was the OP writing to a
filesystem on a small RAID5 array while it was doing a rebuild? My
assumption here was that the seek load due to the rebuild was what
caused xfssyncd to block. IO Load is a common cause of this particular
kernel message when it relates to XFS.
Note this msg is not an error, but informational, thus the "INFO: " tag.
This is why you have the option to turn off the message. Also note
there are many conditions that will prompt this "blocked for 120s"
kernel message, and that this message is not unique to XFS. Point of
fact, Google for "blocked for more than 120 seconds" and no results on
the first page relate to XFS.
--
Stan
From: Mikael Abrahamsson <hidden> Date: 2012-10-11 11:16:14
On Thu, 11 Oct 2012, Stan Hoeppner wrote:
quoted
Doesn't he still have 3 good drives? So since sdb was failed, there
would be no reason for sdb to cause blocking or writes to the (now
degraded) raid5? OP said he saw write IO errors to the array (?), which
I thought was strange.
I think the more important question is, why was the OP writing to a
filesystem on a small RAID5 array while it was doing a rebuild?
Why is that an important question?
Even if he was, should there ever be IO write errors on it, even if it has
a lot of load on it?
--
Mikael Abrahamsson email: swmike@swm.pp.se
That's not the same problem as you. The INFO text doesn't tell you the
source of the problem, the included thread information does. To debug
that you need to be an XFS developer. Which is why you should report
that to the xfs list, not here.
I use the tilera platform, so it is hard to upgrade to kernel 3.2.31 or 3.4.13.
Ahh, I didn't recognize your name. You're the guy trying to build the 8
drive storage appliance with a 36 core Tilera chip.
What is the latest Tilera supported kernel? Do they support any 3.x
series kernels? How often do they update their kernel and userland code
so you can get badly needed bug fixes?
This is the 2nd glaring reason demonstrating you should have gone with
x86, or another fully supported low core count low power CPU (MIPS), as
I previously mentioned.
--
Stan