From: Igor M Podlesny <hidden> Date: 2012-05-27 11:32:40
A power outage has happened to system (which was in "suspend"
state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
auto-readonly mode with resync=DELAYED. When they flipped to
Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
Occurred once on Linux 3.2.12, x86_64.
--
On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny [off-list ref]
wrote:
A power outage has happened to system (which was in "suspend"
state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
auto-readonly mode with resync=DELAYED. When they flipped to
Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
Maybe. But with so little detail it is hard to say.
What made you think the bitmaps weren't being used?
Did that arrays still have their bitmaps?
Where they degraded?
How many bits were set in the bitmaps (probably too late to check now)?
Is it reproducable ? ... OK, I don't really expect you to try to reproduce
it, but if you could that would be awesome!
NeilBrown
Occurred once on Linux 3.2.12, x86_64.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Igor M Podlesny <hidden> Date: 2012-05-27 16:10:33
On 27 May 2012 20:00, NeilBrown [off-list ref] wrote:
On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny [off-list ref]
wrote:
quoted
A power outage has happened to system (which was in "suspend"
state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
auto-readonly mode with resync=DELAYED. When they flipped to
Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
Maybe. But with so little detail it is hard to say.
What made you think the bitmaps weren't being used?
Well, it is reasonable to assume that host being put in suspend
mode wouldn't have much not synced data at all. RAID-10 had been
progressing its resync quite fast, it was obvious that it had been
using bitmaps for that. Moreover — RAID-10 was the most I/O active
comparing to others, AFAIR.
Did that arrays still have their bitmaps?
I still have that RAID-6, yep:
md127 : active raid6 sdg5[2] sdf5[6] sde5[7] sda5[0] sdd5[3] sdc5[1]
1071984640 blocks super 1.0 level 6, 1024k chunk, algorithm 2
[6/6] [UUUUUU]
bitmap: 0/1 pages [0KB], 131072KB chunk
Where they degraded?
No way.
How many bits were set in the bitmaps (probably too late to check now)?
Please see above. And the RAID-5 one was recently dismissed, alas.
Is it reproducable ? ... OK, I don't really expect you to try to reproduce
it, but if you could that would be awesome!
Well, might be VirtualBox would allow for such kind of experiment, dunno. )
P. s. Oh, and BTW, I see 2.6.18 supplied by RedHat still has the
bug causing kernel panic when using WIB -- couldn't you please point
out is it due to they're missing some important bugfix you made later?
--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 28 May 2012 00:10:33 +0800 Igor M Podlesny [off-list ref]
wrote:
On 27 May 2012 20:00, NeilBrown [off-list ref] wrote:
quoted
On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny [off-list ref]
wrote:
quoted
A power outage has happened to system (which was in "suspend"
state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
auto-readonly mode with resync=DELAYED. When they flipped to
Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
Maybe. But with so little detail it is hard to say.
What made you think the bitmaps weren't being used?
Well, it is reasonable to assume that host being put in suspend
mode wouldn't have much not synced data at all. RAID-10 had been
progressing its resync quite fast, it was obvious that it had been
using bitmaps for that. Moreover — RAID-10 was the most I/O active
comparing to others, AFAIR.
Unfortunately vague statements about "it was obvious" or "AFAIR" don't really
help in analysing a situation to find any bugs.
I can only really work with concrete facts. Without them I cannot help.
quoted
Did that arrays still have their bitmaps?
I still have that RAID-6, yep:
md127 : active raid6 sdg5[2] sdf5[6] sde5[7] sda5[0] sdd5[3] sdc5[1]
1071984640 blocks super 1.0 level 6, 1024k chunk, algorithm 2
[6/6] [UUUUUU]
bitmap: 0/1 pages [0KB], 131072KB chunk
quoted
Where they degraded?
No way.
quoted
How many bits were set in the bitmaps (probably too late to check now)?
Please see above. And the RAID-5 one was recently dismissed, alas.
quoted
Is it reproducable ? ... OK, I don't really expect you to try to reproduce
it, but if you could that would be awesome!
Well, might be VirtualBox would allow for such kind of experiment, dunno. )
P. s. Oh, and BTW, I see 2.6.18 supplied by RedHat still has the
bug causing kernel panic when using WIB -- couldn't you please point
out is it due to they're missing some important bugfix you made later?
I have no time or interest for submiting bug reports to distros that I don't
use. If the presence of the bug concerns you, I suggest you report it.
NeilBrown
From: Igor M Podlesny <hidden> Date: 2012-05-28 02:37:14
On 28 May 2012 09:45, NeilBrown [off-list ref] wrote:
On Mon, 28 May 2012 00:10:33 +0800 Igor M Podlesny [off-list ref]
wrote:
quoted
On 27 May 2012 20:00, NeilBrown [off-list ref] wrote:
quoted
On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny [off-list ref]
wrote:
quoted
A power outage has happened to system (which was in "suspend"
state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
auto-readonly mode with resync=DELAYED. When they flipped to
Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
Maybe. But with so little detail it is hard to say.
What made you think the bitmaps weren't being used?
Well, it is reasonable to assume that host being put in suspend
mode wouldn't have much not synced data at all. RAID-10 had been
progressing its resync quite fast, it was obvious that it had been
using bitmaps for that. Moreover — RAID-10 was the most I/O active
comparing to others, AFAIR.
Unfortunately vague statements about "it was obvious" or "AFAIR" don't really
help in analysing a situation to find any bugs.
I can only really work with concrete facts. Without them I cannot help.
Well, than probably you could tell me how one does find out whether
recovery uses WIB, or it doesn't? :-) I mean aside from vague
estimating based on watching re-sync progressing and considering
overall time it finally took to complete.
quoted
P. s. Oh, and BTW, I see 2.6.18 supplied by RedHat still has the
bug causing kernel panic when using WIB -- couldn't you please point
out is it due to they're missing some important bugfix you made later?
I have no time or interest for submiting bug reports to distros that I don't
use. If the presence of the bug concerns you, I suggest you report it.
I thought you could have some recollections bout this rather nasty
behavior. It has nothing to do with bug-reports for RedHat since
vanilla 2.6.18 for sure has the same bug, and hopefully it has been
fixed in later releases -- that was the only thing I was asking you
about. But your competent suggestions are also appreciated, of course.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 28 May 2012 10:37:14 +0800 Igor M Podlesny [off-list ref]
wrote:
On 28 May 2012 09:45, NeilBrown [off-list ref] wrote:
quoted
On Mon, 28 May 2012 00:10:33 +0800 Igor M Podlesny [off-list ref]
wrote:
quoted
On 27 May 2012 20:00, NeilBrown [off-list ref] wrote:
quoted
On Sun, 27 May 2012 19:32:40 +0800 Igor M Podlesny [off-list ref]
wrote:
quoted
A power outage has happened to system (which was in "suspend"
state) with RAID-10, RAID-5 and RAID-6. Later on boot, RAID-10 was
fast re-synced using bitmap, but both RAID-5 and RAID-6 were in
auto-readonly mode with resync=DELAYED. When they flipped to
Read-Write, LSR started re-syncing not using bitmaps seemingly. A bug?
Maybe. But with so little detail it is hard to say.
What made you think the bitmaps weren't being used?
Well, it is reasonable to assume that host being put in suspend
mode wouldn't have much not synced data at all. RAID-10 had been
progressing its resync quite fast, it was obvious that it had been
using bitmaps for that. Moreover — RAID-10 was the most I/O active
comparing to others, AFAIR.
Unfortunately vague statements about "it was obvious" or "AFAIR" don't really
help in analysing a situation to find any bugs.
I can only really work with concrete facts. Without them I cannot help.
Well, than probably you could tell me how one does find out whether
recovery uses WIB, or it doesn't? :-) I mean aside from vague
estimating based on watching re-sync progressing and considering
overall time it finally took to complete.
The best indicator is total time that it takes (which can probably be
extracted from logs as start and end are logged). Divide that into size of a
device to get average MB/sec. If the bitmap was used, that will normally be
much less the best throughput of the device.
quoted
quoted
P. s. Oh, and BTW, I see 2.6.18 supplied by RedHat still has the
bug causing kernel panic when using WIB -- couldn't you please point
out is it due to they're missing some important bugfix you made later?
I have no time or interest for submiting bug reports to distros that I don't
use. If the presence of the bug concerns you, I suggest you report it.
I thought you could have some recollections bout this rather nasty
behavior. It has nothing to do with bug-reports for RedHat since
vanilla 2.6.18 for sure has the same bug, and hopefully it has been
fixed in later releases -- that was the only thing I was asking you
about. But your competent suggestions are also appreciated, of course.
2.6.18 is a long time ago. I don't really have any recollections - so many
bugs, so many releases. I'm not even entirely sure what bug you are
referring to. I would probably have tagged any patch for it to be included
in -stable, but I cannot be certain without looking more deeply.
NeilBrown
From: Igor M Podlesny <hidden> Date: 2012-05-28 04:03:30
On 28 May 2012 10:54, NeilBrown [off-list ref] wrote:
[...]
The best indicator is total time that it takes (which can probably be
extracted from logs as start and end are logged). Divide that into size of a
device to get average MB/sec. If the bitmap was used, that will normally be
much less the best throughput of the device.
That's exactly why I am stating it didn't use WIBs 2 times of 3. It
went resyncing from beginning till the end -- I had my finger on its
pulse. :)
I can't promise (due to popular "lack of time" disease, yeah), but
if I'll get on it using VirtualBox or real environment [Lord forbid
:)] once again, I'll let you know, sure.
[...]
2.6.18 is a long time ago. I don't really have any recollections - so many
bugs, so many releases. I'm not even entirely sure what bug you are
referring to. I would probably have tagged any patch for it to be included
in -stable, but I cannot be certain without looking more deeply.
Okay, thanks, it gonna take some kind of bisecting I guess, then.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Igor M Podlesny <hidden> Date: 2012-06-20 02:49:32
On 28 May 2012 12:03, Igor M Podlesny [off-list ref] wrote:
On 28 May 2012 10:54, NeilBrown [off-list ref] wrote:
[...]
quoted
The best indicator is total time that it takes (which can probably be
extracted from logs as start and end are logged). Divide that into size of a
device to get average MB/sec. If the bitmap was used, that will normally be
much less the best throughput of the device.
That's exactly why I am stating it didn't use WIBs 2 times of 3. It
went resyncing from beginning till the end -- I had my finger on its
pulse. :)
I can't promise (due to popular "lack of time" disease, yeah), but
if I'll get on it using VirtualBox or real environment [Lord forbid
:)] once again, I'll let you know, sure.
Here it is: http://pastie.org/4118133
— 3.4.2-rt10 went suspend ok for 2 times, the 3rd it didn't.
Reboot, resync, no bitmaps use.
And, BTW, it's rather slow in despite of having max. stripe_cache_size set.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 20 Jun 2012 10:49:32 +0800 Igor M Podlesny [off-list ref]
wrote:
On 28 May 2012 12:03, Igor M Podlesny [off-list ref] wrote:
quoted
On 28 May 2012 10:54, NeilBrown [off-list ref] wrote:
[...]
quoted
The best indicator is total time that it takes (which can probably be
extracted from logs as start and end are logged). Divide that into size of a
device to get average MB/sec. If the bitmap was used, that will normally be
much less the best throughput of the device.
That's exactly why I am stating it didn't use WIBs 2 times of 3. It
went resyncing from beginning till the end -- I had my finger on its
pulse. :)
I can't promise (due to popular "lack of time" disease, yeah), but
if I'll get on it using VirtualBox or real environment [Lord forbid
:)] once again, I'll let you know, sure.
This doesn't appear to show the "start to end" that I suggested, just 10% to
30%, but it does suggest that the bitmap isn't speeding things up at all, at
least for that part of the array... That is assuming you are talking about
md124. I note that md125 doesn't have a bitmap. Is that what you are
referring to?
The next thing to do would be to look at the bitmap immediately after reboot
to see how many bits are set. Maybe lots are set for some reason.
— 3.4.2-rt10 went suspend ok for 2 times, the 3rd it didn't.
Reboot, resync, no bitmaps use.
And, BTW, it's rather slow in despite of having max. stripe_cache_size set.
A larger stripe cache isn't going to affect resync speed much (I think).
50MB/sec on each drive seems fairly good to me. What were you expecting?
NeilBrown
This doesn't appear to show the "start to end" that I suggested, just 10% to
30%, but it does suggest that the bitmap isn't speeding things up at all, at
least for that part of the array... That is assuming you are talking about
Well, I decided that's enough to be "pasted", but anyway, I can
assure you — it took ~ 1 hour, as it was estimated, so yeah — no use
of bitmap. For sure.
md124. I note that md125 doesn't have a bitmap. Is that what you are
referring to?
Yeah, md125 is legacy RAID which is not being in use for a long
time, it's just hanging around till re-layout happens.
The next thing to do would be to look at the bitmap immediately after reboot
to see how many bits are set. Maybe lots are set for some reason.
quoted
— 3.4.2-rt10 went suspend ok for 2 times, the 3rd it didn't.
Reboot, resync, no bitmaps use.
And, BTW, it's rather slow in despite of having max. stripe_cache_size set.
A larger stripe cache isn't going to affect resync speed much (I think).
50MB/sec on each drive seems fairly good to me. What were you expecting?
I see, thanks.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
This doesn't appear to show the "start to end" that I suggested, just 10% to
30%, but it does suggest that the bitmap isn't speeding things up at all, at
least for that part of the array... That is assuming you are talking about
Well, I decided that's enough to be "pasted", but anyway, I can
assure you — it took ~ 1 hour, as it was estimated, so yeah — no use
of bitmap. For sure.
quoted
md124. I note that md125 doesn't have a bitmap. Is that what you are
referring to?
Yeah, md125 is legacy RAID which is not being in use for a long
time, it's just hanging around till re-layout happens.
quoted
The next thing to do would be to look at the bitmap immediately after reboot
to see how many bits are set. Maybe lots are set for some reason.
That would more than unlikely due to:
— it's prepended with several sync-sleeps
— there's no intensive I/O
— kernel should have flush all buffers on suspend as well
--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html