From: Georg C. F. Greve <hidden> Date: 2004-12-30 16:23:58
[ update ]
Okay, tried to find out what is causing the kernel to crash and so I
replaced the dm-crypt part by cryptoloop: same effect.
Then I tried ext3 on top of LVM2 RAID5 with no encryption and it still
crashes. Not sure what is causing the problem exactly, but it does not
seem that dm-crypt is to blame anymore.
The message I saw on the remote console when it crashed with pure ext3
on raid5 was:
Assertion failure in journal_start() at fs/jbd/transaction.c:271: "handle->h_transaction->t_journal == journal"
Hope this helps -- filed the bug as #3968 on buzilla.kernel.org, more
info at
http://bugzilla.kernel.org/show_bug.cgi?id=3968
Help appreciated, let me know if you have an idea.
Regards,
Georg
--
Georg C. F. Greve [off-list ref]
Free Software Foundation Europe (http://fsfeurope.org)
Brave GNU World (http://brave-gnu-world.org)
From: Peter T. Breuer <hidden> Date: 2004-12-30 17:39:43
In gmane.linux.raid Georg C. F. Greve [off-list ref] wrote:
The message I saw on the remote console when it crashed with pure ext3
on raid5 was:
Assertion failure in journal_start() at fs/jbd/transaction.c:271: "handle->h_transaction->t_journal == journal"
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
Peter
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
???, do you mean it? which filesystem would you use for a 2TB RAID5 array? I
always used reiserfs for raid1/raid5 arrays...
sandro
*:-)
--
Sandro Dentella *:-)
e-mail: sandro@e-den.it
http://www.tksql.org TkSQL Home page - My GPL work
From: Peter T. Breuer <hidden> Date: 2004-12-30 18:31:40
In gmane.linux.raid Sandro Dentella [off-list ref] wrote:
quoted
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
???, do you mean it? which filesystem would you use for a 2TB RAID5 array? I
Whatever one you are using now. Just turn off journalling, or at least move
the journal somewhere else (and safe!).
Or perhaps use metadata-only journalling (but reiser does that by
default, does it not?). That should keep you happy.
From: Michael Tokarev <hidden> Date: 2004-12-30 19:51:03
Peter T. Breuer wrote:
In gmane.linux.raid Georg C. F. Greve [off-list ref] wrote:
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
This is a sort of a nonsense, really. Both claims, it seems.
I can't say for sure whenever write ordering is preserved by
raid -- it should, and if it isn't, it's a bug and should be
fixed. Nothing else is wrong with placing journal into raid
(the same as the filesystem in question). Suggesting to remove
journal just isn't fair: the journal is here for a reason.
And, finally, the kernel should not crash. If something like
this is unsupported, it should refuse to do so, instead of
crashing randomly.
/mjt
From: berk walker <hidden> Date: 2004-12-30 20:56:45
Michael Tokarev wrote:
Peter T. Breuer wrote:
quoted
In gmane.linux.raid Georg C. F. Greve [off-list ref] wrote:
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
This is a sort of a nonsense, really. Both claims, it seems.
I can't say for sure whenever write ordering is preserved by
raid -- it should, and if it isn't, it's a bug and should be
fixed. Nothing else is wrong with placing journal into raid
(the same as the filesystem in question). Suggesting to remove
journal just isn't fair: the journal is here for a reason.
And, finally, the kernel should not crash. If something like
this is unsupported, it should refuse to do so, instead of
crashing randomly.
/mjt
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I might have missed some of this thread....
but have you tried this on a completely different box? I have seen, and
am fighting some problems such as yours, and having nothing to do with raid.
If you haven't, then try it. You might get different results. Hardware
can sometimes be a dog to chase down, problemwise.
b-
From: Peter T. Breuer <hidden> Date: 2004-12-30 21:39:42
In gmane.linux.raid Michael Tokarev [off-list ref] wrote:
Peter T. Breuer wrote:
quoted
In gmane.linux.raid Georg C. F. Greve [off-list ref] wrote:
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
This is a sort of a nonsense, really. Both claims, it seems.
It's perfectly correct, as far as I know!
I can't say for sure whenever write ordering is preserved by
raid --
There is nothing that attempts expliciitly to maintain the ordering in
RAID (talking about mirroring here). Mirror requests are submitted
_asynchronously_ to the block subsystem for each device in the mirror,
for each incoming request. The kernel doesn't even have any way of
tracking in what order requests are emitted (it would need a counter
field in each request and there is not one), let alone in what order
they are emitted per device, under the device it is aiming at.
And then of course there is no way at all of telling the underlying
devices in what order to treat the requests either - and what about
request aggregation? Requests are normally aggregated by the kernel
before being sent to devices - ok, I think I recall that RAID turns
that off on itself by using its own make_request function, but it
doesn't control request aggregation in the sub-devices.
And I don't know what happens if you throw the extra resync thread into
the mix, but there certainly IS a RAID kernel thread that does nothing
else than retry failed requests (and do resyncs?) - which of course will
be out of order if ever they are successfully completed by the thread.
If we move on to RAID5, then the situation is simply even more
complicated because we no longer have to think about when solid,
physical, mirrored data is written, but when "virtual" redundant data is
written (and read).
I'm not even sure what in the kernel in general can possibly guarantee
that the sequence write-read-read-write-read can remain ordered that way
when an unplug event interrupts the sequence.
it should, and if it isn't, it's a bug and should be
fixed. Nothing else is wrong with placing journal into raid
It's been that way forever.
(the same as the filesystem in question).
What's wrong is that the journal will be mirrored (if it's a mirror).
That means that (1) its data will be written twice, which is a big deal
since ALL the i/o goes through the journal first, and (2) the journal
is likely to be inconsistent (since it is so active) if you get one of
those creeping invisible RAID corruptions that can crop up inevitably
in RAID normal use.
Suggesting to remove
journal just isn't fair: the journal is here for a reason.
Well, I'd remove it: presumably his aim is to reduce fsck times after a
crash. But consider - if he has had a crash, it is likely that his data is
corrupted, so he WANTS to check.
All that a journal does is guarantee consistency of a FS, not
correctness. Personally, I prefer to see the incorrectness. If you
don't want to check the filesystem you can always just choose to not run
fsck!
And in this case the journal is a significant extra risk factor,
because it is ON the falied medium, and on the part that is most
active, moreover!
All you have to do to make things safer is take the journal OFF the
raid array. You immediately remove the potential for corruption IN the
journal (I believe that's what he has seen anyway - damage to the disk
under the journal), which is where we have deduced by the above argument
that the major source of likely corruptions must lie.
There's also no good sense in data-journalling, but I don't think
reiserfs does that anyway (it didn't use to, I know - ext3 was the
first to do data journalling, although even that's a misnomer, since
you try writing a 4GB file as an atomic operation ...).
Journals do no magic. You have to consider if they introduce more
benefits than dangers.
And, finally, the kernel should not crash.
Well, I'm afraid that like everyone else it is dependent on hardware
and authors, both of which are fallible!
If something like
this is unsupported, it should refuse to do so, instead of
crashing randomly.
On Thu, Dec 30, 2004 at 10:50:55PM +0300, Michael Tokarev wrote:
Peter T. Breuer wrote:
quoted
In gmane.linux.raid Georg C. F. Greve [off-list ref] wrote:
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
This is a sort of a nonsense, really. Both claims, it seems.
I can't say for sure whenever write ordering is preserved by
raid -- it should, and if it isn't, it's a bug and should be
fixed. Nothing else is wrong with placing journal into raid
(the same as the filesystem in question). Suggesting to remove
journal just isn't fair: the journal is here for a reason.
And, finally, the kernel should not crash. If something like
this is unsupported, it should refuse to do so, instead of
crashing randomly.
Write ordering trouble shouldn't crash the kernel, the way I
understand it. Your journalled fs could be lost/inconsistent
if the machine crashes for other reasons, due to bad write
ordering. But the ordering trouble shouldn't cause a crash,
and all should be fine as soon as all the writes complete
without other incidents.
Helge Hafting
From: Andy Smith <hidden> Date: 2005-01-02 19:42:40
On Thu, Dec 30, 2004 at 10:39:42PM +0100, Peter T. Breuer wrote:
In gmane.linux.raid Michael Tokarev [off-list ref] wrote:
quoted
Peter T. Breuer wrote:
quoted
In gmane.linux.raid Georg C. F. Greve [off-list ref] wrote:
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
This is a sort of a nonsense, really. Both claims, it seems.
From: Peter T. Breuer <hidden> Date: 2005-01-02 20:18:13
Andy Smith [off-list ref] wrote:
[-- text/plain, encoding quoted-printable, charset: us-ascii, 22 lines --]
On Thu, Dec 30, 2004 at 10:39:42PM +0100, Peter T. Breuer wrote:
quoted
In gmane.linux.raid Michael Tokarev [off-list ref] wrote:
quoted
Peter T. Breuer wrote:
quoted
In gmane.linux.raid Georg C. F. Greve [off-list ref] wrote:
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
This is a sort of a nonsense, really. Both claims, it seems.
Interesting - I'll post it (there is no flame war):
* From: "Stephen C. Tweedie" <sct redhat com>
* To: Andy Smith <andy lug org uk>
* Cc: Stephen Tweedie <sct redhat com>, ext3 users list <ext3-users
* redhat com>
* Subject: Re: ext3 journal on software raid
* Date: Sat, 01 Jan 2005 22:19:23 +0000
(snip)
Disks and IO subsystems in general don't preserve IO ordering.
This is true.
ext3 is
designed not to care.
That is surprising - write-order preservation is precisely the
condition that reiserfs requires for correct journal behaviour, and
Hans Reiser tld be so himself (sometime, some reiserfs mailing list, at
the time, etc).
It would be surprising if Stephen managed to do without it, but his
condition is definitely weaker. He merely requires to be _told_
(synchronously) when each i/o has ended, in the order that it ends, I
think is what he says below.
I'm not sure that raid can guarrantee precisely that either. There
might be minor disorderings on a SMP system with preemption, if for
example, one request is handled on one cpu and another on another, and
the acks are handled crosswise. There might be a small temporal
displacement. I haven't thought about it.
What I can say is that it makes no _attempt_ to respect that condition.
Whether it does or not I cannot exactly say.
As long as the raid device tells the truth about
when the data is actually committed to disk (all of the mirror volumes
are uptodate) for a given IO, ext3 should be quite happy.
Uuff .. as I said, it is not quite clear to me that this (very weak)
condition is absolutely respected. Umm ... no, endio for the whole
request is sent back AFTER the mirror i/os have completed, but exactly
WHEN after is indeterminate on a preemptive (SMP) system. The mirrors
might have been factually updated for two requests in temporal order A
B, but might report endio in order B A. However, I think that he
probably is calling A then B in a single thread, which means that
B won't even be generated until A is acked.
OK - I think Stephen is probably saying that the ack must be sent back
AFTER the status of the writes on the mirror disks is known.
Yes, that is guarranteed (unless you apply an async raid patch ...).
quoted
What's wrong is that the journal will be mirrored (if it's a mirror).
That means that (1) its data will be written twice, which is a big deal
since ALL the i/o goes through the journal first
Not true; by default, only metadata goes through the journal, not data.
He is saying that data is not journalled by default on ext3. I don't
see that as a comment about raid, and inasmuch as it means anything it
means that his comment "not true" is about as close to a rather strange
(political?) untruth as you can get in CS, since all the journal's data
WILL be written twice - it's up to you how much that is. Whether you
pass the data through the journal or not, all the data you choose to
pass will be written twice, be it zero, some, or all.
quoted
and (2) the journal
is likely to be inconsistent (since it is so active) if you get one of
those creeping invisible RAID corruptions that can crop up inevitably
in RAID normal use.
Umm, if soft raid is expected to have silent invisible corruptions in
normal use,
It is, just as is all types of RAID. This is a very strange thing for
Stephen to say - I cannot believe that he is as naive as he makes
himself out to be about RAID here and I don't know why he should say
that (presuming that he really knows better).
then you shouldn't be using it, period. That's got zero to
do with journaling.
It implies that one should not be doing journalling on top of it.
(The logic for why RAID corrupts silently is that errors accumulate at
n times the normal rate per sector, but none of them are detected by
RAID (no crc), and when a disk drops out then you get a good chance of
picking up a corrupted copy instead of a good copy, because nobody
has checked the copy meanwhiles to see if it matches the original).
Peter
From: Andy Smith <hidden> Date: 2005-01-03 00:30:06
On Sun, Jan 02, 2005 at 09:18:13PM +0100, Peter T. Breuer wrote:
Stephen Tweedie wrote:
quoted
Umm, if soft raid is expected to have silent invisible corruptions in
normal use,
It is, just as is all types of RAID. This is a very strange thing for
Stephen to say - I cannot believe that he is as naive as he makes
himself out to be about RAID here and I don't know why he should say
that (presuming that he really knows better).
quoted
then you shouldn't be using it, period. That's got zero to
do with journaling.
It implies that one should not be doing journalling on top of it.
(The logic for why RAID corrupts silently is that errors accumulate at
n times the normal rate per sector, but none of them are detected by
RAID (no crc), and when a disk drops out then you get a good chance of
picking up a corrupted copy instead of a good copy, because nobody
has checked the copy meanwhiles to see if it matches the original).
I have no idea which of you to believe now. :(
I currently only have one system using software raid, and several of
my employer's machines using hardware raid, all of which have
various raid-1, -5 and -10 setups and all use only ext3.
Let's focus on the personal machine of mine for now since it uses
Linux software RAID and therefore on-topic here. It has /boot on a
small RAID-1, and the rest of the system is on RAID-5 with an
additional RAID-0 just for temporary things.
There is nowhere that is not software RAID to put the journals, so
would you be recommending that I turn off journalling and basically
use it as ext2?
What I do know is that none of what you say is in the software raid
howto, and if you are right, it really should be. Neither is it in
any ext3 documentation and there is no warning on any distribution
installer I have ever used (those that understand RAID and LVM and
are happy to set that up at install time with ext3). Also everyone
that I have spoken to about this knows nothing about it, so what you
are saying, if correct, would seem to have far-reaching
implications.
From: Peter T. Breuer <hidden> Date: 2005-01-03 08:03:14
Andy Smith [off-list ref] wrote:
[-- text/plain, encoding quoted-printable, charset: us-ascii, 45 lines --]
On Sun, Jan 02, 2005 at 09:18:13PM +0100, Peter T. Breuer wrote:
quoted
Stephen Tweedie wrote:
quoted
Umm, if soft raid is expected to have silent invisible corruptions in
normal use,
It is, just as is all types of RAID. This is a very strange thing for
Stephen to say - I cannot believe that he is as naive as he makes
himself out to be about RAID here and I don't know why he should say
that (presuming that he really knows better).
quoted
then you shouldn't be using it, period. That's got zero to
do with journaling.
It implies that one should not be doing journalling on top of it.
(The logic for why RAID corrupts silently is that errors accumulate at
n times the normal rate per sector, but none of them are detected by
RAID (no crc), and when a disk drops out then you get a good chance of
picking up a corrupted copy instead of a good copy, because nobody
has checked the copy meanwhiles to see if it matches the original).
I have no idea which of you to believe now. :(
Both of us. We have not disagreed fundamentally. Read closely! Stephen
says "IF (my caps) soft raid is expected to have ...". Well, it is,
just like any RAID.
Similarly he didn't disagree that journal data is written twice, if you
read closely, he merely pointed out that the DEFAULT (my caps) setting
in ext3 is not to write data (as opposed to metadata) into the journal
at all.
So he avoided issues of substance there (and/but gave a strange spin to
them).
What he did claim that is factually interesting and new is that ext3
works if acks from the media are merely received after the fact. That's
a far weaker requirement than for reiserfs, for example. It seems to me
to imply that the implementation is single-threaded and highly
synchronous.
I currently only have one system using software raid, and several of
my employer's machines using hardware raid, all of which have
various raid-1, -5 and -10 setups and all use only ext3.
All fine - as I said, the only thing I'd do is make sure that the
journal is not kept on the raid partition(s), and possibly turn off
data journalling in favour of metadata journalling only.
Let's focus on the personal machine of mine for now since it uses
Linux software RAID and therefore on-topic here. It has /boot on a
small RAID-1,
This is always a VERY bad idea. /boot and /root want to be on as simple
and uncomplicated a system as possible. Moreover, they never change, so
what is the point of having a real time mirror for them? It would be
sufficient to copy them every day (which is what I do) at file system
level to another partition, if you want a spare copy for emergencies.
and the rest of the system is on RAID-5 with an
additional RAID-0 just for temporary things.
That's fine.
There is nowhere that is not software RAID to put the journals, so
Well, you can make somewhere. You only require an 8MB (one cylinder)
partition.
would you be recommending that I turn off journalling and basically
use it as ext2?
No, I'd be recommending that you make an 8MB partition for a journal.
This is also handy in case you "wear through" the disk under the
journal because of the high i/o there. Well, it's happened to me on two
disks, but doubtless people will cntest that! IF it happens, all you
have to do is use a cylinder somewhere else.
What I do know is that none of what you say is in the software raid
howto,
But nothing said is other than obvious, and is a matter of probabilities
and risk management, so I don't see why it should be in a howto! That's
your business, not the howto's.
and if you are right, it really should be. Neither is it in
I don't think it should be. It should be somewhere in ext3 docs (there
was a time when ext3 wouldn't work on raid1 at all, butthat got fixed
somehow), but then documenting how your FS works on some particular
media is not really part of the documentation scope for the FS!
any ext3 documentation and there is no warning on any distribution
installer I have ever used (those that understand RAID and LVM and
are happy to set that up at install time with ext3). Also everyone
that I have spoken to about this knows nothing about it, so what you
Everyone knows about it, because none of us is saying anything that is
not obvious. Yes, data is written through the journal twice. EVERYTHING
is written through the journal twice if the journal is on RAID1,
because everything on RAID1 is written twice. That is obvious, no?
And then you get i/o storms through the journal in any case on
journalled raid, whenever you do data journalling. It is just doubled if
you do that on a raid system.
And there is a risk of silent corruption on all raid systems - that is
well known. DIfferent raid systems do different thigs to compensate,
such as periodically recalculating the parity on everything. But when
you have redundant data and a corruption occurs, which of the two
datasets do you believe? You have to choose one of them! You guess
wrong half the time, if you guess ("you" is a raid system). Hence
"silent corruption".
are saying, if correct, would seem to have far-reaching
implications.
I don't think so! Why? RAID protects you against certain sorts of risk.
It also exposes you to other sorts of risk. Where is the far-reaching
implication in that? It is for you to balance the risks and benefits.
Peter
"Well, you can make somewhere. You only require an 8MB (one cylinder)
partition."
So, it is ok for your system to fail when this disk fails?
I don't want system failures when a disk fails, so mirror (or RAID5)
everything required to keep your system running.
"And there is a risk of silent corruption on all raid systems - that is
well known."
I question this....
I bet a non-mirror disk has similar risk as a RAID1. But with a RAID1, you
know when a difference occurs, if you want.
Guy
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Peter T. Breuer
Sent: Monday, January 03, 2005 3:03 AM
To: linux-raid@vger.kernel.org
Subject: Re: ext3 journal on software raid (was Re: PROBLEM: Kernel 2.6.10
crashing repeatedly and hard)
Andy Smith [off-list ref] wrote:
[-- text/plain, encoding quoted-printable, charset: us-ascii, 45 lines --]
On Sun, Jan 02, 2005 at 09:18:13PM +0100, Peter T. Breuer wrote:
quoted
Stephen Tweedie wrote:
quoted
Umm, if soft raid is expected to have silent invisible corruptions in
normal use,
It is, just as is all types of RAID. This is a very strange thing for
Stephen to say - I cannot believe that he is as naive as he makes
himself out to be about RAID here and I don't know why he should say
that (presuming that he really knows better).
quoted
then you shouldn't be using it, period. That's got zero to
do with journaling.
It implies that one should not be doing journalling on top of it.
(The logic for why RAID corrupts silently is that errors accumulate at
n times the normal rate per sector, but none of them are detected by
RAID (no crc), and when a disk drops out then you get a good chance of
picking up a corrupted copy instead of a good copy, because nobody
has checked the copy meanwhiles to see if it matches the original).
I have no idea which of you to believe now. :(
Both of us. We have not disagreed fundamentally. Read closely! Stephen
says "IF (my caps) soft raid is expected to have ...". Well, it is,
just like any RAID.
Similarly he didn't disagree that journal data is written twice, if you
read closely, he merely pointed out that the DEFAULT (my caps) setting
in ext3 is not to write data (as opposed to metadata) into the journal
at all.
So he avoided issues of substance there (and/but gave a strange spin to
them).
What he did claim that is factually interesting and new is that ext3
works if acks from the media are merely received after the fact. That's
a far weaker requirement than for reiserfs, for example. It seems to me
to imply that the implementation is single-threaded and highly
synchronous.
I currently only have one system using software raid, and several of
my employer's machines using hardware raid, all of which have
various raid-1, -5 and -10 setups and all use only ext3.
All fine - as I said, the only thing I'd do is make sure that the
journal is not kept on the raid partition(s), and possibly turn off
data journalling in favour of metadata journalling only.
Let's focus on the personal machine of mine for now since it uses
Linux software RAID and therefore on-topic here. It has /boot on a
small RAID-1,
This is always a VERY bad idea. /boot and /root want to be on as simple
and uncomplicated a system as possible. Moreover, they never change, so
what is the point of having a real time mirror for them? It would be
sufficient to copy them every day (which is what I do) at file system
level to another partition, if you want a spare copy for emergencies.
and the rest of the system is on RAID-5 with an
additional RAID-0 just for temporary things.
That's fine.
There is nowhere that is not software RAID to put the journals, so
Well, you can make somewhere. You only require an 8MB (one cylinder)
partition.
would you be recommending that I turn off journalling and basically
use it as ext2?
No, I'd be recommending that you make an 8MB partition for a journal.
This is also handy in case you "wear through" the disk under the
journal because of the high i/o there. Well, it's happened to me on two
disks, but doubtless people will cntest that! IF it happens, all you
have to do is use a cylinder somewhere else.
What I do know is that none of what you say is in the software raid
howto,
But nothing said is other than obvious, and is a matter of probabilities
and risk management, so I don't see why it should be in a howto! That's
your business, not the howto's.
and if you are right, it really should be. Neither is it in
I don't think it should be. It should be somewhere in ext3 docs (there
was a time when ext3 wouldn't work on raid1 at all, butthat got fixed
somehow), but then documenting how your FS works on some particular
media is not really part of the documentation scope for the FS!
any ext3 documentation and there is no warning on any distribution
installer I have ever used (those that understand RAID and LVM and
are happy to set that up at install time with ext3). Also everyone
that I have spoken to about this knows nothing about it, so what you
Everyone knows about it, because none of us is saying anything that is
not obvious. Yes, data is written through the journal twice. EVERYTHING
is written through the journal twice if the journal is on RAID1,
because everything on RAID1 is written twice. That is obvious, no?
And then you get i/o storms through the journal in any case on
journalled raid, whenever you do data journalling. It is just doubled if
you do that on a raid system.
And there is a risk of silent corruption on all raid systems - that is
well known. DIfferent raid systems do different thigs to compensate,
such as periodically recalculating the parity on everything. But when
you have redundant data and a corruption occurs, which of the two
datasets do you believe? You have to choose one of them! You guess
wrong half the time, if you guess ("you" is a raid system). Hence
"silent corruption".
are saying, if correct, would seem to have far-reaching
implications.
I don't think so! Why? RAID protects you against certain sorts of risk.
It also exposes you to other sorts of risk. Where is the far-reaching
implication in that? It is for you to balance the risks and benefits.
Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
And there is a risk of silent corruption on all raid systems - that is
well known. DIfferent raid systems do different thigs to compensate,
such as periodically recalculating the parity on everything. But when
you have redundant data and a corruption occurs, which of the two
datasets do you believe? You have to choose one of them! You guess
wrong half the time, if you guess ("you" is a raid system). Hence
"silent corruption".
Just on this point. With RAID-6 I guess you can get a majority rules type of ruling on which data
block is the dud. Unless you come up with 3 different results in which case something is really not
right in lego land.
I wonder perhaps about a userspace app that you can run to check all the parity blocks. On RAID-5 it
should be able to tell you, you have a naff stripe, but on RAID-6 in theory if it's only a single
drive that is a problem you should be able to correct the block.
Regards,
Brad
From: Michael Tokarev <hidden> Date: 2005-01-03 12:11:03
Peter T. Breuer wrote:
[]
quoted
Let's focus on the personal machine of mine for now since it uses
Linux software RAID and therefore on-topic here. It has /boot on a
small RAID-1,
This is always a VERY bad idea. /boot and /root want to be on as simple
and uncomplicated a system as possible. Moreover, they never change, so
what is the point of having a real time mirror for them? It would be
sufficient to copy them every day (which is what I do) at file system
level to another partition, if you want a spare copy for emergencies.
Raid1 (mirror) is the most "trivial" raid level out there, especially
having in mind that the underlying devices -- all of them -- contains
(or should, in theory -- modulo the "50% chance of any difference
being unnoticied" etc) exact copy of the filesystem. Also, root (and
/boot -- i for one have both /boot in root in a single small filesystem)
do change -- not that often but often enouth so that "newaliases problem"
(when you "forgot" to backup it after a change) happens from time to time.
After several years of expirience with alot of systems (and alot of various
disk failure scenarios too: when you have many systems, you have good
chances to see a failure ;), I now use very simple and (so far) reliable
approach, which I explained here on this list before. You have several
(we use 2, 3 or 4) disks which are the same (or almost: eg some 36Gb
disks are really 35Gb or 37Gb; in case they're differ, "extra" space
on large disk isn't used); root and /boot are on small raid1 partition
which is mirrored on *every* disk; swap is on raid1; the rest (/usr,
/home, /var etc) are on raid5 arrays (maybe also raid0 for some "scratch"
space). This way, you have "equal" drives, and *any* drive, including
boot one, may fail at any time and the system will continue working
as if all where working, including reboot (except of a (very rare in
fact) failure scenario when your boot disk has failed MBR or other
sectors required to boot, but "the rest" of that disk is working,
in which case you'll need physical presence to bring the machine up).
All the drives are "symmetrical", usage patterns for all drives are
the same, and due to usage of raid arrays, load is spread among them
quite nicely. You're free to reorder the drives in any way you want,
to replace any of them (maybe rearranging the rest if you're
replacing the boot drive) and so on.
Yes, root fs does not changes often, and yes it is small enouth
(I use 1Gb, or 512Mb, or even 256Mb for root fs - not a big deal
to allocate that space on every of 2 or 3 or 4 or 5 disks). So
it isn't quite relevant how fast the filesystem will be on writes,
and hence it's ok to place it on raid1 composed from 5 components.
The stuff just works, it is very simple to administer/support,
and does all the "backups" automatically. In case of some problem
(yes I dislike any additional layers for critical system components
as any layer may fail to start during boot etc), you can easily
bring the system up by booting off the underlying root-raid partiton
to repair the system -- all the utilities are here. More, you can
boot from one disk (without raid) and try to repair root fs on
another drive (if things are really screwed up), and when you're
done, bring the raid up on that repaired partition and add other
drives to the array.
To summarize: having /boot and root on raid1 is a very *good* idea. ;)
It saved our data alot of times in the past few years already.
If you're worried about "silent data corruption" due to different
data being read from different components of the raid array.. Well,
first of all, we never saw that yet (we have quite good "testcase")
(and no, I'm not saying it's impossible ofcourse). On rarely-changed
filesystem, with real drives which does no silent remapping of an
undeadable blocks to new place with the data on them becoming all-0s,
without drives with uncontrollable write caching (quite common for
IDE drives) and things like that, and with real memory (ECC I mean),
where you *know* what you're writing to each disk (yes, there's also
another possible cause of a problem: software errors aka bugs ;),
that case with different data on different drives becomes quite..
rare. In order to be really sure, one can mount -o remount,ro /
and just compare all components of the root raid, periodically.
When there's more than 2 components on that array, it should be
easy to determine which drive is "lying" in case of any difference.
I do similar procedure on my systems during boot.
quoted
There is nowhere that is not software RAID to put the journals, so
Well, you can make somewhere. You only require an 8MB (one cylinder)
partition.
Note scsi disks in linux only supports up to 14 partitions, which
isn't sometimes sufficient even without additional partitions for
journal. When you have large amount of disks (so having that
"fully-symmetrical" layout as I described above becomes impractical),
you can use one set of drives for data and another set of drives
for journal for that data. When you only have 4 (or less) drives...
And yes I'm aware of mdp devices (partitions inside the raid
arrays).. but that's just another layer "which may fail": if
raid5 array won't start, I at least can reconstruct filesystem
image by reading chunks of data from appropriate places from
all drives and try to recover that image; with any additional
structure inside the array (and the lack of "loopP" aka partitioned
loop devices) it becomes more and more tricky to recover any
data (from this point of view, raid1 is the niciest raid level ;)
Again: instead of using a partition for the journal, use (another?)
raid array. This way, the system will work if the drive wich
contains the journal fails. Note above about swap: in all my
systems, swap is also on raid (raid1 in this case). At the first
look, that can be a nonsense: having swap on raid. But we had
enouth cases when due to a failed drive swap becomes corrupt
(unreadable really), and the system goes havoc, *damaging*
other data which was unaffected by the disk failure! With
swap on raid1, the system continues working if any drive
fails, which is good. (Older kernels, esp. 2.2.* series,
had several probs with swap on raid, but that has been fixed
now; there where other bugs fixed too (incl. bugs in ext3fs)
so there should be no such damage to other data due to
unreadable swap.. hopefully. But I can't trust my systems
anymore after seeing (2 times in 4 years) what can happen with
the data...)
[]
And I also want to "re-reply" to the first your message in this
thread, where I was saying that "it's a nonsense that raid does
not preserve write ordering". Ofcourse I mean not write ordering
but working write barriers (as Neil pointed out, md subsystem does
not implement write barriers directly but the concept is "emulated"
by linux block subsystem). Write barriers should be sufficient to
implement journalling safely.
/mjt
From: Peter T. Breuer <hidden> Date: 2005-01-03 14:23:38
Michael Tokarev [off-list ref] wrote:
Peter T. Breuer wrote:
quoted
This is always a VERY bad idea. /boot and /root want to be on as simple
and uncomplicated a system as possible. Moreover, they never change, so
what is the point of having a real time mirror for them? It would be
sufficient to copy them every day (which is what I do) at file system
level to another partition, if you want a spare copy for emergencies.
Raid1 (mirror) is the most "trivial" raid level out there, especially
Hi!
having in mind that the underlying devices -- all of them -- contains
(or should, in theory -- modulo the "50% chance of any difference
being unnoticied" etc) exact copy of the filesystem. Also, root (and
/boot -- i for one have both /boot in root in a single small filesystem)
do change -- not that often but often enouth so that "newaliases problem"
(when you "forgot" to backup it after a change) happens from time to time.
Well, my experience is that anything "unusual" is bad: sysadmins change
over the years; the guy who services the system may not be the one that
built it; the "rescue" cd or floppy he has may not have MD support
built into the kernel (and he probably will need a rescue cd just to get
support for a raid card, if the machine has hardware raid as well as or
instead of software raid).
Therefore, I have learned not to build a system that is more complicated
than the most simple human being that may administer it. This always
works - if it breaks AND they cannot fix it, then THEY get the blame.
So I "prefer" to not have a raided boot partition, but instead to rsync
the root partition every day to a spare on a differet disk, or/and at the
other end of the same disk. This also saves the system from sysadmin
gaffes - I don't WANT an instantaneous copy of every error made by the
humans.
This is not to say that I do not like your ideas, expressed here. I do.
I even agree with them.
It is just that when they mess up the root partition, I can point to
the bootloader entry that says "boot from spare root partition".
And let's not get into what they can do to the labelling on the
partition types - FD? Must be a mistake!
After several years of expirience with alot of systems (and alot of various
disk failure scenarios too: when you have many systems, you have good
chances to see a failure ;), I now use very simple and (so far) reliable
approach, which I explained here on this list before. You have several
(we use 2, 3 or 4) disks which are the same (or almost: eg some 36Gb
Well, whenever I buy anything, I buy two. I buy two _controller_ cards,
and tape the extra one inside the case. But of course I buy two
machines, so that is four cards ... .
And I betcha softraid sb has changed format ver the years. I am still
running P100s!
disks are really 35Gb or 37Gb; in case they're differ, "extra" space
on large disk isn't used); root and /boot are on small raid1 partition
which is mirrored on *every* disk; swap is on raid1; the rest (/usr,
I like this - except of course that I rsync them, not raid them. I
don't mind if I have to reboot a server. Nobody will notice the tcp
outage and the other one of the pair will failover for it, albeit in
readonly mode, for the maximum of the few minutes required.
Your swap idea is crazy, but crazy enough to be useful. YES, there used
to be a swap bug which corrupted swap every so often (in 2.0? 2.2?) and
meant one had to swapoff and swapon again, having first cleared all
processes by an init 1 and back. Obviously that bug would bite whatever
you had as media, but it still is a nice idea to have raided memory
:-).
/home, /var etc) are on raid5 arrays (maybe also raid0 for some "scratch"
I don't put /var on raid if I can help it. But there is nothing
particularly bad about it. It is just that /var is the most active
place and therefore the most likely to suffer damage of some kind, somehow.
And damaged raided partitions are really not nice. Raid does not
protect you against hardware corruption - on the contrary, it makes it
more difficult to spot and doubles the probabilities of it happening.
space). This way, you have "equal" drives, and *any* drive, including
boot one, may fail at any time and the system will continue working
as if all where working, including reboot (except of a (very rare in
fact) failure scenario when your boot disk has failed MBR or other
sectors required to boot, but "the rest" of that disk is working,
in which case you'll need physical presence to bring the machine up).
That's actually not so. Over new year I accidently booted my home
server (222 days uptime!) and discovered its boot sector had evaporated.
Well, maybe I moved the kernels .. anyway, it has no floppy and the
nearest boot cd was an hour's journey away in the cold, on new year. Uh
uh. It took me about 8 hrs, but I booted it via PXE DHCP TFTP
wake-on-lan and the wireless network, from my laptop, without leaving
the warm.
Next time I may even know how to do it beforehand :).
All the drives are "symmetrical", usage patterns for all drives are
the same, and due to usage of raid arrays, load is spread among them
quite nicely. You're free to reorder the drives in any way you want,
to replace any of them (maybe rearranging the rest if you're
replacing the boot drive) and so on.
You can do this hot? How? Oh, you must mean at reboot.
Yes, root fs does not changes often, and yes it is small enouth
(I use 1Gb, or 512Mb, or even 256Mb for root fs - not a big deal
Mine are always under 256MB, but I give 512MB.
to allocate that space on every of 2 or 3 or 4 or 5 disks). So
it isn't quite relevant how fast the filesystem will be on writes,
and hence it's ok to place it on raid1 composed from 5 components.
That is, uh, paranoid.
The stuff just works, it is very simple to administer/support,
and does all the "backups" automatically.
Except that it doesn't - backups are not raid images. Backups are
snapshots. Maybe you mean that.
In case of some problem
(yes I dislike any additional layers for critical system components
as any layer may fail to start during boot etc), you can easily
bring the system up by booting off the underlying root-raid partiton
to repair the system -- all the utilities are here. More, you can
Well, you could, and I could, but I doubt if the standard tech could.
boot from one disk (without raid) and try to repair root fs on
another drive (if things are really screwed up), and when you're
done, bring the raid up on that repaired partition and add other
drives to the array.
But why bother? If you didn't have raid there on root you wouldn't
need to repair it. Nothing is quite as horrible as having a
fubarred root partition. That's why I also always have two! But I
don't see that having the copy made by raid rather than rsync wins
you anything in the situaton where you have to reboot - rather, it
puts off that moment to a moment of your choosing, which may be good,
but is not an unqualified bonus, given the cons.
To summarize: having /boot and root on raid1 is a very *good* idea. ;)
It saved our data alot of times in the past few years already.
No - it saved you from taking the system down at that moment in time.
You could always have rebooted it from a spare root partition whether
you had raid there or not.
If you're worried about "silent data corruption" due to different
data being read from different components of the raid array.. Well,
first of all, we never saw that yet (we have quite good "testcase")
It's hard to see, and youhave to crash and come back up quite a lot to
make it probable. A funky scsi cable would help you see it!
(and no, I'm not saying it's impossible ofcourse). On rarely-changed
filesystem, with real drives which does no silent remapping of an
undeadable blocks to new place with the data on them becoming all-0s,
Yes, I agree. On rarely changing systems raid is a benefit, because
it enables you to carry on in case the unthinkable happens and one disk
vaporizes (while letting the rest of the system carry on, with much luck).
On rapidly changing systems like /var I start to get a little uneasy.
On /home I am quite happy with it. I wouldn't have it any other way
there..
without drives with uncontrollable write caching (quite common for
IDE drives) and things like that, and with real memory (ECC I mean),
where you *know* what you're writing to each disk (yes, there's also
another possible cause of a problem: software errors aka bugs ;),
Indeed, and very frequent they are too.
that case with different data on different drives becomes quite..
rare. In order to be really sure, one can mount -o remount,ro /
and just compare all components of the root raid, periodically.
When there's more than 2 components on that array, it should be
easy to determine which drive is "lying" in case of any difference.
I do similar procedure on my systems during boot.
Well, voting is one possible procedure. I don't know if softraid does
that anywhere, or attempts repairs.
Neil?
quoted
quoted
There is nowhere that is not software RAID to put the journals, so
Well, you can make somewhere. You only require an 8MB (one cylinder)
partition.
Note scsi disks in linux only supports up to 14 partitions, which
You can use lvm (device mapper). Admittedly I was thinking of IDE.
If you like I can patch scsi for 63 partitions?
isn't sometimes sufficient even without additional partitions for
journal. When you have large amount of disks (so having that
"fully-symmetrical" layout as I described above becomes impractical),
you can use one set of drives for data and another set of drives
for journal for that data. When you only have 4 (or less) drives...
And yes I'm aware of mdp devices (partitions inside the raid
arrays).. but that's just another layer "which may fail": if
raid5 array won't start, I at least can reconstruct filesystem
image by reading chunks of data from appropriate places from
all drives and try to recover that image; with any additional
Now that is just perverse.
structure inside the array (and the lack of "loopP" aka partitioned
loop devices) it becomes more and more tricky to recover any
data (from this point of view, raid1 is the niciest raid level ;)
Agree.
Again: instead of using a partition for the journal, use (another?)
raid array. This way, the system will work if the drive wich
contains the journal fails.
But the journal will also contain corruptions if the whole system
crashes, and is rebooted. You just spent several paragraphs (?) arguing
so. Do you really want those rolled forward to complete? I would
rather they were rolled back! I.e. that the journal were not there -
I am in favour of a zero size journal, in other words, which only acts
to guarantee atomicity of FS ops (FS code on its own may do that), but
which does not contain data.
Note above about swap: in all my
systems, swap is also on raid (raid1 in this case). At the first
look, that can be a nonsense: having swap on raid. But we had
enouth cases when due to a failed drive swap becomes corrupt
(unreadable really), and the system goes havoc, *damaging*
other data which was unaffected by the disk failure! With
Yes, this used to be quite common when swap had that size bug.
swap on raid1, the system continues working if any drive
fails, which is good. (Older kernels, esp. 2.2.* series,
had several probs with swap on raid, but that has been fixed
now; there where other bugs fixed too (incl. bugs in ext3fs)
so there should be no such damage to other data due to
unreadable swap.. hopefully. But I can't trust my systems
anymore after seeing (2 times in 4 years) what can happen with
the data...)
[]
And I also want to "re-reply" to the first your message in this
thread, where I was saying that "it's a nonsense that raid does
not preserve write ordering". Ofcourse I mean not write ordering
but working write barriers (as Neil pointed out, md subsystem does
not implement write barriers directly but the concept is "emulated"
by linux block subsystem). Write barriers should be sufficient to
implement journalling safely.
I am not confident that Neil did say so. I have not reexamined his
post, but I got the impression that he hummed and hawed over that.
I do not recall that he said that raid implements write barriers -
perhaps he did. Anyway, I do not recall any code to handle "special"
requests, which USED to be the kernel's barrier mechanism. Has that
mechanism changed (it could have!)?
What is the write barrier mechanism in the 2.6 series (and what was it
in 2.4? I don't recall one at all)?
I seem to recall that Neil said instead that raid acks writes only after
they have been carried out on all components, which Stephen said was
sufficient for ext3. OTOH we do not know if it is sufficient for
reiserfs, xfs, jfs, etc.
Peter
On Monday 03 January 2005 15:23, Peter T. Breuer wrote:
Michael Tokarev [off-list ref] wrote:
quoted
Peter T. Breuer wrote:
Therefore, I have learned not to build a system that is more complicated
than the most simple human being that may administer it. This always
works - if it breaks AND they cannot fix it, then THEY get the blame.
So I "prefer" to not have a raided boot partition, but instead to rsync
the root partition every day to a spare on a differet disk, or/and at the
other end of the same disk. This also saves the system from sysadmin
gaffes - I don't WANT an instantaneous copy of every error made by the
humans.
There certainly is something to be said for that...
However, I do expect an admin to know about the raid system that's being used,
else they would have no business being near that server in the first place.
quoted
disks are really 35Gb or 37Gb; in case they're differ, "extra" space
on large disk isn't used); root and /boot are on small raid1 partition
which is mirrored on *every* disk; swap is on raid1; the rest (/usr,
I like this - except of course that I rsync them, not raid them. I
don't mind if I have to reboot a server. Nobody will notice the tcp
outage and the other one of the pair will failover for it, albeit in
readonly mode, for the maximum of the few minutes required.
I tend to agree, but it varies widely with the circumstances. I've had
servers in unattended colo facilties, and your approach will not work too
well there.
That's actually not so. Over new year I accidently booted my home
server (222 days uptime!) and discovered its boot sector had evaporated.
We've all been there... :-(
Well, maybe I moved the kernels .. anyway, it has no floppy and the
nearest boot cd was an hour's journey away in the cold, on new year. Uh
uh. It took me about 8 hrs, but I booted it via PXE DHCP TFTP
wake-on-lan and the wireless network, from my laptop, without leaving
the warm.
Congrats, but I do hope you did that for your home server...! Cause I'd have
severe moral and practical difficulties selling that to a paying customer:
"So instead of billing me a cab fare and two hours, you spent eight hours to
fix this. And you seriously expect me to pay for those extra hours ?"
quoted
to allocate that space on every of 2 or 3 or 4 or 5 disks). So
it isn't quite relevant how fast the filesystem will be on writes,
and hence it's ok to place it on raid1 composed from 5 components.
That is, uh, paranoid.
We also did use three-way raid-1 mirrors as a rule.
(but I am indeed somewhat paranoid ;-)
quoted
In case of some problem
(yes I dislike any additional layers for critical system components
as any layer may fail to start during boot etc), you can easily
bring the system up by booting off the underlying root-raid partiton
to repair the system -- all the utilities are here. More, you can
Well, you could, and I could, but I doubt if the standard tech could.
I've said it before and I'll say it again: An admin has to be competent. If
not, there is little you can do. You can't have fresh MCSE people fix linux
problems, and you cannot have a carpenter selling stock on wall street.
A "standard tech" as you say, has a skill level that enables him to swap a
drive of a hotswap server if so directed, but anything beyond that is
unrealistic, and he will need adequate help (be it remote by telephone, or
whatever means). Or very extensive onsite step by step documentation.
But why bother? If you didn't have raid there on root you wouldn't
need to repair it. Nothing is quite as horrible as having a
fubarred root partition. That's why I also always have two! But I
don't see that having the copy made by raid rather than rsync wins
you anything in the situaton where you have to reboot - rather, it
puts off that moment to a moment of your choosing, which may be good,
but is not an unqualified bonus, given the cons.
Both approaches have their merits. In one case the danger lies in not having
updated the rsync mirror recently enough, in the other a rogue change will
affect all your mirrors. Without further info on the specific circumstances
no choice can be made, it really depends on too many factors.
quoted
And yes I'm aware of mdp devices (partitions inside the raid
arrays).. but that's just another layer "which may fail": if
raid5 array won't start, I at least can reconstruct filesystem
image by reading chunks of data from appropriate places from
all drives and try to recover that image; with any additional
Now that is just perverse.
Not neccessarily. I've had to rely on using dd_rescue to get data back at
some point is time. In such scenarios, any additional layer can quickly
complicate things beyond reasonable recourse.
As you noted yourself, keeping a backup stategy can be hard work. ;-|
quoted
Note above about swap: in all my
systems, swap is also on raid (raid1 in this case). At the first
look, that can be a nonsense: having swap on raid. But we had
enouth cases when due to a failed drive swap becomes corrupt
(unreadable really), and the system goes havoc, *damaging*
other data which was unaffected by the disk failure! With
Yes, this used to be quite common when swap had that size bug.
When you have swap on a failed disk, often the safer way is to stop the
machine by using the reset button instead of attempting a shutdown.
The shutdown would probably fail halfway through anyway...
Maarten
From: Michael Tokarev <hidden> Date: 2005-01-03 21:36:49
Peter T. Breuer wrote:
Michael Tokarev [off-list ref] wrote:
quoted
Peter T. Breuer wrote:
quoted
This is always a VERY bad idea. /boot and /root want to be on as simple
and uncomplicated a system as possible....
[]
Well, my experience is that anything "unusual" is bad: sysadmins change
over the years; the guy who services the system may not be the one that
built it; the "rescue" cd or floppy he has may not have MD support
built into the kernel (and he probably will need a rescue cd just to get
support for a raid card, if the machine has hardware raid as well as or
instead of software raid).
It is trivial to learn (or teach) that you can boot with root=/dev/sda1
instead of root=/dev/md1. All our technicians knows that. Indeed, most
will not be able to recover the system in most cases anyway IF root raid
will not start "by its own", but that's a different matter entirely.
Sometimes it may be a trivial case (like we had just a few months ago:
I asked our guy to go to the remote office to replace a drive and gave
him a replacement, which contained raid component in the first partition..
and "stupid boot code" (Mine!.. sort of ;) descided to bring THAT array
instead of the real raid array on original disks because the event
counter was greather. So we ended up booting with
root=/dev/sda1 ro init=/bin/sh
and just zeroing the partition in the new drive.. which I simple forgot
to do in the first place. All by phone, it took about 5 minutes to
complete the whole task and bring the server up after the reboot that
he performed when went there.. total downtime was about 15 minutes.
After which, I logged into the system remotely, verified data integrity
of existing arrays and added the new disk to all the arrays -- while
the system was in production (just a bit slow) and our guy was having
some tea (or whatever... ;) before going back.
Having several root partitions is a good thing. I rarely need to
tweak the boot process (that to say: it rarely fails, except of
several stupid cases like the above; And when it fails, it is
even possible to bring the system up while on phone with any non-
technicial "monkey" from their remote office who don't even know
latin letters (we're in russia so english isn't our native language;
when describing what to do I sometimes tell to press "cyrillic"
keys on the keyboard to produce latin characters... ;)
And yes, any bug that "crashes" this damn root system will mess
with the whole thing, with all mirrors.. which is a fatal error,
so to say. Well.. if you can't trust the processor for example
to correctly multiple 2*2 and want to use two (or more) processors
just to be sure one isn't lying.. such systems do exists too, but
hey, they aren't cheap at all... ;)
Therefore, I have learned not to build a system that is more complicated
than the most simple human being that may administer it. This always
works - if it breaks AND they cannot fix it, then THEY get the blame.
Perhaps our conditions are a bit different.. who kows.
Raid1 IS simple -- both in implementation and in usage.
Our guys are trained very hard to ensure they will NOT try to mess things
up unless they're absolutely sure what they're doing. And I don't care
who to blame (me or my technicians or whatever): we all will lose money
in case of serious problems (we're managing networks/servers for $customers,
they're paying us for the system to work with at most one business day
downtime in case of any problem, with all their data intact -- we're
speaking of remote locations here)... etc.. ;)
If you don't trust your guys to do right (or to ask and *understand*
when they don't know), or if your guys are doing mistakes all over --
perhaps it's a time to hire them off? ;)
And let's not get into what they can do to the labelling on the
partition types - FD? Must be a mistake!
BTW, letting the kernel to start arrays is somewhat.. wrong (i mean the
auto-detection and that "FD" type). Again, I learned it the hard way ;)
The best is to have initrd and pass GUUID of your raid array into it
(this is what I *don't* do currently ;)
[]
Well, whenever I buy anything, I buy two. I buy two _controller_ cards,
and tape the extra one inside the case. But of course I buy two
machines, so that is four cards ... .
Oh well... that all depends on the amount of $money. We have several
"double-machines" too here, but only several. The rest of systems
are single, with a single (usually onboard) scsi controller.
And I betcha softraid sb has changed format ver the years. I am still
running P100s!
It isn't difficult to re-create the arrays, even remotely (if you're
accurate enouth ofcourse). Not that it is needed too often either... ;)
quoted
disks are really 35Gb or 37Gb; in case they're differ, "extra" space
on large disk isn't used); root and /boot are on small raid1 partition
which is mirrored on *every* disk; swap is on raid1; the rest (/usr,
I like this - except of course that I rsync them, not raid them. I
don't mind if I have to reboot a server. Nobody will notice the tcp
outage and the other one of the pair will failover for it, albeit in
readonly mode, for the maximum of the few minutes required.
Depends alot of your usage patterns, or tasks running, or other
conditions (esp. physical presense). I for one can't afford
rebooting many of them at all, for a very simple reason: many
of them are on a very bad dialups, and some are several 100s KMs
(or miles for that matter) away... ;) And also most of the
boxes are running oracle with quite complex business rules,
they're calculating some reports which sometimes takes quite
some time to complete, esp at the end of the year.
For this very reason -- they're all quite far away from me,
out of reach -- I tend to ensure they WILL boot and will
be able to dial out and bring that damn tunnel so I can
log in and repair whatever is needed... ;)
Your swap idea is crazy, but crazy enough to be useful. YES, there used
to be a swap bug which corrupted swap every so often (in 2.0? 2.2?) and
meant one had to swapoff and swapon again, having first cleared all
processes by an init 1 and back. Obviously that bug would bite whatever
you had as media, but it still is a nice idea to have raided memory
:-).
It sounds crazy at the first look, I wrote just that. But it
helps to ensure the system is running as if nothing happened.
I just receive email telling me node X has one array degraded,
I log in there remotely, diagnose the problem and do whatever
is needed (remapping the bad block, or arranging to send a guy
there with a replacement drive when there will be such a chance..
whatever). The system continues working just fine for all that
time (UNLESS another drive fails too ofcourse -- all the raid5
arrays will be dead and "urgent help" will be needed in that
case -- at *that* time only it will be possible to reboot as
many times as needed).
The key point with my setup is that the system will continue
working in case of any single drive failure. If I need more
protection, I'll use raid6 or raid10 (or raid1 on several
drives, whatever) so the system will continue working in case
of multiple drive failures. But it will still be running,
giving me a time/chance to diagnose the prob and find good
schedule for our guys to come and fix things if needed --
maybe "right now", maybe "next week" or even "next month",
depending on the exact problem.
quoted
/home, /var etc) are on raid5 arrays (maybe also raid0 for some "scratch"
I don't put /var on raid if I can help it. But there is nothing
particularly bad about it. It is just that /var is the most active
place and therefore the most likely to suffer damage of some kind, somehow.
And damaged raided partitions are really not nice. Raid does not
protect you against hardware corruption - on the contrary, it makes it
more difficult to spot and doubles the probabilities of it happening.
Heh.. In our case, the most active (and largest) filesystem is /oracle ;)
And yes I know using raid5 for a database isn't quite a good idea..
but that's entirely different topic (and nowadays, when raid5 checksumming
is very cheap in terms of cpu, maybe it isn't that bad anymore ;)
Yes raid5 is complex beast compared to raid1. Yes, raid5 may not be
appropriate for some workloads. And still -- yes, this all is about
a compromise in money one have and what he can afford to lose and
for what duration, and how much (money again, but in this our case
that'll be our money, not $client money) one can spend to fix the
problem IF it will need to be fixed the "hard way" (so far we have
two cases which required restoration the hard way, one was because
$client used cheap hardware instead of our recommendations and
non-ECC memory failed -- in that case, according to contract, it
was $client who paid for the restore; and second was due to
software error (oracle bug, now fixed), but we had "remote backup"
of the database (it's a large distributed system and the data is
replicated among several nodes), so I exported the "backup" and
just re-initialized their database. And yes, we simulated various
recovery scenarios in our own office on a toy data, to be sure
we will be able to recover the thing the "hard way" if that'll
be needed).
quoted
space). This way, you have "equal" drives, and *any* drive, including
boot one, may fail at any time and the system will continue working
as if all where working, including reboot (except of a (very rare in
fact) failure scenario when your boot disk has failed MBR or other
sectors required to boot, but "the rest" of that disk is working,
in which case you'll need physical presence to bring the machine up).
That's actually not so. Over new year I accidently booted my home
server (222 days uptime!) and discovered its boot sector had evaporated.
Uh-oh. So just replace the first and second (or third, 4th) disks
and boot from that... ;) Yes that can happen -- after all, lilo may
have a bug, or a bios, or mbr code... But that's again about whenever
you can afford to "trust the processor", above. Yes again, there are
humans which tend to make mistakes (I made alot of mistakes in my
life, oh, alot of them!, once I even formatted the "wrong disk" and
lost half a year of our work.. and started doing some backups finally ;).
I don't think there's anything that can protect against human mistakes --
I mean humans who manage the system, not who use it.
Well, maybe I moved the kernels .. anyway, it has no floppy and the
nearest boot cd was an hour's journey away in the cold, on new year. Uh
uh. It took me about 8 hrs, but I booted it via PXE DHCP TFTP
wake-on-lan and the wireless network, from my laptop, without leaving
the warm.
Heh.
Well, I always have bootable cd or a floppy, just in case. Not to say
I really used that at least once (but I do know it contains all tools
needed for boot and recovery). Yes, shit happens (tm) too... ;)
quoted
All the drives are "symmetrical", usage patterns for all drives are
the same, and due to usage of raid arrays, load is spread among them
quite nicely. You're free to reorder the drives in any way you want,
to replace any of them (maybe rearranging the rest if you're
replacing the boot drive) and so on.
You can do this hot? How? Oh, you must mean at reboot.
Yes -- here I was speaking about the "worst case", when boot fails
for whatever reason. I never needed the "boot floppy" just because
of this: I can make any drive bootable just by changing the SCSI IDs,
and the system will not notice anything changed (it will in fact:
somewhere in dmesg you'll find "device xx was yy before" message
from md, that's basically all). 99% of the systems we manage don't
have hot-swap drives, so in case a drive have to be replaced, reboot
is needed anyway. The nice thing is that I don't care which drive
is being replaced (except of the boot one -- in that case our guys
knows they have to set up another - any - drive as bootable), and
when system boots, I just do
for f in 1 2 3 5 6 7 8 9; do
mdadm --add /dev/md$f /dev/sdX$f
done
(note the device numbering too: mdN is built of sd[abc..]N)
and be done with that (not really *that* simple, I prefer to
verify integrity of other drives before adding the new one,
but that's just details).
quoted
Yes, root fs does not changes often, and yes it is small enouth
(I use 1Gb, or 512Mb, or even 256Mb for root fs - not a big deal
Mine are always under 256MB, but I give 512MB.
quoted
to allocate that space on every of 2 or 3 or 4 or 5 disks). So
it isn't quite relevant how fast the filesystem will be on writes,
and hence it's ok to place it on raid1 composed from 5 components.
That is, uh, paranoid.
The point isn't about paranoia, it's about simplicitly. Or symmetry,
which leads to that same simplicity again. They're all the same and
can be used interchangeable, period. For larger amount of disks,
such a layout may be not as practical, but it should work find with
up to, say, 6 disks (but I'm somewhat afraid to use raid5 with 6
components, as the chance to have two failed drives, which is "fatal"
for raid5, increases).
quoted
The stuff just works, it is very simple to administer/support,
and does all the "backups" automatically.
Except that it doesn't - backups are not raid images. Backups are
snapshots. Maybe you mean that.
"Live" backups ;)... with all the human errors on them too.
Raid1 can't manage snapshots.
quoted
In case of some problem
(yes I dislike any additional layers for critical system components
as any layer may fail to start during boot etc), you can easily
bring the system up by booting off the underlying root-raid partiton
to repair the system -- all the utilities are here. More, you can
[]
quoted
boot from one disk (without raid) and try to repair root fs on
another drive (if things are really screwed up), and when you're
done, bring the raid up on that repaired partition and add other
drives to the array.
But why bother? If you didn't have raid there on root you wouldn't
need to repair it.
See above for a (silly) example -- wrong replacement disk ;)
And indeed that's silly example. I once had another "test case"
to deal with, when my root raid was composed of 3 components
and each had different event counter, for whatever reason (I
don't remember the details already) -- raid1 was refusing to
start. It was with 2.2.something kernel -- things changed
since that time alot, but the "testcase" still was here
(it happened on our test machine in office).
Nothing is quite as horrible as having a
fubarred root partition. That's why I also always have two! But I
don't see that having the copy made by raid rather than rsync wins
you anything in the situaton where you have to reboot - rather, it
puts off that moment to a moment of your choosing, which may be good,
but is not an unqualified bonus, given the cons.
It helps keeping the machine running *and* bootable even after
losing the boot drive (for some reason our drives fails completely
most of the time, instead of developing bad sectors; so next reboot
(our remote offices have somewhat unstable power and gets rebooted
from time to time) will be from the 2nd drive...). It saves you
from the "newaliases" problem ("forgot to rsync" maybe silly, but
even after a small change, remembering/repeating it after the recovery
again (if the crash happened before rsync but after the change) --
I'm lazy ;) Yes this technique places more "load" on the administrator,
because every his mistake gets mirrored automatically and immediately...
There was a funny case with another box I installed for myself.
It's in NY, USA (I'm in Russia, remember?), and there's noone
at the colo facility who knows linux well enouth, and the box
in question has no serial console (esp bios support). After
installing the system (there was some quick-n-dirty linux
"preinstalled" by the colo guys -- thanks god they created
two partitions (2nd one was swap), -- I loaded another distro
to the swap partition, rebooted and re-partitioned the rest
moving /var etc into real place... Fun by itself, but that's
not the point. After successeful install, "in a hurry", I
did some equivalent of... rm -rf /* ! Just a typo, but WHAT
typo! ;)
I was doing all that from home over a dialup. I hit Ctrl-C
when it wiped out /boot, /bin (mknod! chmod! cat!), /etc, /dev,
and started removing /home (which was quite large). Damn fast
system!.. ;) I had only one ssh window opened...
With the help from uudecode (wich is in /usr/bin) and alot of
cut-n-pasteing, I was able to create basic utilities on that
system -- took them from asmutils site. Restored /dev somehow,
cut-n-pasted small wget, and slowly reinstalled the whole
system again (it's debian). Took me 3 damn hours of a very
hard work to reinstall and configure and to ensure everything
is ok to reboot -- I had no other chance to correct any boot
mistakes, and, having in mind our bad phone lines, the whole
procedure was looking almost impossible.
I asked a friend of mine to log in before the reboot and to
check if everything looks ok and it will actually boot. But
it just worked. After that excersise, I was sleeping for more
than 12 hours in a row, because I was really tired.
That to say: ugh-oh, damn humans, there's nothing here to
protect the poor machines from them, they always will find
their ways to screw things up... ;)
Yes, having non-raid rsynced backup helps in that case,
and yes, such a case is damn rare...
Dunno whichever is "right". After all, nothing stops to
have BOTH mirrored AND backed-up root filesystem... ;)
quoted
To summarize: having /boot and root on raid1 is a very *good* idea. ;)
It saved our data alot of times in the past few years already.
No - it saved you from taking the system down at that moment in time.
You could always have rebooted it from a spare root partition whether
you had raid there or not.
Quite a problem when the system is away from you... ;)
quoted
If you're worried about "silent data corruption" due to different
data being read from different components of the raid array.. Well,
first of all, we never saw that yet (we have quite good "testcase")
It's hard to see, and youhave to crash and come back up quite a lot to
make it probable. A funky scsi cable would help you see it!
We did alot of testing by our own too. Sure that's not cover every
possible case. Yet all of the 200+ systems we manage are working
just fine since 1999, with no single "bad" failure so far (I already
mentioned 2 cases which aren't really count for obvious reasons).
[]
quoted
without drives with uncontrollable write caching (quite common for
IDE drives) and things like that, and with real memory (ECC I mean),
where you *know* what you're writing to each disk (yes, there's also
another possible cause of a problem: software errors aka bugs ;),
Indeed, and very frequent they are too.
quoted
that case with different data on different drives becomes quite..
rare. In order to be really sure, one can mount -o remount,ro /
and just compare all components of the root raid, periodically.
When there's more than 2 components on that array, it should be
easy to determine which drive is "lying" in case of any difference.
I do similar procedure on my systems during boot.
Well, voting is one possible procedure. I don't know if softraid does
that anywhere, or attempts repairs.
Neil?
It does not do that.. yet.
quoted
quoted
quoted
There is nowhere that is not software RAID to put the journals, so
Well, you can make somewhere. You only require an 8MB (one cylinder)
partition.
Note scsi disks in linux only supports up to 14 partitions, which
You can use lvm (device mapper). Admittedly I was thinking of IDE.
The same problem as with partitionable raid arrays, and with your
statement about simplicity: lvm layout may be quite complex and
quite difficult to repair *if* something goes really wrong.
And.. oh, no IDE please, thanks alot!.. :)
If you like I can patch scsi for 63 partitions?
I did that once myself - patched 2.2.something to have 63 partitions
on scsi disks. But had alot of problems with other software after
that, because some software assumes device 8,16 is sdb, and because
I always have to remember to boot the "right" kernel.
Nowadays, for me at least, things aren't that bad anymore. I was
using (trying to anyway) raw devices with oracle instead of using
the filesystem (oracle works better that way because of no double-
caching in oracle and in the filesystem). Now there's such a thing
as O_DIRECT which works just as good. Still, we use 8 partitions
on most systems, and having single "journal partition" means we
will need 16 partitions which is more than linux allows.
quoted
[] I at least can reconstruct filesystem
image by reading chunks of data from appropriate places from
all drives and try to recover that image; with any additional
Now that is just perverse.
*If* things really goes wrong. I managed to restore fubared
raid5 once this way, several years ago. Not that the approach
is "practical" or "easy", but if you must and the bad has already
happened... ;)
[]
quoted
Again: instead of using a partition for the journal, use (another?)
raid array. This way, the system will work if the drive wich
contains the journal fails.
But the journal will also contain corruptions if the whole system
crashes, and is rebooted. You just spent several paragraphs (?) arguing
so. Do you really want those rolled forward to complete? I would
rather they were rolled back! I.e. that the journal were not there -
I am in favour of a zero size journal, in other words, which only acts
to guarantee atomicity of FS ops (FS code on its own may do that), but
which does not contain data.
That's another case again. Trust your cpu? Trust the kernel?
If the system can go havoc for some random reason and throw your
(overwise prefectly valid) data away, there's nothing to protect
it.. except of good backup, and, ofcourse, fixing the damn bug.
And it's really irrelevant in this case whenever we have journal
at all or not.
If there IS a journal (my main reason to use it is that sometimes
ext2fs can't repair on reboot without prompting (which is about to
unacceptable to me because the system is remote and I need it to
boot and "phone home" for repair), while with ext3 we had no case
(yet) when it was unable to boot without human intervention), again,
in my "usage case", it should be safe against disk failures, ie,
the system should continue working the the drive where the journal
is gets lost or develops bad sectors. For the same reason... ;)
quoted
[]
And I also want to "re-reply" to the first your message in this
thread, where I was saying that "it's a nonsense that raid does
not preserve write ordering". Ofcourse I mean not write ordering
but working write barriers (as Neil pointed out, md subsystem does
not implement write barriers directly but the concept is "emulated"
by linux block subsystem). Write barriers should be sufficient to
implement journalling safely.
I am not confident that Neil did say so. I have not reexamined his
post, but I got the impression that he hummed and hawed over that.
I do not recall that he said that raid implements write barriers -
perhaps he did. Anyway, I do not recall any code to handle "special"
requests, which USED to be the kernel's barrier mechanism. Has that
mechanism changed (it could have!)?
"Too bad" I haven't looked at the code *at all* (almost, really) ;)
I saw numerous discussions here and there, but it's difficult to
understand *that* amount of code with all the "edge cases".
I just "believe" there's some way to know the data has been written
and that it indeed has been written; and I know this is sufficient
to build a "safe" (in some sense) filesystem (or whatever) based on
this; and what ext3 IS that "safe" filesystem. Just believe, that's
all... ;)
BTW, thanks for a good discussion. Seriously. It's very rare one
can see this level of expirience as you demonstrate.
/mjt
From: Roger Ellison <hidden> Date: 2005-01-05 05:50:41
I've been having an enjoyable time tinkering with software raid with
Sarge and the RC2 installer. The system boots fine with Raid 1 for
/boot and Raid 5 for /. I decided to experiment with Raid 10 for /opt
since there's nothing there to destroy :). Using mdadm to create a Raid
0 array from two Raid 1 arrays was simple enough, but getting the Raid
10 array activated at boot isn't working well. I used update-rc.d to
add the symlinks to mdadm-raid using the defaults, but the Raid 10 array
isn't assembled at boot time. After getting kicked to a root shell, if
I check /proc/mdstat only md1 (/) is started. After running mdadm-raid
start, md0 (/boot), md2, and md3 start. If I run mdadm-raid start again
md4 (/opt) starts. Fsck'ing the newly assembled arrays before
successfully issuing 'mount -a' shows no filesystem errors. I'm at a
loss and haven't found any similar issue mentions on this list or the
debian-users list. Here's mdadm.conf:
DEVICE partitions
DEVICE /dev/md*
ARRAY /dev/md4 level=raid0 num-devices=2
UUID=bf3456d3:2af15cc9:18d816bf:d630c183
devices=/dev/md2,/dev/md3
ARRAY /dev/md3 level=raid1 num-devices=2
UUID=a51da14e:41eb27ad:b6eefb94:21fcdc95
devices=/dev/sdb5,/dev/sde5
ARRAY /dev/md2 level=raid1 num-devices=2
UUID=ac25a75b:3437d397:c00f83a3:71ea45de
devices=/dev/sda5,/dev/sdc5
ARRAY /dev/md1 level=raid5 num-devices=4 spares=1
UUID=efec4ae2:1e74d648:85582946:feb98f0c
devices=/dev/sda3,/dev/sdb3,/dev/sdc3,/dev/sde3,/dev/sdd3
ARRAY /dev/md0 level=raid1 num-devices=4 spares=1
UUID=04209b62:6e46b584:06ec149f:97128bfb
devices=/dev/sda1,/dev/sdb1,/dev/sdc1,/dev/sde1,/dev/sdd1
Roger
From: Andy Smith <hidden> Date: 2005-01-05 09:56:58
As a small request please could those who are posting opinions on
the following:
- What level of schooling any person's mathematical ability resembles
- Whether Peter's computing environment is sane or not
- IQ level and admin skills of various persons
- Issues of p2p, rootkits, unruly students, etc. etc.
Plus various other personal attacks and ad hominem:
Please consider if what you are writing is relevant to this list
(linux-raid) or the subject of this thread (whether it is wise to
put the journal for an ext3 filesystem internal to the filesystem
when it is on a RAID-1 mirror).
Obviously I am not a moderator or even anyone of any influence, but
the majority of text I am now seeing in this thread is not useful to
read and (since it is archived) may actually be giving a bad
impression of its poster for all time.
From what I can understand of the thread so far, Peter is saying the
following:
RAID mirrors are susceptible to increasing undetectable
inconsistencies because, as we all know, filesystems sustain
corruption over time.
On a filesystem that runs from one disk, corruption serious
enough to affect the stability of the file system will do so
and so will be detected. As more disks are added to the
mirror, the probability of that corruption never being seen
naturally goes up.
Peter personally does not put the journal inside the mirror
because if he ever came to need to use the journal and found
that it was corrupted, it could risk his whole filesystem.
Peter prefers to put the journal on a separate device that
is not mirrored.
I am not trying to put words into your mouth Peter, just trying to
summarise what your points are. If I haven't represented your views
correctly then by all means correct me but please try to do so
succinctly and informatively.
Now, others are saying in response to this, things like:
Spontaneous corruption is rare compared to outright or
catastrophic device failure, and although it is more
likely to go unnoticed with RAID mirrors, while it IS
unnoticed, this presumably correct data is also being rewritten
back to the filesystem.
Mirrors help protect against the more common complete device
failure and so a journal should surely be on a mirror since
if you lose the journal then the machine needs to go down
anyway. It is unavailability of the server we're trying to
avoid; consistency of the data can be protected with regular
backups and possibly measured with other methods like
md5sum.
Discuss? ;)
hi ya andy
good summary ... thanx..
one more item :-)
On Wed, 5 Jan 2005, Andy Smith wrote:
..
From what I can understand of the thread so far, Peter is saying the
following:
RAID mirrors are susceptible to increasing undetectable
inconsistencies because, as we all know, filesystems sustain
corruption over time.
On a filesystem that runs from one disk, corruption serious
enough to affect the stability of the file system will do so
and so will be detected. As more disks are added to the
mirror, the probability of that corruption never being seen
naturally goes up.
Peter personally does not put the journal inside the mirror
because if he ever came to need to use the journal and found
that it was corrupted, it could risk his whole filesystem.
Peter prefers to put the journal on a separate device that
is not mirrored.
I am not trying to put words into your mouth Peter, just trying to
summarise what your points are. If I haven't represented your views
correctly then by all means correct me but please try to do so
succinctly and informatively.
Now, others are saying in response to this, things like:
Spontaneous corruption is rare compared to outright or
catastrophic device failure, and although it is more
likely to go unnoticed with RAID mirrors, while it IS
unnoticed, this presumably correct data is also being rewritten
back to the filesystem.
Mirrors help protect against the more common complete device
failure and so a journal should surely be on a mirror since
if you lose the journal then the machine needs to go down
anyway. It is unavailability of the server we're trying to
avoid; consistency of the data can be protected with regular
backups and possibly measured with other methods like
md5sum.
some other issues ...
how one can detect failures, errors would be completely
up to the tools they use ... various tools does specific
functions and cannot tell you anything about any other
causes of the problems
for swap ... i personally don't see any reason to mirror
swap partitions ...
- once the system dies, ( power off ), all temp
data is useless unless one continues from a coredump
( from the same state as when it went down initially )
if a disk did fail, die, error, hiccup, then whatever cause the
problem can also affect the data and the metadata and the parity
and the "mirror"
- which set of "bytes" on the disk "raid" trust to
restore from is up to the code and its predefined
set of assumptions of various failure modes
- partially written data is very bad thing to have
unless you know "exactly why and how if failed/eerrored,
there is no sane way to bet the house on which data is more
correct than the other
- i'm excluding bad memory, bad cpu, bad power supply
from the lsit of possible problems
and yes, bad (generic) memory has corrupted my systems
once in 10 yrs...
have fun
alvin
for swap ... i personally don't see any reason to mirror
swap partitions ...
- once the system dies, ( power off ), all temp
data is useless unless one continues from a coredump
( from the same state as when it went down initially )
I beg to differ on this one. Having spend several weeks tracking down random processes dying on a
machine that turned out to be a bad sector in the swap partition, I have had great results by
running swap on a RAID-1. If you develop a bad sector in a non-mirrored swap, bad things happen
indeterminately and can be a royal PITA to chase down. It's just a little extra piece of mind.
Regards,
Brad
hi ya brad
On Wed, 5 Jan 2005, Brad Campbell wrote:
Alvin Oga wrote:
quoted
for swap ... i personally don't see any reason to mirror
swap partitions ...
- once the system dies, ( power off ), all temp
data is useless unless one continues from a coredump
( from the same state as when it went down initially )
I beg to differ on this one. Having spend several weeks tracking down random processes dying on a
machine that turned out to be a bad sector in the swap partition, I have had great results by
running swap on a RAID-1. If you develop a bad sector in a non-mirrored swap, bad things happen
indeterminately and can be a royal PITA to chase down. It's just a little extra piece of mind.
okay .... if the parts of disks is bad that is used for swap,
mirroring might help ...
but, i wonder, how/why the system used that portion of swap in the first
place
- even for raid, if sector-10 in swap is bad, why would raid keep
trying to write there instead of to sector-1000
( the systems should be writing data, and read it back to verify
( what it wrote to disk, not the disk cache, is the same as what
( it just read back, before it says, "data is written"
and i spent days trackign down a bad memory ... that killed the raid ...
- various backups the only reasonable way to make sure
[supposedly good] data is not lost where backups does NOT overwrite
previous[ly good] backups
c ya
alvin
hi ya brad
On Wed, 5 Jan 2005, Brad Campbell wrote:
quoted
Alvin Oga wrote:
quoted
for swap ... i personally don't see any reason to mirror
swap partitions ...
- once the system dies, ( power off ), all temp
data is useless unless one continues from a coredump
( from the same state as when it went down initially )
I beg to differ on this one. Having spend several weeks tracking down random processes dying on a
machine that turned out to be a bad sector in the swap partition, I have had great results by
running swap on a RAID-1. If you develop a bad sector in a non-mirrored swap, bad things happen
indeterminately and can be a royal PITA to chase down. It's just a little extra piece of mind.
okay .... if the parts of disks is bad that is used for swap,
mirroring might help ...
but, i wonder, how/why the system used that portion of swap in the first
place
- even for raid, if sector-10 in swap is bad, why would raid keep
trying to write there instead of to sector-1000
Picture lpd gets swapped out on a friday night. Over the weekend it is not used and the drive
develops a bad sector in the middle of the file. Monday morning I want to print and the system tries
to page lpd back in again. *boom*.
I have not looked at the system swap algorithms, but I doubt they include automatic bad block
management and read after write verification. I'm making big assumptions here, but I'm assuming they
rely on a bad block table created by mkswap and an otherwise clean, functioning swap area.
Regards,
Brad
Picture lpd gets swapped out on a friday night. Over the weekend it is not used and the drive
develops a bad sector in the middle of the file. Monday morning I want to print and the system tries
to page lpd back in again. *boom*.
admin issue ... user related services like printers should NOT be on
critical servers and take down what everybody will notice due to unrelated
printer problems
swap ... swap partitions is by default checked for bad blocks during
formatting as swap ...
- it is highly unlikely that you'd get a bad sector in swap space
- any normal bad things happening to user area of the disks will also
happen to swap space
- but its unlikely that swap will have bad blocks, while its
more likely that users did nto do a badblock check during
formatting across the 100GB or 300GB disks ... and even more
time twiddling your thumbs on raid'd disks
c ya
alvin
I have not looked at the system swap algorithms, but I doubt they include automatic bad block
management and read after write verification. I'm making big assumptions here, but I'm assuming they
rely on a bad block table created by mkswap and an otherwise clean, functioning swap area.
Regards,
Brad
Picture lpd gets swapped out on a friday night. Over the weekend it is not used and the drive
develops a bad sector in the middle of the file. Monday morning I want to print and the system tries
to page lpd back in again. *boom*.
admin issue ... user related services like printers should NOT be on
critical servers and take down what everybody will notice due to unrelated
printer problems
Ok, perhaps bad example. Pick any service that may get swapped out. Hell.. any bad block in any swap
space that develops while that block is in use is going to cause a problem.
swap ... swap partitions is by default checked for bad blocks during
formatting as swap ...
- it is highly unlikely that you'd get a bad sector in swap space
Yeah? I have 3 hard disks sitting on my desk that say you are wrong..
- any normal bad things happening to user area of the disks will also
happen to swap space
I agree.
- but its unlikely that swap will have bad blocks, while its
more likely that users did nto do a badblock check during
formatting across the 100GB or 300GB disks ... and even more
time twiddling your thumbs on raid'd disks
I strongly disagree.
I have a 250GB drive I just swapped out that was "growing" bad sectors at the rate of 3 per day that
did a clean badblocks 5 months ago when it was installed.
I refuse to argue this any more. If you want to tell the world that there is no benefit to swap on
raid, go ahead.
Brad
From: Andy Smith <hidden> Date: 2005-01-05 13:36:54
On Wed, Jan 05, 2005 at 05:23:19AM -0800, Alvin Oga wrote:
On Wed, 5 Jan 2005, Brad Campbell wrote:
quoted
Picture lpd gets swapped out on a friday night. Over the weekend
it is not used and the drive develops a bad sector in the middle
of the file. Monday morning I want to print and the system tries
to page lpd back in again. *boom*.
admin issue ... user related services like printers should NOT be on
critical servers and take down what everybody will notice due to unrelated
printer problems
The problem is not that an lpd process exists, the problem is that
an attempt was made to swap back in a swapped out process while the
swap device is screwed. It could happen to any process. Swap is
part of your virtual memory, either keep it working or don't use it
at all -- you cannot expect your server to keep on working when half
its virtual memory is on a device that just died.
From: Michael Tokarev <hidden> Date: 2005-01-05 13:41:04
[Please don't start new thread by replying to another message.]
Roger Ellison wrote:
I've been having an enjoyable time tinkering with software raid with
Sarge and the RC2 installer. The system boots fine with Raid 1 for
/boot and Raid 5 for /. I decided to experiment with Raid 10 for /opt
since there's nothing there to destroy :). Using mdadm to create a Raid
0 array from two Raid 1 arrays was simple enough, but getting the Raid
Note there's special raid10 module in recent kernels (and supported by
recent mdadm). I don't think it's very stable yet, but.. JFYI ;)
10 array activated at boot isn't working well. I used update-rc.d to
add the symlinks to mdadm-raid using the defaults, but the Raid 10 array
Shouldn't the links be made automatically when installing mdadm?
Also, make sure you're using recent debian package of mdadm --
earlier versions was.. with issues concerning assembling the
arrays (a problem specific to debian mdadm-raid scripts).
Make sure AUTOSTART is set to true in /etc/default/mdadm
(I'm not sure if that's really `AUTOSTART' and `true',
you can look at the file and/or use dpkg-reconfigure mdadm
to set that).
isn't assembled at boot time. After getting kicked to a root shell, if
I check /proc/mdstat only md1 (/) is started. After running mdadm-raid
start, md0 (/boot), md2, and md3 start. If I run mdadm-raid start again
md4 (/opt) starts. Fsck'ing the newly assembled arrays before
successfully issuing 'mount -a' shows no filesystem errors. I'm at a
loss and haven't found any similar issue mentions on this list or the
debian-users list. Here's mdadm.conf:
You have two problems.
First of all, mdadm-raid should be started at very early in the
boot process, and mdadm package post-install scripts ensures this
(you added mdadm-raid links at default order which is 20; but
it should run before filesystem mounts etc, nearly 01 or something).
Ditto for stop scripts -- at the very end, after umounting the
filesystems. Take a look at /var/lib/dpkg/info/mdadm.postinst --
it sets up the links properly. When you correct this, your system
will go further in boot process... ;)
And second problem is the order of lines in mdadm.conf.
[edited a bit]
Re-order the lines so that md4 will be listed AFTER all it's
components. Mdadm tries to assemble them in turn as they're
listed in mdadm.conf. But at the time when it tries to start
md4, it's components aren't here yet so it fails.
HTH.
/mjt
I'm having some troubles with an Adaptec 2400A ATA Raid controller on FC3.
Everything worked fine during installation and Operating System recognized
automatically the RAID card.
I also installed raidutils-0.0.4 from http://i2o.shadowconnect.com, but when I
try to use the command raidutil this is the result:
[root@omissis ~]# raidutil -L
osdIOrequest : File /dev/dpti17 Could Not Be OpenedEngine connect
failed: COMPATIBILITY number
I need to manage the raid card at least for display the status of the RAID
array.
Can anybody help me?
Thanks
Angelo Piraino
Linux FC3 on PIII 450MHz - chipset Intel P2BF - 256Mb Ram - Ata Raid
Adaptec 2400A
From: Erik Mouw <hidden> Date: 2005-01-05 14:12:51
On Wed, Jan 05, 2005 at 02:56:30PM +0400, Brad Campbell wrote:
I beg to differ on this one. Having spend several weeks tracking down
random processes dying on a machine that turned out to be a bad sector in
the swap partition, I have had great results by running swap on a RAID-1.
If you develop a bad sector in a non-mirrored swap, bad things happen
indeterminately and can be a royal PITA to chase down. It's just a little
extra piece of mind.
If you have a bad block in your swap partition and the device doesn't
report an error about it, no amount of RAID is going to help you
against it.
Erik
--
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
From: Michael Tokarev <hidden> Date: 2005-01-05 14:37:34
Erik Mouw wrote:
On Wed, Jan 05, 2005 at 02:56:30PM +0400, Brad Campbell wrote:
quoted
I beg to differ on this one. Having spend several weeks tracking down
random processes dying on a machine that turned out to be a bad sector in
the swap partition, I have had great results by running swap on a RAID-1.
If you develop a bad sector in a non-mirrored swap, bad things happen
indeterminately and can be a royal PITA to chase down. It's just a little
extra piece of mind.
If you have a bad block in your swap partition and the device doesn't
report an error about it, no amount of RAID is going to help you
against it.
The drive IS reporting read errors in most cases. But that does not
help, really: kernel swapped out some memory but can't read it back,
so things are screwed. Just like if you hot-remove a DIMM while the
system is running: the kernel loses parts of it's memory, and it can't
work anymore. Depending on what was in there ofcourse: the whole
system may be screwed, or a single process... The talks isn't about
"undetectable" (unreported etc) errors here, but about the fact that
the error is here. And if your swap is on raid, in case one component
of the array behaves badly, another component will continue to work,
so with swap on raid the system will work just fine as if nothing
happened in case one of "swap components" (i mean underlying devices)
failed for whatever reason.
And please, pretty PLEASE stop talking about those mysterious
"undetectable" or "unreported" errors here. A drive that develops
"unreported" errors just does not work and should not be here in
the first place, just like bad memory or CPU: if your cpu or memory
is failing, no software tricks helps and the failing part should
be replaced BEFORE even thinking about possible ways to recover.
/mjt
Ok, perhaps bad example. Pick any service that may get swapped out. Hell.. any bad block in any swap
space that develops while that block is in use is going to cause a problem.
yup ... if you have an unstable system .. swapping things in and out of
memory and disks will cause you problems
quoted
swap ... swap partitions is by default checked for bad blocks during
formatting as swap ...
- it is highly unlikely that you'd get a bad sector in swap space
Yeah? I have 3 hard disks sitting on my desk that say you are wrong..
that was my whole initial point a few days ago
-- get better hardware
-- get your hardware from another source
-- i do NOT have hardware problems ...
- i do NOT buy parts from mthe cheapest place
- i do NOT buy parts from people that i had bad parts
- i do get an occasional bad part, like the ibm deathstars are
infamous
- i do get an occasional 1% infant mortality rate ...
which is normal
- once a server has been up for say 30-60 days ... it stays
up for years ....
- given the same identical hard disks from different vendors/stores,
you will get different failure rates
- i use maxtor/quantum, ibm, western digital, seagates, fujitsu
( no one is better than the other disks, other than
( the stupid ibm deathstar problem
- how you install it
- how you cool it makes all the difference in the world
- think about it ...
== we all use the same motherboards
== we all use the same disks
== we all use the same memory
== we all use the same linux
== whats different ??
( where you bought yours from and more importantly,
( how you cool things down
- i do NOT have problems you guyz are having ....
and i've personally use thousnds of disks
=== get better hardware .. or more to the point ... buy from a
computer parts only tier-1 vendor ... not from the millions of me-too
online "we are the cheapest mom-n-pop we sell camera's and
clothes too webstores"
I have a 250GB drive I just swapped out that was "growing" bad sectors at the rate of 3 per day that
did a clean badblocks 5 months ago when it was installed.
you're buying bad hardware from bad vendors
c ya
alvin
hi ya michael
On Wed, 5 Jan 2005, Michael Tokarev wrote:
The drive IS reporting read errors in most cases.
and by the time it reports an error ... its too late ...
- one has to play detective and figure out why the error occured
and prevent it next time
- the worst possible thing to do is .. "disk seems to be dying"
so 95% of the people say backup the disks ..
-- its too too late
-- exercising a dead/dying disks will only aggravate the
problem even more
- luckily, most "disks errors" are silly errors
- bad (cheap, not-too-specs) ide cables
- 80 conductor vs 40 conductor
- combining 2 different ata speed drives on the same idea cable
- too long of an ide cable ... 18" is max
- disk running too hot ... if its warm to the touch,
its too hot
( 35C is maybe okay .. use hddtemp to see what its running at )
- bent disk cables
- bad (electrical characteristics) of the ide drivers from the
motherboard
- ... on and on ...
- todays drives are 1000x better quality than 5-10 years ago
so things are screwed. Just like if you hot-remove a DIMM while the
system is running:
those that do hot-swap of memory deserve what they get :-)
( i know you're joking ... and i've seen it done by forgetful admins
- these atx power supplies are bad for that reason, since
the motherboard is still live, even if the pc is off, due to
standby voltages floating around
The talks isn't about
"undetectable" (unreported etc) errors here, but about the fact that
the error is here.
the trick is to find out what caused the error ...
- if you dont figure out what happened ... the problem will
continue
have fun raiding
alvin
I agree, but for a different reason. Your reason is new to me.
I don't want a down system due to a single disk failure.
Loosing the swap disk would kill the system.
Maybe this is Peter's cause of frequent corruption?
I mirror everything, or RAID5. Normally, no downtime due to disk failures.
Guy
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Brad Campbell
Sent: Wednesday, January 05, 2005 5:57 AM
To: Alvin Oga
Cc: Andy Smith; linux-raid@vger.kernel.org
Subject: Re: ext3 journal on software raid (was Re: PROBLEM: Kernel 2.6.10
crashing repeatedly and hard)
Alvin Oga wrote:
for swap ... i personally don't see any reason to mirror
swap partitions ...
- once the system dies, ( power off ), all temp
data is useless unless one continues from a coredump
( from the same state as when it went down initially )
I beg to differ on this one. Having spend several weeks tracking down random
processes dying on a
machine that turned out to be a bad sector in the swap partition, I have had
great results by
running swap on a RAID-1. If you develop a bad sector in a non-mirrored
swap, bad things happen
indeterminately and can be a royal PITA to chase down. It's just a little
extra piece of mind.
Regards,
Brad
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I agree, but for a different reason. Your reason is new to me.
..
Loosing the swap disk would kill the system.
if one is using swap space ... i'd add more memory .. before i'd use raid
- swap is too slow and as you folks point out, it could die
due to (unlikely) bad disk sectors in swap area
I don't want a down system due to a single disk failure.
that's what raid's for :-)
I mirror everything, or RAID5. Normally, no downtime due to disk failures.
the problem with mirror ( raid1 ).. or raid5 ...
- if you have a bad diska ... all "bad data" will/could also get
copied to the good disk
- "bad data" is hard to figure out in code ... to prevent it from
getting copied ... how does it know with 100% certainty
- if you know why it's bad data, it's lot easier to know which
data is more correct than the bad one
- as everybody has pointed out .. bad data ( disk errors )
can occur for any number of gazillion reasons
have fun raiding
alvin
From: Peter T. Breuer <hidden> Date: 2005-01-05 15:48:00
Guy [off-list ref] wrote:
I agree, but for a different reason. Your reason is new to me.
I don't want a down system due to a single disk failure.
Loosing the swap disk would kill the system.
Maybe this is Peter's cause of frequent corruption?
I wouldn't have aid there were frequent failures! There are relatively
frequent disk errors, at the rate of probably 10 bits per hundred
machines per week. I do not think that is very frequent.
Peter
From: Michael Tokarev <hidden> Date: 2005-01-05 16:22:04
Alvin Oga wrote:
On Wed, 5 Jan 2005, Guy wrote:
quoted
I agree, but for a different reason. Your reason is new to me.
...
quoted
Loosing the swap disk would kill the system.
if one is using swap space ... i'd add more memory .. before i'd use raid
- swap is too slow and as you folks point out, it could die
due to (unlikely) bad disk sectors in swap area
It isn't always practical. You add as much memory as needed for
your "typical workload". But there may be "spikes" of load with
that you have to deal somehow. Adding more memory to cover that
"spikes" may be too expensive.
Also, if your "typical workload" requires eg 2Gb memory, adding
another, say, 2Gb to cover "spikes" means you have to reconfigure
the kernel to support large amount of memory, which also costs
something in terms of speed on i386 architecture.
Disks are *much* cheaper than ram in terms of money/Mb.
quoted
I don't want a down system due to a single disk failure.
that's what raid's for :-)
quoted
I mirror everything, or RAID5. Normally, no downtime due to disk failures.
the problem with mirror ( raid1 ).. or raid5 ...
- if you have a bad diska ... all "bad data" will/could also get
copied to the good disk
Again: pretty PLEASE, stop talking about thouse mysterious "silent
corruption/errors". Errors gets detected. It is *very* unlikely
case when an error on disk (either unability to read, or reading
the "wrong" (aka not the same as has been written) data) will not
be detected during read, and if you do care about that cases, you
have to use some very different hardware with every component
(CPU, memory, buses, controllers etc etc) at least tripled, with
hardware-level online monitoring/comparing stuff to detect errors
at any level and to switch to another component if one is "lying".
- "bad data" is hard to figure out in code ... to prevent it from
getting copied ... how does it know with 100% certainty
Nothing is 100% certain.. maybe except that we all will die sometime...
- if you know why it's bad data, it's lot easier to know which
data is more correct than the bad one
Nothing is "more correct". If the disk isn't working somehow, we know
this (as it reports errors) and kick it from the array. If disk
"does not work silently", see above.
/mjt
From: Andy Smith <hidden> Date: 2005-01-05 16:23:37
On Wed, Jan 05, 2005 at 07:33:55AM -0800, Alvin Oga wrote:
On Wed, 5 Jan 2005, Guy wrote:
quoted
I agree, but for a different reason. Your reason is new to me.
..
quoted
Loosing the swap disk would kill the system.
if one is using swap space ... i'd add more memory .. before i'd use raid
- swap is too slow and as you folks point out, it could die
due to (unlikely) bad disk sectors in swap area
This again is besides the point. As I said already swap is part of
your virtual memory and if you can't keep it available you should
not be using it. It is the same as saying that your machine will
die if half its RAM suddenly ceased to exist while it was running.
Your recommendations so far have been "don't run processes which
might swap" and "don't let swap be used". If you yourself are able
to keep to both of these 100% of the time may I ask why you yourself
have any swap at all?
ometimes) or those of us in the real world where swap is desirable
and may (sometimes) be used, a swap device failure *will* *take*
*the* *machine* *down*.
From: Andy Smith <hidden> Date: 2005-01-05 16:30:34
On Wed, Jan 05, 2005 at 04:23:37PM +0000, Andy Smith wrote:
ometimes) or those of us in the real world where swap is desirable
and may (sometimes) be used, a swap device failure *will* *take*
*the* *machine* *down*.
Something strange happened with vim there. That was meant to read:
"For those of us in the real world where swap is (sometimes)
desirable and may (sometimes) be used, a swap device failure *will*
*take* *the* *machine* *down*."
On Wed, Jan 05, 2005 at 07:33:55AM -0800, Alvin Oga wrote:
quoted
if one is using swap space ... i'd add more memory .. before i'd use raid
- swap is too slow and as you folks point out, it could die
due to (unlikely) bad disk sectors in swap area
...
Your recommendations
that'd be your comment ... that is not whati said above
and all i said, was use memory before you use swap on disks
and if you are at the 2GB lmit of your mb ... you obviously don't
have a choice
- if you are using 128MB of memory and using 2GB of swap
and bitching about a slow box or system crashing due to swap...
- that was the point ... add more memory
- and i don't think anybody is idiotic enough to add
more memory for the "spikes" in the workload
so far have been "don't run processes which
might swap" and "don't let swap be used". If you yourself are able
to keep to both of these 100% of the time may I ask why you yourself
have any swap at all?
you're twisting things again... but no problem.. have fun doing that
ometimes) or those of us in the real world where swap is desirable
and may (sometimes) be used, a swap device failure *will* *take*
*the* *machine* *down*.
my real world is obviously better .... since we do NOT suffer
from these mysterious disks crashes
we go to fix peoples problems
c ya
alvin
RAID does not cause bad data!
Bad data can get on any disk, even if it is part of a RAID system.
The bad data does not come from the hard disk, CRCs prevent that.
The problem is: Where does the bad data come from?
Bad memory? No, everyone use ECC memory, right?
Guy
-----Original Message-----
From: Alvin Oga [mailto:aoga@ns.Linux-Consulting.com]
Sent: Wednesday, January 05, 2005 10:34 AM
To: Guy
Cc: linux-raid@vger.kernel.org
Subject: RE: ext3 journal on software raid (was Re: PROBLEM: Kernel 2.6.10
crashing repeatedly and hard)
On Wed, 5 Jan 2005, Guy wrote:
I agree, but for a different reason. Your reason is new to me.
..
Loosing the swap disk would kill the system.
if one is using swap space ... i'd add more memory .. before i'd use raid
- swap is too slow and as you folks point out, it could die
due to (unlikely) bad disk sectors in swap area
I don't want a down system due to a single disk failure.
that's what raid's for :-)
I mirror everything, or RAID5. Normally, no downtime due to disk
failures.
the problem with mirror ( raid1 ).. or raid5 ...
- if you have a bad diska ... all "bad data" will/could also get
copied to the good disk
- "bad data" is hard to figure out in code ... to prevent it from
getting copied ... how does it know with 100% certainty
- if you know why it's bad data, it's lot easier to know which
data is more correct than the bad one
- as everybody has pointed out .. bad data ( disk errors )
can occur for any number of gazillion reasons
have fun raiding
alvin
From: Erik Mouw <hidden> Date: 2005-01-05 17:11:04
On Wed, Jan 05, 2005 at 05:37:34PM +0300, Michael Tokarev wrote:
Erik Mouw wrote:
quoted
If you have a bad block in your swap partition and the device doesn't
report an error about it, no amount of RAID is going to help you
against it.
The drive IS reporting read errors in most cases.
"most cases" and "all cases" makes quite a difference.
And please, pretty PLEASE stop talking about those mysterious
"undetectable" or "unreported" errors here. A drive that develops
"unreported" errors just does not work and should not be here in
the first place, just like bad memory or CPU: if your cpu or memory
is failing, no software tricks helps and the failing part should
be replaced BEFORE even thinking about possible ways to recover.
Indeed: any suspicious part should not be used in the recovery process.
Erik
--
+-- Erik Mouw -- www.harddisk-recovery.com -- +31 70 370 12 90 --
| Lab address: Delftechpark 26, 2628 XH, Delft, The Netherlands
again .. that's not what i said ..
the point i've been tring to say .. one MUST figure out "exactly"
where the bad data or errors or crashes are coming from
it will usually be bad ("cheap") parts or operator error
or simple "slapped" together boxes
Bad data can get on any disk, even if it is part of a RAID system.
The bad data does not come from the hard disk, CRCs prevent that.
The problem is: Where does the bad data come from?
Bad memory? No, everyone use ECC memory, right?
that's what i was saying... but you do a better job
even if one uses ecc... ecc can only fix certain errors,
and non-correctable errors are flagged and not fixed
( burst errors are hard to fix )
and people that use ecc memory ... usually have higher-end motherboards
and more memory in the system vs the $50 motherboard+cpu combo (disasters)
from fries ( a local pc store )
c ya
alvin
You confuse them here with failures (which probably get detected, but
then who can say!). An error occurs when you do a sum on paper and you
forget to carry one in the third column. In colloquial terms, it's a
"mistake". Life carries on. A failure occurs when your brain explodes
and CNN comes round and interviews your next door neignbour about the
hole in their wall.
It is *very* unlikely
case when an error on disk (either unability to read, or reading
the "wrong" (aka not the same as has been written) data) will not
be detected during read.
It's practically certain that it won't be "detected", because it is
on disk as far as anyone and anything can tell - there would have been a
failure if that were not the case. It's an ordinary datum.
, and if you do care about that cases, you
have to use some very different hardware with every component
(CPU, memory, buses, controllers etc etc) at least tripled, with
hardware-level online monitoring/comparing stuff to detect errors
No, that detects errors internally (and corrects them, or else it
produces "failures" that are externally visible in place of them, or
else it doesn't detect them and the errors are also externally
visible).
at any level and to switch to another component if one is "lying".
It still leaves the errors. I don't know why everyone has such semantic
problems with this! Think of an error as a "bug". You catch those bugs
you can see, and don't catch the bugs you can't see. There are always
bugs you can't see (hey!, if you saw them you would correct them, right?
Or at least die die die). It is simply a classification.
Peter
From: Andy Smith <hidden> Date: 2005-01-05 17:26:27
On Wed, Jan 05, 2005 at 09:04:57AM -0800, Alvin Oga wrote:
On Wed, 5 Jan 2005, Andy Smith wrote:
quoted
On Wed, Jan 05, 2005 at 07:33:55AM -0800, Alvin Oga wrote:
quoted
if one is using swap space ... i'd add more memory .. before i'd use raid
- swap is too slow and as you folks point out, it could die
due to (unlikely) bad disk sectors in swap area
...
quoted
Your recommendations
that'd be your comment ... that is not whati said above
Direct quote:
i'd add more memory before i'd use raid
and all i said, was use memory before you use swap on disks
Which means what? Who is there on this list who likes to use swap
*before* physical memory?
Do all your machines which you believe have adequate memory also
have no swap configured?
Fact is if your machine has swap configured then that swap is part
of your virtual memory and if a device that is providing part of
your virtual memory suddenly fails then your machine is going down.
and if you are at the 2GB lmit of your mb ... you obviously don't
have a choice
- if you are using 128MB of memory and using 2GB of swap
and bitching about a slow box or system crashing due to swap...
- that was the point ... add more memory
No one here is bitching about a slow machine due to swap usage and
if they were I'd wonder why they are doing it on linux-raid.
I repeat, if "add more memory" is your answer to "swapping on a
single device which then dies kills my machine" then does that mean
that your machines are configured with no swap?
All people are saying is that if you don't mirror swap then disk
failures cause downtime. You are replying to add more memory and
don'trun things that can get swapped. Those don't seem like very
useful recommendations.
- and i don't think anybody is idiotic enough to add
more memory for the "spikes" in the workload
OK so you must add swap to handle those spikes which means either
you are happy that your machine will crash should the device that
the swap is on die, or you use swap on a mirror to try to mitigate
that.
quoted
so far have been "don't run processes which
might swap" and "don't let swap be used". If you yourself are able
to keep to both of these 100% of the time may I ask why you yourself
have any swap at all?
you're twisting things again... but no problem.. have fun doing that
From previous email (not a direct quote):
Don't run lpd on user-accessible machine
(after given an example of some lpd process which gets swapped
out and back in again).
quoted
ometimes) or those of us in the real world where swap is desirable
and may (sometimes) be used, a swap device failure *will* *take*
*the* *machine* *down*.
my real world is obviously better .... since we do NOT suffer
from these mysterious disks crashes
If you do not suffer disk crashes then why do you use mirrors at
all? If you do suffer disk crashes then any disk that is providing
swap may very well cause the machine to crash.
I thought everyone suffered disk crashes and that was the point of
RAID.
From: David Greaves <hidden> Date: 2005-01-05 17:26:35
Guy wrote:
RAID does not cause bad data!
Guy - how can you speak such heresy!! ;)
Haven't you seen the special 'make_undetectable_error(float p)' function?
Bad data can get on any disk, even if it is part of a RAID system.
The bad data does not come from the hard disk, CRCs prevent that.
The problem is: Where does the bad data come from?
Bad memory? No, everyone use ECC memory, right?
I like to blame cosmic rays - I just like the image ;)
Of course voltage fluctuations, e/m interference, thermal variations,
mechanical interfaces, dust capacitance/resistance, insects shorts...
anywhere inside the tin box between the media and the corners of the
motherboard. It's amazing that some machines even boot.
David
You said this:
" - as everybody has pointed out .. bad data ( disk errors )
can occur for any number of gazillion reasons
have fun raiding
Alvin"
Bad data is not caused by disk errors. This seemed like you were blaming
the hard disk. I understand you now. It seems we agree!
Guy
-----Original Message-----
From: Alvin Oga [mailto:aoga@ns.Linux-Consulting.com]
Sent: Wednesday, January 05, 2005 12:22 PM
To: Guy
Cc: 'Alvin Oga'; linux-raid@vger.kernel.org
Subject: RE: ext3 journal on software raid (was Re: PROBLEM: Kernel 2.6.10
crashing repeatedly and hard)
hi ya guy
On Wed, 5 Jan 2005, Guy wrote:
RAID does not cause bad data!
again .. that's not what i said ..
the point i've been tring to say .. one MUST figure out "exactly"
where the bad data or errors or crashes are coming from
it will usually be bad ("cheap") parts or operator error
or simple "slapped" together boxes
Bad data can get on any disk, even if it is part of a RAID system.
The bad data does not come from the hard disk, CRCs prevent that.
The problem is: Where does the bad data come from?
Bad memory? No, everyone use ECC memory, right?
that's what i was saying... but you do a better job
even if one uses ecc... ecc can only fix certain errors,
and non-correctable errors are flagged and not fixed
( burst errors are hard to fix )
and people that use ecc memory ... usually have higher-end motherboards
and more memory in the system vs the $50 motherboard+cpu combo (disasters)
from fries ( a local pc store )
c ya
alvin
From: Gordon Henderson <hidden> Date: 2005-01-05 17:34:16
On Wed, 5 Jan 2005, Alvin Oga wrote:
even if one uses ecc... ecc can only fix certain errors,
and non-correctable errors are flagged and not fixed
( burst errors are hard to fix )
and people that use ecc memory ... usually have higher-end motherboards
and more memory in the system vs the $50 motherboard+cpu combo (disasters)
from fries ( a local pc store )
One thing thats been irritating me for a long time is that it's all very
well using (and paying for!) ECC memory, but there are parts of the system
that don't have ECC or parity - eg. the PCI bus, processor bus and
internal paths, and so on...
But where do you draw the line?
Gordon
You forgot one!
Peter said: "There might be a small temporal displacement."
This worries me! RED ALERT!
Beam me up! Wait, I will take the shuttle craft please.
"You cannot change the laws of physics!" - Scotty
But, really, I do understand what Peter is saying. It just seemed too funny
to me. And since I have disk read errors more than temporal displacement, I
need RAID more than a temporal dampening field. Maybe a temporal phase
converter?
And, a single non-RAID disk can suffer from temporal displacement.
I love Star Trek, so I love temporal displacements! :)
Guy
-----Original Message-----
From: David Greaves [mailto:david@dgreaves.com]
Sent: Wednesday, January 05, 2005 12:27 PM
To: Guy
Cc: linux-raid@vger.kernel.org
Subject: Re: ext3 journal on software raid (was Re: PROBLEM: Kernel 2.6.10
crashing repeatedly and hard)
Guy wrote:
RAID does not cause bad data!
Guy - how can you speak such heresy!! ;)
Haven't you seen the special 'make_undetectable_error(float p)' function?
Bad data can get on any disk, even if it is part of a RAID system.
The bad data does not come from the hard disk, CRCs prevent that.
The problem is: Where does the bad data come from?
Bad memory? No, everyone use ECC memory, right?
I like to blame cosmic rays - I just like the image ;)
Of course voltage fluctuations, e/m interference, thermal variations,
mechanical interfaces, dust capacitance/resistance, insects shorts...
anywhere inside the tin box between the media and the corners of the
motherboard. It's amazing that some machines even boot.
David
I think that is in some boot-up scripts! :)
But only if you don't have mirrors! :)
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Peter T. Breuer
Sent: Wednesday, January 05, 2005 1:16 PM
To: linux-raid@vger.kernel.org
Subject: Re: ext3 journal on software raid (was Re: PROBLEM: Kernel 2.6.10
crashing repeatedly and hard)
David Greaves [off-list ref] wrote:
Haven't you seen the special 'make_undetectable_error(float p)' function?
dd if=/dev/urandom of=/dev/hda1 bs=512 count=1 skip=$RANDOM
Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
that'd be your comment ... that is not whati said above
Direct quote:
i'd add more memory before i'd use raid
i see ... when i say "i would use" ... i dont mean or imply
others to do so ..
if i do mean "you" should, or i strongly recommend ... i usually
explicitly say so
quoted
and all i said, was use memory before you use swap on disks
Which means what? Who is there on this list who likes to use swap
*before* physical memory?
you'd be surprized how many people wonder why their system is slow
and they using 100% swap ( hundreds of MB of swap ) and little memory
beause its $50 of expensive memory
Do all your machines which you believe have adequate memory also
have no swap configured?
yes .... in my machines .. they are all mostly tuned for specfic
tasks ... very little ( say 0.05% swap usage even at peak )
Fact is if your machine has swap configured then that swap is part
of your virtual memory and if a device that is providing part of
your virtual memory suddenly fails then your machine is going down.
yup...
and in some systems ... having swap ( too slow ) is NOT an option ...
( embedded systems, realtime operations, ... )
I repeat, if "add more memory" is your answer to "swapping on a
single device which then dies kills my machine" then does that mean
that your machines are configured with no swap?
i'm NOT the one having hardware problems of any sort ...
i'm stating people that are having problems usually have problems
because of many possible reasons ...
- most common one is, that it's a machine just slapped
together with parts on sales from far away website vendors
- if the machine cannot handle swapp ... to one partiton or
or swap on other disks, than they have a serious hardware problem
that they bought pieces of hw junk vs buying good parts
from known good vendors
==
== you should NEVER have swap problems
== otherwise, toss that system out or salvage what you can
==
All people are saying is that if you don't mirror swap then disk
failures cause downtime.
yes ...
and i'm just saying, if you dont know why swapp failed or
that if downtime is important, use better quality hardware
and install it properly ... maintain it properly ...
if you forget about it .. the disk ... the disk will get
lonely too will forget about holding its 1's and 0's too
You are replying to add more memory and
don'trun things that can get swapped.
you're twisting things gain
Those don't seem like very useful recommendations.
that is becuase you're twisting things so you can make
comments i didnt say
quoted
- and i don't think anybody is idiotic enough to add
more memory for the "spikes" in the workload
OK so you must add swap to handle those spikes which means either
you are happy that your machine will crash should the device that
the swap is on die, or you use swap on a mirror to try to mitigate
that.
this is pointless isnt it ...
are you an idiot or what ..
do you like to put words and recommendations that i didnt say
and twist it in your favor
From previous email (not a direct quote):
Don't run lpd on user-accessible machine
you obviously do NOT understand where to run lpd and where
to run dns and where to run pop and where to run ssh and where
to run mta ... and on and on and on ...
If you do not suffer disk crashes then why do you use mirrors at
all?
i do NOT use mirrors for the protection against disk failures
nobody said i did .. you again are twisting things into your
own ideas and misconceptions
If you do suffer disk crashes then any disk that is providing
swap may very well cause the machine to crash.
i dont have that problem ... but if someone ddid have that
problem ... i assuem they are smart enough to figure out
within 5-10 minutes why the disk/system is crashing
I thought everyone suffered disk crashes and that was the point of
RAID.
not everybody suffers disk crashes ... in such great numbers
that raid is better solution ...
- raid is NOT theonly solution
== ever hear of high availability ...
= clusters ...
= even plain ole backups
there are more than one solution to one disk crashing
c ya
alvin
One thing thats been irritating me for a long time is that it's all very
well using (and paying for!) ECC memory, but there are parts of the system
that don't have ECC or parity - eg. the PCI bus, processor bus and
internal paths, and so on...
bingo !!!
But where do you draw the line?
because the silly motherboard will nto work at all without those
expensive registered ecc memory
and you want that motherboard because it has the memory
capacity you want ot the support to the fastest cpu around
or blah features that no other mb has
c ya
alvin
You said this:
" - as everybody has pointed out .. bad data ( disk errors )
can occur for any number of gazillion reasons
...
Bad data is not caused by disk errors. This seemed like you were blaming
the hard disk. I understand you now. It seems we agree!
i should have said "wrong data", where i don;t care why
the data is not what one expects... due to failures, errors, problems,
features, bugs ...
and the point is .. if the data is wrong ... find out precisely
why its wrong ... and experiment to confirm the suspicions
- usually you will toss out one hardware item at a time
to isolate the problem
c ya
alvin
From: Roger Ellison <hidden> Date: 2005-01-05 19:15:49
On Wed, 2005-01-05 at 08:41, Michael Tokarev wrote:
[Please don't start new thread by replying to another message.]
Roger Ellison wrote:
quoted
I've been having an enjoyable time tinkering with software raid with
Sarge and the RC2 installer. The system boots fine with Raid 1 for
/boot and Raid 5 for /. I decided to experiment with Raid 10 for /opt
since there's nothing there to destroy :). Using mdadm to create a Raid
0 array from two Raid 1 arrays was simple enough, but getting the Raid
Note there's special raid10 module in recent kernels (and supported by
recent mdadm). I don't think it's very stable yet, but.. JFYI ;)
I didn't see Raid 10 mentioned in the mdadm man page. Do you have the
command line syntax handy? I don't have access to the machine now, but
I believe it's mdadm version 1.7 (from August '04).
quoted
10 array activated at boot isn't working well. I used update-rc.d to
add the symlinks to mdadm-raid using the defaults, but the Raid 10 array
Shouldn't the links be made automatically when installing mdadm?
Also, make sure you're using recent debian package of mdadm --
earlier versions was.. with issues concerning assembling the
arrays (a problem specific to debian mdadm-raid scripts).
Make sure AUTOSTART is set to true in /etc/default/mdadm
(I'm not sure if that's really `AUTOSTART' and `true',
you can look at the file and/or use dpkg-reconfigure mdadm
to set that).
The symlinks on Sarge installation were:
rc0.d --> S50mdadm-raid
rc6.d --> S50mdadm-raid
rcS.d --> S25mdadm-raid
I removed them before running 'update-rc.d'.
quoted
isn't assembled at boot time. After getting kicked to a root shell, if
I check /proc/mdstat only md1 (/) is started. After running mdadm-raid
start, md0 (/boot), md2, and md3 start. If I run mdadm-raid start again
md4 (/opt) starts. Fsck'ing the newly assembled arrays before
successfully issuing 'mount -a' shows no filesystem errors. I'm at a
loss and haven't found any similar issue mentions on this list or the
debian-users list. Here's mdadm.conf:
You have two problems.
First of all, mdadm-raid should be started at very early in the
boot process, and mdadm package post-install scripts ensures this
(you added mdadm-raid links at default order which is 20; but
it should run before filesystem mounts etc, nearly 01 or something).
Ditto for stop scripts -- at the very end, after umounting the
filesystems. Take a look at /var/lib/dpkg/info/mdadm.postinst --
it sets up the links properly. When you correct this, your system
will go further in boot process... ;)
And second problem is the order of lines in mdadm.conf.
[edited a bit]
Re-order the lines so that md4 will be listed AFTER all it's
components. Mdadm tries to assemble them in turn as they're
listed in mdadm.conf. But at the time when it tries to start
md4, it's components aren't here yet so it fails.
Reordering the ARRAY entries has no effect. 'mdadm-raid start' has to
be run twice, same as before. Changing the start symlinks from '20' to
'13' (just after syslogd) doesn't help. (sigh).
From: Andy Smith <hidden> Date: 2005-01-05 22:35:56
On Wed, Jan 05, 2005 at 10:32:37AM -0800, Alvin Oga wrote:
hi ya andy
On Wed, 5 Jan 2005, Andy Smith wrote:
quoted
quoted
quoted
Your recommendations
that'd be your comment ... that is not whati said above
Direct quote:
i'd add more memory before i'd use raid
i see ... when i say "i would use" ... i dont mean or imply
others to do so ..
That is not a very useful thing to say in response to someone who
suggests using RAID mirrors for swap.
quoted
quoted
and all i said, was use memory before you use swap on disks
Which means what? Who is there on this list who likes to use swap
*before* physical memory?
you'd be surprized how many people wonder why their system is slow
and they using 100% swap ( hundreds of MB of swap ) and little memory
beause its $50 of expensive memory
What you said was (and I quoted it for you) "use memory before you
use swap". I am asking you if this also is a useful thing to say,
because I was not aware there was anyone who would prefer to use
swap before memory.
Unless you simply mean "add more memory" which again would be a
strange thing to say in a discussion about putting swap on a RAID
mirror -> implies "always have enough RAM, configure no swap" but
since you seem to have a great objection to me trying to make sense
of what you're saying I won't go any further stating what I think
you imply.
quoted
Do all your machines which you believe have adequate memory also
have no swap configured?
yes .... in my machines .. they are all mostly tuned for specfic
tasks ... very little ( say 0.05% swap usage even at peak )
So "no" then as I asked if your machines had any swap configured and
you reply that they use 0.05% at peak. To use 0.05% they need to
have swap configured.
So your machines do have swap configured, that is what the question
was.
quoted
Fact is if your machine has swap configured then that swap is part
of your virtual memory and if a device that is providing part of
your virtual memory suddenly fails then your machine is going down.
yup...
and in some systems ... having swap ( too slow ) is NOT an option ...
( embedded systems, realtime operations, ... )
That's great but it's irrelevant to this discussion since the
situation discussed is swap on mirror or not. Whether swap is
required or beneficial is an admin issue outside the scope of RAID.
We can assume that the admin ahs already determined that configuring
some amount of swap is required, otherwise we cannot assume anything
about their setup and will be telling them how to lay out their
filesystems, what distribution to use, etc. etc.. There are some
things that just don't need to be said and on a RAID list, "add more
memory, swap is slow" is IMHO one of them.
== you should NEVER have swap problems
== otherwise, toss that system out or salvage what you can
==
If you use RAID-1 (-4, -5,- 10, etc.) then I think you are
acknowledging that your disks are a cause for concern for you.
Otherwise no one would be using RAID. If you would not be willing
to say the following:
you should NEVER have filesystem problems
otherwise, toss that system out or salvage what you can
then I don't understand why you are willing to say it when it comes
to swap and not a filesystem. Disks fail, it's why we're here on
this list.
quoted
You are replying to add more memory and
don'trun things that can get swapped.
you're twisting things gain
All I've got to work with is what you're giving me. Direct quotes
of yours lead me to believe that these are your points. In a
discussion about swap on RAID you have clearly stated to add more
RAM, use higher quality components, not run certain userland
processes. If you say these things in a discussion about swap on
RAID then I'm left to believe either that you see these things as an
alternative to swap on RAID or else you are just posting irrelevant
basic admin tips that have nothing to do with linux RAID.
quoted
Those don't seem like very useful recommendations.
that is becuase you're twisting things so you can make
comments i didnt say
Then please explain yourself better using points that are relevant
to RAID using md on Linux.
quoted
quoted
- and i don't think anybody is idiotic enough to add
more memory for the "spikes" in the workload
OK so you must add swap to handle those spikes which means either
you are happy that your machine will crash should the device that
the swap is on die, or you use swap on a mirror to try to mitigate
that.
this is pointless isnt it ...
are you an idiot or what ..
It's possible to disagree and debate while still being civil. If you
don't have that skill then I'm not willing to spend time teaching
you it.
do you like to put words and recommendations that i didnt say
and twist it in your favor
No, I like understanding your point but at the moment all I can see
is basic unix admin recommendations that are unrelated to the
discussion at hand - you admit you have swap configured on your
servers so all the other stuff you have mentioned is irrelevant. So
help me understand.
I don't understand how you intend to survive a disk failure
involving your swap unless you put it on a mirror or configure no
swap or never have a disk failure, ever, or intend to have downtime
when you do have a disk failure.
So help me understand.
quoted
From previous email (not a direct quote):
Don't run lpd on user-accessible machine
you obviously do NOT understand where to run lpd and where
to run dns and where to run pop and where to run ssh and where
to run mta ... and on and on and on ...
You have no idea whether I do or do not know these things and on
this list you never will, because placement of these things is
irrelevant to linux raid.
quoted
If you do not suffer disk crashes then why do you use mirrors at
all?
i do NOT use mirrors for the protection against disk failures
nobody said i did .. you again are twisting things into your
own ideas and misconceptions
Well indeed, if you never suffer disk failures as you go on to say
then I imagine you don't use mirrors for that..
quoted
If you do suffer disk crashes then any disk that is providing
swap may very well cause the machine to crash.
i dont have that problem ... but if someone ddid have that
problem ... i assuem they are smart enough to figure out
within 5-10 minutes why the disk/system is crashing
Such a person, if they were not as fortunate and/or skiled at
purchasing hardware as you appear to be, and so happened to suffer
the occasional disk failure, may wish that their system did not
crash and suffer 5 to 10 minutes of downtime because a disk failed.
They may wish that the device would just fail and they could
schedule a downtime or hot swap it.
If you are not one of those people (and, since you say you don't
have disk failures then you wouldn't be), it doesn't mean those
people don't exist and don't have a valid requirement.
quoted
I thought everyone suffered disk crashes and that was the point of
RAID.
not everybody suffers disk crashes ... in such great numbers
that raid is better solution ...
- raid is NOT theonly solution
== ever hear of high availability ...
= clusters ...
= even plain ole backups
there are more than one solution to one disk crashing
Is there any reason why someone could not use swap on raid as well
as any of the above as they see fit? Maybe those who advocate swap
on raid already use high availablity and clusters. And we'd
certainly hope they have backups.
Going off topic, but...
I come from an HP-UX environment.
With HP-UX, I have always had to configure swap space.
Maybe only for 1 reason. If you used shared memory, you must have an equal
amount of swap space available. We always used shared memory (shmget). It
was called a "backing store", if I recall. Including Informix, we used
almost the hardware (OS) limit of about 1.7 Gig of shared memory on some
systems. So we needed at least 1.7 Gig of swap space. We would allocate
3-4 Gig I think, just to be safe. The swap space would be allocated
(reserved), but not used, unless you really ran low on RAM.
So, the question is:
Assume I have enough RAM to never need to swap.
Do I need any swap space with Linux?
Silly story:
I once had a 486 with 4 Meg of ram. Running Windows 3.1 I think.
I had 8 Meg of swap space. So 12 meg total available virtual memory.
One day I added 16 Meg of RAM. So now I had 20 Meg of RAM. I deleted my
swap space. Everyone told me I needed 20-40 Meg of swap space now! Swap
space should be 2 time RAM size. How crazy, my memory requirements did not
change, just the amount of memory. I used that system for a year or so like
that. Go figure!
Guy
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Andy Smith
Sent: Wednesday, January 05, 2005 5:36 PM
To: linux-raid@vger.kernel.org
Subject: Re: swp - Re: ext3 journal on software raid (was Re: PROBLEM:
Kernel 2.6.10 crashing repeatedly and hard)
On Wed, Jan 05, 2005 at 10:32:37AM -0800, Alvin Oga wrote:
hi ya andy
On Wed, 5 Jan 2005, Andy Smith wrote:
quoted
quoted
quoted
Your recommendations
that'd be your comment ... that is not whati said above
Direct quote:
i'd add more memory before i'd use raid
i see ... when i say "i would use" ... i dont mean or imply
others to do so ..
That is not a very useful thing to say in response to someone who
suggests using RAID mirrors for swap.
quoted
quoted
and all i said, was use memory before you use swap on disks
Which means what? Who is there on this list who likes to use swap
*before* physical memory?
you'd be surprized how many people wonder why their system is slow
and they using 100% swap ( hundreds of MB of swap ) and little memory
beause its $50 of expensive memory
What you said was (and I quoted it for you) "use memory before you
use swap". I am asking you if this also is a useful thing to say,
because I was not aware there was anyone who would prefer to use
swap before memory.
Unless you simply mean "add more memory" which again would be a
strange thing to say in a discussion about putting swap on a RAID
mirror -> implies "always have enough RAM, configure no swap" but
since you seem to have a great objection to me trying to make sense
of what you're saying I won't go any further stating what I think
you imply.
quoted
Do all your machines which you believe have adequate memory also
have no swap configured?
yes .... in my machines .. they are all mostly tuned for specfic
tasks ... very little ( say 0.05% swap usage even at peak )
So "no" then as I asked if your machines had any swap configured and
you reply that they use 0.05% at peak. To use 0.05% they need to
have swap configured.
So your machines do have swap configured, that is what the question
was.
quoted
Fact is if your machine has swap configured then that swap is part
of your virtual memory and if a device that is providing part of
your virtual memory suddenly fails then your machine is going down.
yup...
and in some systems ... having swap ( too slow ) is NOT an option ...
( embedded systems, realtime operations, ... )
That's great but it's irrelevant to this discussion since the
situation discussed is swap on mirror or not. Whether swap is
required or beneficial is an admin issue outside the scope of RAID.
We can assume that the admin ahs already determined that configuring
some amount of swap is required, otherwise we cannot assume anything
about their setup and will be telling them how to lay out their
filesystems, what distribution to use, etc. etc.. There are some
things that just don't need to be said and on a RAID list, "add more
memory, swap is slow" is IMHO one of them.
== you should NEVER have swap problems
== otherwise, toss that system out or salvage what you can
==
If you use RAID-1 (-4, -5,- 10, etc.) then I think you are
acknowledging that your disks are a cause for concern for you.
Otherwise no one would be using RAID. If you would not be willing
to say the following:
you should NEVER have filesystem problems
otherwise, toss that system out or salvage what you can
then I don't understand why you are willing to say it when it comes
to swap and not a filesystem. Disks fail, it's why we're here on
this list.
quoted
You are replying to add more memory and
don'trun things that can get swapped.
you're twisting things gain
All I've got to work with is what you're giving me. Direct quotes
of yours lead me to believe that these are your points. In a
discussion about swap on RAID you have clearly stated to add more
RAM, use higher quality components, not run certain userland
processes. If you say these things in a discussion about swap on
RAID then I'm left to believe either that you see these things as an
alternative to swap on RAID or else you are just posting irrelevant
basic admin tips that have nothing to do with linux RAID.
quoted
Those don't seem like very useful recommendations.
that is becuase you're twisting things so you can make
comments i didnt say
Then please explain yourself better using points that are relevant
to RAID using md on Linux.
quoted
quoted
- and i don't think anybody is idiotic enough to add
more memory for the "spikes" in the workload
OK so you must add swap to handle those spikes which means either
you are happy that your machine will crash should the device that
the swap is on die, or you use swap on a mirror to try to mitigate
that.
this is pointless isnt it ...
are you an idiot or what ..
It's possible to disagree and debate while still being civil. If you
don't have that skill then I'm not willing to spend time teaching
you it.
do you like to put words and recommendations that i didnt say
and twist it in your favor
No, I like understanding your point but at the moment all I can see
is basic unix admin recommendations that are unrelated to the
discussion at hand - you admit you have swap configured on your
servers so all the other stuff you have mentioned is irrelevant. So
help me understand.
I don't understand how you intend to survive a disk failure
involving your swap unless you put it on a mirror or configure no
swap or never have a disk failure, ever, or intend to have downtime
when you do have a disk failure.
So help me understand.
quoted
From previous email (not a direct quote):
Don't run lpd on user-accessible machine
you obviously do NOT understand where to run lpd and where
to run dns and where to run pop and where to run ssh and where
to run mta ... and on and on and on ...
You have no idea whether I do or do not know these things and on
this list you never will, because placement of these things is
irrelevant to linux raid.
quoted
If you do not suffer disk crashes then why do you use mirrors at
all?
i do NOT use mirrors for the protection against disk failures
nobody said i did .. you again are twisting things into your
own ideas and misconceptions
Well indeed, if you never suffer disk failures as you go on to say
then I imagine you don't use mirrors for that..
quoted
If you do suffer disk crashes then any disk that is providing
swap may very well cause the machine to crash.
i dont have that problem ... but if someone ddid have that
problem ... i assuem they are smart enough to figure out
within 5-10 minutes why the disk/system is crashing
Such a person, if they were not as fortunate and/or skiled at
purchasing hardware as you appear to be, and so happened to suffer
the occasional disk failure, may wish that their system did not
crash and suffer 5 to 10 minutes of downtime because a disk failed.
They may wish that the device would just fail and they could
schedule a downtime or hot swap it.
If you are not one of those people (and, since you say you don't
have disk failures then you wouldn't be), it doesn't mean those
people don't exist and don't have a valid requirement.
quoted
I thought everyone suffered disk crashes and that was the point of
RAID.
not everybody suffers disk crashes ... in such great numbers
that raid is better solution ...
- raid is NOT theonly solution
== ever hear of high availability ...
= clusters ...
= even plain ole backups
there are more than one solution to one disk crashing
Is there any reason why someone could not use swap on raid as well
as any of the above as they see fit? Maybe those who advocate swap
on raid already use high availablity and clusters. And we'd
certainly hope they have backups.
From: Mike Hardy <hidden> Date: 2005-01-06 01:28:00
Guy wrote:
So, the question is:
Assume I have enough RAM to never need to swap.
Do I need any swap space with Linux?
This has been hashed out at great length on linux-kernel - with a few
entrenched positions emerging, if I recall correctly.
There are those that think if they size the machine correctly, they
shouldn't need swap, and they're right.
I had 8 Meg of swap space. So 12 meg total available virtual memory.
One day I added 16 Meg of RAM. So now I had 20 Meg of RAM. I deleted my
swap space. Everyone told me I needed 20-40 Meg of swap space now! Swap
space should be 2 time RAM size. How crazy, my memory requirements did not
change, just the amount of memory. I used that system for a year or so like
that. Go figure!
This story pretty much sums up that position, and I've certainly been in
that position myself.
There are others (notably the maintainer of part of that code - Andrea
Arcangeli if I recall correctly, though I apologize if that's a
misattribution) who believe you should always have swap because it will
allow your system to have higher throughput. If you process a large I/O,
for instance, the kernel can swap out live processes to devote more RAM
to VFS caching. That hurts latency though, and the nightly slocate run
is a pathlogical example of this. You wake up in the morning and your
machine is crawling while it swaps everything back in.
There's a "swappiness" knob you can twiddle in /proc or /sys to alter
this, but the general idea is that you can improve throughput on a
machine by having swap even if there is enough ram for all processes
I've got machines with and without swap, but I typically run all servers
with swap to handle ram-usage spikes I'm not expecting (you never know)
while I run my laptop without swap when possible to avoid latency issues
with swappiness.
As with all things, its policy decisions and tradeoffs :-)
-Mike
Ok, good answer. Thanks. I understand the "spikes" issue. I really do
plan to have swap space all the time, but good to know it is optional if you
have enough RAM.
Maybe the concept of swapping is becoming obsolete?
In the past, some systems had hardware limits. Like 64K, 640K, 16M, 64M.
You had no choice but to swap stuff out. Or use overlays. I have worked on
systems where the app had to swap chunks of memory to disk and back. There
was no OS support for virtual memory. We had fixed head disks to give
better performance. I even recall hardware that could bank switch sections
of memory to break the 640K limit! Even IBM mainframes had a 256Meg limit,
but could go beyond that by bank switching. But now hardware has caught up
to programmers. Most systems can support more RAM than most programs need.
With PC based systems (and some others I assume), 1 Gig of RAM is very
cheap! But I did note someone said you had to do something to break the
2Gig boundary in Linux with x68 based systems. That's too bad. In a few
years, most people will be in the 64 bit world I guess. I hope that
corrects the 2Gig boundary. The next boundary (8,589,934,592 Gig) should be
well beyond anything we would ever need, well, for at least a few more
years! :) But then we could go un-signed and double that! Of course,
disks will be even bigger! I can't wait!
Of course you can't put 8,589,934,592 Gig of RAM in a 64 bit computer today.
If you consider Moore's Law and RAM. "transistors on integrated circuits
tend to double every 18 to 24 months" Assuming we are at a about 36 bits
today (I think 64 Gig is about as much as is reasonable today). We should
reach the 64 bit limit in 42-56 years. Maybe it will be a long wait? :)
Guy
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Mike Hardy
Sent: Wednesday, January 05, 2005 8:28 PM
To: linux-raid@vger.kernel.org
Subject: Re: swp - Re: ext3 journal on software raid (was Re: PROBLEM:
Kernel 2.6.10 crashing repeatedly and hard)
Guy wrote:
So, the question is:
Assume I have enough RAM to never need to swap.
Do I need any swap space with Linux?
This has been hashed out at great length on linux-kernel - with a few
entrenched positions emerging, if I recall correctly.
There are those that think if they size the machine correctly, they
shouldn't need swap, and they're right.
I had 8 Meg of swap space. So 12 meg total available virtual memory.
One day I added 16 Meg of RAM. So now I had 20 Meg of RAM. I deleted my
swap space. Everyone told me I needed 20-40 Meg of swap space now! Swap
space should be 2 time RAM size. How crazy, my memory requirements did
not
change, just the amount of memory. I used that system for a year or so
like
that. Go figure!
This story pretty much sums up that position, and I've certainly been in
that position myself.
There are others (notably the maintainer of part of that code - Andrea
Arcangeli if I recall correctly, though I apologize if that's a
misattribution) who believe you should always have swap because it will
allow your system to have higher throughput. If you process a large I/O,
for instance, the kernel can swap out live processes to devote more RAM
to VFS caching. That hurts latency though, and the nightly slocate run
is a pathlogical example of this. You wake up in the morning and your
machine is crawling while it swaps everything back in.
There's a "swappiness" knob you can twiddle in /proc or /sys to alter
this, but the general idea is that you can improve throughput on a
machine by having swap even if there is enough ram for all processes
I've got machines with and without swap, but I typically run all servers
with swap to handle ram-usage spikes I'm not expecting (you never know)
while I run my laptop without swap when possible to avoid latency issues
with swappiness.
As with all things, its policy decisions and tradeoffs :-)
-Mike
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Mike Hardy <hidden> Date: 2005-01-06 04:49:02
Guy wrote:
Maybe the concept of swapping is becoming obsolete?
I think its definitely headed that way, if not already there for most
new systems
to programmers. Most systems can support more RAM than most programs need.
With PC based systems (and some others I assume), 1 Gig of RAM is very
cheap! But I did note someone said you had to do something to break the
2Gig boundary in Linux with x68 based systems. That's too bad. In a few
To go bigger than 2GB on x86 you have to enable PAE (pointer address
extensions? I forget the TLA expansion) which does have a speed hit and
so is sub-optimal. Additionally, you still have limits on the amount of
RAM any process can address and its more policy decisions and tradeoffs.
Even with 64-bit, in my professional world (Java programming) I've had
occasion to want 4GB heaps to work with for caching purposes, and
programs are only just getting there. I think you're right then, the
hardware and software have just met each other in the last year or so
except for the edge cases.
Bandwidth is the usual limiter at this point :-)
today (I think 64 Gig is about as much as is reasonable today). We should
reach the 64 bit limit in 42-56 years. Maybe it will be a long wait? :)
:-)
To make this more on topic, I'll say that I just switched all my servers
(which had raid1 /boot and / but parallel JBOD swap) to mirrored swap
after this discussion. All using simple mdadm/swapon/swapoff commands
while they were humming, thanks to all the author's work. The servers
are all doing fine, post-change.
As further penance for taking up everyone's time, I'll point to the
specific section of the software raid howto that discusses swapping on raid:
http://www.tldp.org/HOWTO/Software-RAID-HOWTO-2.html#ss2.3
They succinctly say what was posted here - which is that you don't swap
on raid for performance, but if you're looking for single-hdd machine
survivability, swapping on raid1 is the only way.
Cheers-
-Mike
There are those that think if they size the machine correctly, they
shouldn't need swap, and they're right.
bingo
quoted
I had 8 Meg of swap space. So 12 meg total available virtual memory.
One day I added 16 Meg of RAM. So now I had 20 Meg of RAM. I deleted my
swap space. Everyone told me I needed 20-40 Meg of swap space now! Swap
space should be 2 time RAM size. How crazy, my memory requirements did not
change, just the amount of memory. I used that system for a year or so like
that. Go figure!
the silly rule of 2x size of RAM == swap space came from the old days
when memory was 10x the costs of disks or some silly cost performance
that it made sense when grandpa was floating around
by todays ram and disk pricing ... and cpu speeds ...2x memory sorta
goes out the door
c ya
alvin
On Wed, Jan 05, 2005 at 05:37:34PM +0300, Michael Tokarev wrote:
quoted
Erik Mouw wrote:
quoted
If you have a bad block in your swap partition and the device doesn't
report an error about it, no amount of RAID is going to help you
against it.
The drive IS reporting read errors in most cases.
"most cases" and "all cases" makes quite a difference.
Actually it *was* reporting *all* read errors. It was an early Maxtor 1GB drive and these have been
notoriuosly bad for being problematic *however* they have been frightfully good at accurately
reporting exactly what was wrong. In this case I was pretty new to the Linux sysadmin thing and
never actually really noticed the disk errors in the syslog and correlated them to the process
dying. (it was a fair few years ago now).
I actually have never had an ATA disk develop errors it did not report.
My point remains the same. By putting your swap on a RAID (of any redundant variety) you are
increasing the chances of machine survival against disk errors, be they single bit, bad block or
dead drive.
Talking of Maxtor drives. I have a unit here with less than 6000 hours on it that has started
growing bad sectors at an alarming rate. All accurately reported by SMART mind you (clever little
disk), but after running a badblocks -n on it (to really shake them loose) the reallocated sector
count has halved! Now how can a drive un-reallocate dud sectors?
Brad
HAY! DON'T CALL ME GRANDPA!!!! Not for a few more years at least!
:)
And I was there! And beyond! My first home computer had a 4K card and an
8K card, if I recall. It was 6800 based. I wonder if I still have it? I
did use punch cards in college! Some computers had core memory, but no tube
based computers. I am not that old!!!
In the days of my 486-33, I paid $400 for 16Meg of RAM. The price doubled
just after that when the only epoxy plant burned down or something. Now my
video card has 64Meg of RAM! My first hard disk to break the $1 per meg
boundary was a Maxtor 340Meg, cost me about $320. Now we are under the $1
per Gig boundary. Today, disk drives are so cheap you get 1 in a happy
meal. :) Today, I guess disk space is 50-100 times cheaper per gig than
RAM.
Ouch! At the prices listed above, 1 gig of RAM would cost $25,000, and a
250Gig disk would cost $235,294. The price of RAM has dropped by 250 times.
The price of disk drives dropped by over 1000 times. The future is going to
be so cool!
You said memory was 10x the cost of disks. In the example above, memory is
25x more than disk. Today it is about 100x. Maybe we should be swapping
even more, now? :)
Guy
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Alvin Oga
Sent: Thursday, January 06, 2005 12:04 AM
To: Mike Hardy
Cc: linux-raid@vger.kernel.org
Subject: Re: swp - Re: ext3 journal on software raid (was Re: PROBLEM:
Kernel 2.6.10 crashing repeatedly and hard)
On Wed, 5 Jan 2005, Mike Hardy wrote:
There are those that think if they size the machine correctly, they
shouldn't need swap, and they're right.
bingo
quoted
I had 8 Meg of swap space. So 12 meg total available virtual memory.
One day I added 16 Meg of RAM. So now I had 20 Meg of RAM. I deleted
my
quoted
swap space. Everyone told me I needed 20-40 Meg of swap space now!
Swap
quoted
space should be 2 time RAM size. How crazy, my memory requirements did
not
quoted
change, just the amount of memory. I used that system for a year or so
like
quoted
that. Go figure!
the silly rule of 2x size of RAM == swap space came from the old days
when memory was 10x the costs of disks or some silly cost performance
that it made sense when grandpa was floating around
by todays ram and disk pricing ... and cpu speeds ...2x memory sorta
goes out the door
c ya
alvin
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
HAY! DON'T CALL ME GRANDPA!!!! Not for a few more years at least!
wasn't calling ya grandpa :-)
And I was there! And beyond! My first home computer had a 4K card and an
8K card, if I recall. It was 6800 based. I wonder if I still have it? I
ahh .. that's 3rd generation stuff ...
4004 and Z8 and lots o no-namebrand what-you-ma-call-it-kits
and i forgot what predated 6800 in motorola/zilog besides z8 ..
getting too old
You said memory was 10x the cost of disks. In the example above, memory is
25x more than disk. Today it is about 100x.
500x - 1000x in my book but that depends on how one counts
Maybe we should be swapping even more, now? :)
disks are cheaper per byte of storage ???
500MB for $50
250,000MB for $250 ( 250GB disks )
i leave it to you math wiz folks to generate $$$/MB
c ya
alvin
From: Andy Smith <hidden> Date: 2005-01-06 09:38:11
On Wed, Jan 05, 2005 at 09:04:00PM -0800, Alvin Oga wrote:
On Wed, 5 Jan 2005, Mike Hardy wrote:
quoted
There are those that think if they size the machine correctly, they
shouldn't need swap, and they're right.
bingo
Yet in an earlier reply you state that your machines have swap
configured.
There is a difference between these two statements:
a) I need some amount of swap configured but don't expect a
significant swap usage under normal load.
b) I expect my server to always be using a significant
amount of swap.
I interpret your views as (a) but I interpret Mike's as "there are
people who are saying that a correctly sized machine can have zero
swap configured."
Having no swap configured and merely using no swap in normal
circumstances are very very different situations.
From: Mike Hardy <hidden> Date: 2005-01-06 17:46:55
Andy Smith wrote:
I interpret your views as (a) but I interpret Mike's as "there are
people who are saying that a correctly sized machine can have zero
swap configured."
Having no swap configured and merely using no swap in normal
circumstances are very very different situations.
You are correct that I was getting at the zero swap argument - and I
agree that it is vastly different from simply not expecting it. It is
important to know that there is no inherent need for swap in the kernel
though - it is simply used as more "memory" (albeit slower, and with
some optimizations to work better with real memory) and if you don't
need it, you don't need it.
That said, I mentioned my servers run with swap for the same reason I
run with raid. I don't plan on having a disk very often (if at all), and
I don't plan on needing swap very often (if at all), but when it
happens, I expect my machine to keep running. (or at least I hope it does)
-Mike
From: Andrew Walrond <hidden> Date: 2005-01-06 22:09:34
On Thursday 06 January 2005 17:46, Mike Hardy wrote:
You are correct that I was getting at the zero swap argument - and I
agree that it is vastly different from simply not expecting it. It is
important to know that there is no inherent need for swap in the kernel
though - it is simply used as more "memory" (albeit slower, and with
some optimizations to work better with real memory) and if you don't
need it, you don't need it.
If I recollect a recent thread on LKML correctly, your 'no inherent need for
swap' might be wrong.
I think the gist was this: the kernel can sometimes needs to move bits of
memory in order to free up dma-able ram, or lowmem. If I recall correctly,
the kernel can only do this move via swap, even if there is stacks of free
(non-dmaable or highmem) memory.
I distinctly remember the moral of the thread being "Always mount some swap,
if you can"
This might have changed though, or I might have got it completely wrong. -
I've cc'ed LKML incase somebody more knowledgeable can comment...
Andrew Walrond
On Thursday 06 January 2005 17:46, Mike Hardy wrote:
quoted
You are correct that I was getting at the zero swap argument - and I
agree that it is vastly different from simply not expecting it. It is
important to know that there is no inherent need for swap in the kernel
though - it is simply used as more "memory" (albeit slower, and with
some optimizations to work better with real memory) and if you don't
need it, you don't need it.
If I recollect a recent thread on LKML correctly, your 'no inherent need for
swap' might be wrong.
I think the gist was this: the kernel can sometimes needs to move bits of
memory in order to free up dma-able ram, or lowmem. If I recall correctly,
the kernel can only do this move via swap, even if there is stacks of free
(non-dmaable or highmem) memory.
I distinctly remember the moral of the thread being "Always mount some swap,
if you can"
This might have changed though, or I might have got it completely wrong. -
I've cc'ed LKML incase somebody more knowledgeable can comment...
From: Mike Hardy <hidden> Date: 2005-01-06 23:03:51
Jesper Juhl wrote:
On Thu, 6 Jan 2005, Andrew Walrond wrote:
quoted
On Thursday 06 January 2005 17:46, Mike Hardy wrote:
quoted
You are correct that I was getting at the zero swap argument - and I
agree that it is vastly different from simply not expecting it. It is
important to know that there is no inherent need for swap in the kernel
though - it is simply used as more "memory" (albeit slower, and with
some optimizations to work better with real memory) and if you don't
need it, you don't need it.
If I recollect a recent thread on LKML correctly, your 'no inherent need for
swap' might be wrong.
I think the gist was this: the kernel can sometimes needs to move bits of
memory in order to free up dma-able ram, or lowmem. If I recall correctly,
the kernel can only do this move via swap, even if there is stacks of free
(non-dmaable or highmem) memory.
I distinctly remember the moral of the thread being "Always mount some swap,
if you can"
This might have changed though, or I might have got it completely wrong. -
I've cc'ed LKML incase somebody more knowledgeable can comment...
Interesting - I was familiar with the original swappiness thread
(http://kerneltrap.org/node/view/3000) but haven't seen anything since
then (I mainly follow via kernel-traffic - enjoyable, but nowhere near
real time). There's clearly been a bunch more discussion...
Not to rehash the performance arguments, but it appears from my read of
the kernel trap page referenced above that the primary argument for swap
is still the performance argument - I didn't see anything referencing
swap being necessary to move DMAable ram or lowmem. Was that posted
previously on linux-kernel but not on kerneltrap?
I'm still under the impression that "to swap or not" is a
performance/policy/risk-management question, not a correctness question.
If I'm wrong, I'd definitely like to know...
-Mike
If I MUST/SHOULD have swap space....
Maybe I will create a RAM disk and use it for swap! :) :) :)
Guy
-----Original Message-----
From: linux-raid-owner@vger.kernel.org
[mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Mike Hardy
Sent: Thursday, January 06, 2005 5:58 PM
To: Jesper Juhl
Cc: Andrew Walrond; linux-raid@vger.kernel.org; linux-kernel@vger.kernel.org
Subject: Re: No swap can be dangerous (was Re: swap on RAID (was Re: swp -
Re: ext3 journal on software raid))
Jesper Juhl wrote:
On Thu, 6 Jan 2005, Andrew Walrond wrote:
quoted
On Thursday 06 January 2005 17:46, Mike Hardy wrote:
quoted
You are correct that I was getting at the zero swap argument - and I
agree that it is vastly different from simply not expecting it. It is
important to know that there is no inherent need for swap in the kernel
though - it is simply used as more "memory" (albeit slower, and with
some optimizations to work better with real memory) and if you don't
need it, you don't need it.
If I recollect a recent thread on LKML correctly, your 'no inherent need
for
quoted
swap' might be wrong.
I think the gist was this: the kernel can sometimes needs to move bits of
memory in order to free up dma-able ram, or lowmem. If I recall correctly,
quoted
the kernel can only do this move via swap, even if there is stacks of free
quoted
(non-dmaable or highmem) memory.
I distinctly remember the moral of the thread being "Always mount some
swap,
quoted
if you can"
This might have changed though, or I might have got it completely wrong. -
quoted
I've cc'ed LKML incase somebody more knowledgeable can comment...
Interesting - I was familiar with the original swappiness thread
(http://kerneltrap.org/node/view/3000) but haven't seen anything since
then (I mainly follow via kernel-traffic - enjoyable, but nowhere near
real time). There's clearly been a bunch more discussion...
Not to rehash the performance arguments, but it appears from my read of
the kernel trap page referenced above that the primary argument for swap
is still the performance argument - I didn't see anything referencing
swap being necessary to move DMAable ram or lowmem. Was that posted
previously on linux-kernel but not on kerneltrap?
I'm still under the impression that "to swap or not" is a
performance/policy/risk-management question, not a correctness question.
If I'm wrong, I'd definitely like to know...
-Mike
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jan 05, 2005 at 09:04:00PM -0800, Alvin Oga wrote:
quoted
On Wed, 5 Jan 2005, Mike Hardy wrote:
quoted
There are those that think if they size the machine correctly, they
shouldn't need swap, and they're right.
bingo
Yet in an earlier reply you state that your machines have swap
configured.
There is a difference between these two statements:
since you're implying/saying i said those two statements
below ( a and b ):
you are one totally confused dude ... sorry to say ...
i never said either of those two statements ... that is yur words
and you're understanding of some mixture of lots of comments
== grep "under normal load" sent-mail raid ==
( pick any set of words of your "quote" for grep )
- it's not even in anybody comments posted, since i save
all of my posts ( in sent-mail ) and not in anybody elses
replies that is saved here
( thus my comment ... you're confused .. )
please refrain from making (re)quotes .. i didn't say so that
i don't have to reply
please do try to use: instead of "implying" i said something
as in your "there is two different statments" in response to me
which i didn't say either statments you're "quoting" incorrectly
"from what i gather, i understand that ... "
"i think this is ..."
.. blah blah ..
a) I need some amount of swap configured but don't expect a
significant swap usage under normal load.
b) I expect my server to always be using a significant
amount of swap.
i personally do not expect any system i'm involved with to use any
swap and if it does, i'd be adding more memory, as soon as they
complain its running too slow when xxx or yyy job is running
- they have choices of what they want done about it
and it is NOT the same thing ... "not using swap" vs not creating one
- i don't use swap and i rather the system not use it ...
but i do create an itty bitty 250MB of swap partition
even if there's is 2GB of system memory
- in other cases ... like embedded systems ...
it has no swap files .. no swap partitions adn things
works just fine
another example might be cell phones .. does those
puppies have swap space ?? ( probably not )
c ya
alvin
From: Andy Smith <hidden> Date: 2005-01-07 02:28:42
On Thu, Jan 06, 2005 at 05:31:48PM -0800, Alvin Oga wrote:
On Thu, 6 Jan 2005, Andy Smith wrote:
quoted
On Wed, Jan 05, 2005 at 09:04:00PM -0800, Alvin Oga wrote:
quoted
On Wed, 5 Jan 2005, Mike Hardy wrote:
quoted
There are those that think if they size the machine correctly, they
shouldn't need swap, and they're right.
bingo
Yet in an earlier reply you state that your machines have swap
configured.
There is a difference between these two statements:
since you're implying/saying i said those two statements
below ( a and b ):
No I was not implying that you said either of them. Why is this hard
to understand? Look at Mike's response to me; he understood what I
was getting at perfectly. I was only trying to point out to you
that what you've said isn't consistent with things you said in other
posts.
you are one totally confused dude ... sorry to say ...
Look. It is quite simple.
Mike above (which I leave quoted so you believe me) said:
There are those that think if they size the machine
correctly, they shouldn't need swap, and they're right.
To which *you* replied:
bingo
*However* in an earlier mail in this thread you said that your
machines do have swap configured. Mike has already clarified that
he meant that these people say that no swap needs to be configured.
So my point was that while you say "bingo" to Mike, you do not in
reality do as Mike describes.
i never said either of those two statements ... that is yur words
and you're understanding of some mixture of lots of comments
They were indeed my words and I never claimed otherwise.
== grep "under normal load" sent-mail raid ==
( pick any set of words of your "quote" for grep )
- it's not even in anybody comments posted, since i save
all of my posts ( in sent-mail ) and not in anybody elses
replies that is saved here
( thus my comment ... you're confused .. )
please refrain from making (re)quotes .. i didn't say so that
i don't have to reply
I didn't attribute any words to you. I quite clearly said
they were my interpretation. A direct quote:
There is a difference between these two statements:
a) I need some amount of swap configured but don't
expect a significant swap usage under normal load.
b) I expect my server to always be using a
significant amount of swap.
I interpret your views as (a) but I interpret Mike's as
"there are people who are saying that a correctly sized
machine can have zero swap configured."
You now go on to say:
i personally do not expect any system i'm involved with to use any
swap and if it does, i'd be adding more memory, as soon as they
complain its running too slow when xxx or yyy job is running
- they have choices of what they want done about it
Which appears to me to be pretty much exactly what (a) above says.
So I'm at a loss to understand why you bothered to write this email
complaining about being misrepresented.
Now that we have in one email original unedited quotes of you saying
you agree with someone who suggests configuring no swap, and then
quotes of you saying you do configure swap, I hope you will now
concede my point that you are being inconsistent. If you don't
agree, fine, I'm confident that what's been said speaks for itself
and don't feel like pursuing the matter further. Especially if it's
going to result in you telling ME that I "must be an idiot" and am
"confused."
and it is NOT the same thing ... "not using swap" vs not creating one
Yes, exactly my point from two emails ago.
- i don't use swap and i rather the system not use it ...
but i do create an itty bitty 250MB of swap partition
even if there's is 2GB of system memory
So you do configure swap, this is now my point from 3 emails ago.
Why it has taken 3 *long* emails to get you to answer one simple question
and concede a blatant inconsistency in your argument is beyond me.
From: Clemens Schwaighofer <hidden> Date: 2005-01-07 06:21:08
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 12/31/2004 02:39 AM, Peter T. Breuer wrote:
In gmane.linux.raid Georg C. F. Greve [off-list ref] wrote:
quoted
The message I saw on the remote console when it crashed with pure ext3
on raid5 was:
Assertion failure in journal_start() at fs/jbd/transaction.c:271: "handle->h_transaction->t_journal == journal"
Yes, well, don't put the journal on the raid partition. Put it
elsewhere (anyway, journalling and raid do not mix, as write ordering
is not - deliberately - preserved in raid, as far as I can tell).
Thats a very new claim. I never ever heard of that. I have a lot of
boxes running with software raid (1 or 5) and they run either XFS or
ext3 on it, and since one year I never had a single problem (and they
all use 2.6.7 or 2.6.8.1 kernels).
- --
[ Clemens Schwaighofer -----=====:::::~ ]
[ TBWA\ && TEQUILA\ Japan IT Group ]
[ 6-17-2 Ginza Chuo-ku, Tokyo 104-0061, JAPAN ]
[ Tel: +81-(0)3-3545-7703 Fax: +81-(0)3-3545-7343 ]
[ http://www.tequila.co.jphttp://www.tbwajapan.co.jp ]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFB3ipTjBz/yQjBxz8RAsExAJ9ZhniY/vsm9TIpRtHhGsfdVjws6wCeO4RS
rJ/o4roBjSBo3z5os/E6tKk=
=d3Rn
-----END PGP SIGNATURE-----
From: Andrew Walrond <hidden> Date: 2005-01-07 09:28:38
On Thursday 06 January 2005 23:15, Guy wrote:
If I MUST/SHOULD have swap space....
Maybe I will create a RAM disk and use it for swap! :) :) :)
Well, indeed, I had the same thought. As long as you could guarantee that the
ram was of the highmem/non-dmaable type...
But we're getting ahead of ourselves. I think we need an authoritive answer to
the original premise. Perhaps Alan (cc-ed) might spare us a moment?
Did I dream this up, or is it correct?
"I think the gist was this: the kernel can sometimes needs to move bits of
memory in order to free up dma-able ram, or lowmem. If I recall correctly,
the kernel can only do this move via swap, even if there is stacks of free
(non-dmaable or highmem) memory."
Andrew