raid10n2/xfs setup guidance on write-cache/barrier

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it.
The filesystem will be used as NFS/Samba storage.

mdadm --zero /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm -v --create /dev/md0 --metadata=1.2 --assume-clean
--level=raid10 --chunk 256 --raid-devices=3 /dev/sdb1 /dev/sdc1
/dev/sdd1
mkfs -t xfs -l lazy-count=1,size=128m -f /dev/md0
mount -t xfs -o barrier=1,logbsize=256k,logbufs=8,noatime /dev/md0
/mnt/raid10xfs

Will my files be safe even on sudden power loss? Is barrier=1 enough?
Do i need to disable the write cache?
with: hdparm -W0 /dev/sdb /dev/sdc /dev/sdd

I tried it but performance is horrendous.

Am I better of with ext4? Data safety/integrity is the priority and
optimization affecting it is not acceptable.

Thanks and any advice/guidance would be appreciated

On 3/14/2012 7:30 PM, Jessie Evangelista wrote:

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it.
The filesystem will be used as NFS/Samba storage.

mdadm --zero /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm -v --create /dev/md0 --metadata=1.2 --assume-clean
--level=raid10 --chunk 256 --raid-devices=3 /dev/sdb1 /dev/sdc1
/dev/sdd1

Why 256KB for chunk size?


Looks like you've been reading a very outdated/inaccurate "XFS guide" on
the web...

What kernel version?  This can make a significant difference in XFS
metadata performance.  You should use 2.6.39+ if possible.  What
xfsprogs version?

mkfs -t xfs -l lazy-count=1,size=128m -f /dev/md0

lazy-count=1 is currently the default with recent xfsprogs so no need to
specify it.  Why are you manually specifying the size of the internal
journal log file?  This is unnecessary.  In fact, unless you have
profiled your workload and testing shows that alternate XFS settings
perform better, it is always best to stick with the defaults.  They
exist for a reason, and are well considered.

mount -t xfs -o barrier=1,logbsize=256k,logbufs=8,noatime /dev/md0
/mnt/raid10xfs

Barrier has no value, it's either on or off.  XFS mounts with barriers
enabled by default so remove 'barrier=1'.  You do not have a RAID card
with persistent write cache (BBWC), so you should leave barriers
enabled.  Barriers mitigate journal log corruption due to power failure
and crashes, which seem seem to be of concern to you.

logbsize=256k and logbufs=8 are the defaults in recent kernels so no
need to specify them.  Your NFS/Samba workload on 3 slow disks isn't
sufficient to need that much in memory journal buffer space anyway.  XFS
uses relatime which is equivalent to noatime WRT IO reduction
performance, so don't specify 'noatime'.

In fact, it appears you don't need to specify anything in mkfs.xfs or
fstab, but just use the defaults.  Fancy that.  And the one thing that
might actually increase your performance a little bit you didn't
specify--sunit/swidth.  However, since you're using mdraid, mkfs.xfs
will calculate these for you (which is nice as mdraid10 with odd disk
count can be a tricky calculation).  Again, defaults work for a reason.

Will my files be safe even on sudden power loss?

Are you unwilling to purchase a UPS and implement shutdown scripts?  If
so you have no business running a server, frankly.  Any system will lose
data due to power loss, it's just a matter of how much based on the
quantity of inflight writes at the time the juice dies.  This problem is
mostly filesytem independent.  Application write behavior does play a
role.  UPS with shutdown scripts, and persistent write cache prevent
this problem.  A cheap UPS suitable for this purpose is less money than
a 1TB 7.2k drive, currently.

You say this is an NFS/Samba server.  That would imply that multiple
people or other systems directly rely on it.  Implement a good UPS
solution and eliminate this potential problem.

Is barrier=1 enough?
Do i need to disable the write cache?
with: hdparm -W0 /dev/sdb /dev/sdc /dev/sdd

Disabling drive write caches does decrease the likelihood of data loss.

I tried it but performance is horrendous.

And this is why you should leave them enabled and use barriers.  Better
yet, use a RAID card with BBWC and disable the drive caches.

Am I better of with ext4? Data safety/integrity is the priority and
optimization affecting it is not acceptable.

You're better off using a UPS.  Filesystem makes little difference WRT
data safety/integrity.  All will suffer some damage if you throw a
grenade at them.  So don't throw grenades.  Speaking of which, what is
your backup/restore procedure/hardware for this array?

Thanks and any advice/guidance would be appreciated

I'll appreciate your response stating "Yes, I have a UPS and
tested/working shutdown scripts" or "I'll be implementing a UPS very
soon." :)

-- 
Stan

On Thu, Mar 15, 2012 at 1:38 PM, Stan Hoeppner [off-list ref] wrote:

On 3/14/2012 7:30 PM, Jessie Evangelista wrote:

quoted

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it.
The filesystem will be used as NFS/Samba storage.

mdadm --zero /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm -v --create /dev/md0 --metadata=1.2 --assume-clean
--level=raid10 --chunk 256 --raid-devices=3 /dev/sdb1 /dev/sdc1
/dev/sdd1

Why 256KB for chunk size?

For reference, the machine has 16GB memory

I've run some benchmarks with dd trying the different chunks and 256k
seems like the sweetspot.
dd if=/dev/zero of=/dev/md0 bs=64k count=655360 oflag=direct

Looks like you've been reading a very outdated/inaccurate "XFS guide" on
the web...

What kernel version?  This can make a significant difference in XFS
metadata performance.  You should use 2.6.39+ if possible.  What
xfsprogs version?

testing was done with ubuntu 10.04LTS with kernel at 2.6.32-33-server
xfsprogs at 3.1.0ubuntu1

quoted

mkfs -t xfs -l lazy-count=1,size=128m -f /dev/md0

lazy-count=1 is currently the default with recent xfsprogs so no need to
specify it.  Why are you manually specifying the size of the internal
journal log file?  This is unnecessary.  In fact, unless you have
profiled your workload and testing shows that alternate XFS settings
perform better, it is always best to stick with the defaults.  They
exist for a reason, and are well considered.

I'll probably forgo setting the journal log file size. It seemed like
a safe optimization from what I've read.

quoted

mount -t xfs -o barrier=1,logbsize=256k,logbufs=8,noatime /dev/md0
/mnt/raid10xfs

Barrier has no value, it's either on or off.  XFS mounts with barriers
enabled by default so remove 'barrier=1'.  You do not have a RAID card
with persistent write cache (BBWC), so you should leave barriers
enabled.  Barriers mitigate journal log corruption due to power failure
and crashes, which seem seem to be of concern to you.

logbsize=256k and logbufs=8 are the defaults in recent kernels so no
need to specify them.  Your NFS/Samba workload on 3 slow disks isn't
sufficient to need that much in memory journal buffer space anyway.  XFS
uses relatime which is equivalent to noatime WRT IO reduction
performance, so don't specify 'noatime'.

I just wanted to be explicit about it so that I know what is set just
in case the defaults change

In fact, it appears you don't need to specify anything in mkfs.xfs or
fstab, but just use the defaults.  Fancy that.  And the one thing that
might actually increase your performance a little bit you didn't
specify--sunit/swidth.  However, since you're using mdraid, mkfs.xfs
will calculate these for you (which is nice as mdraid10 with odd disk
count can be a tricky calculation).  Again, defaults work for a reason.

The reason I did not set sunit/swidth is because I read somewhere that
mkfs.xfs will calculate based on mdraid.

quoted

Will my files be safe even on sudden power loss?

Are you unwilling to purchase a UPS and implement shutdown scripts?  If
so you have no business running a server, frankly.  Any system will lose
data due to power loss, it's just a matter of how much based on the
quantity of inflight writes at the time the juice dies.  This problem is
mostly filesytem independent.  Application write behavior does play a
role.  UPS with shutdown scripts, and persistent write cache prevent
this problem.  A cheap UPS suitable for this purpose is less money than
a 1TB 7.2k drive, currently.

The server is for a non-profit org that I am helping out.
I think a APC Smart-UPS SC 420VA 230V may fit their shoe string budget.

You say this is an NFS/Samba server.  That would imply that multiple
people or other systems directly rely on it.  Implement a good UPS
solution and eliminate this potential problem.

quoted

Is barrier=1 enough?
Do i need to disable the write cache?
with: hdparm -W0 /dev/sdb /dev/sdc /dev/sdd

Disabling drive write caches does decrease the likelihood of data loss.

quoted

I tried it but performance is horrendous.

And this is why you should leave them enabled and use barriers.  Better
yet, use a RAID card with BBWC and disable the drive caches.

Budget does not allow for RAID card with BBWC

quoted

Am I better of with ext4? Data safety/integrity is the priority and
optimization affecting it is not acceptable.

You're better off using a UPS.  Filesystem makes little difference WRT
data safety/integrity.  All will suffer some damage if you throw a
grenade at them.  So don't throw grenades.  Speaking of which, what is
your backup/restore procedure/hardware for this array?

nightly backups will be stored on an external USB disk
is xfs going to be prone to more data loss in case the non-redundant
power supply goes out?

quoted

Thanks and any advice/guidance would be appreciated

I'll appreciate your response stating "Yes, I have a UPS and
tested/working shutdown scripts" or "I'll be implementing a UPS very
soon." :)

I don't have shutdown scripts yet but will look into it.
Meatware would have to do for now as the server will probably be ON
only when there's people at the office. And yes I will be asking them
to not go into production without a UPS

--
Stan

Thanks for you input Stan.

I just updated the kernel to 3.0.0-16.
Did they take out barrier support in mdraid? or was the implementation
replaced with FUA?
Is there a definitive test to determine if the off the shelf consumer
sata drives honor barrier or cache flush requests?

I think I'd like to go with device cache turned ON and barrier enabled.

Am still torn between ext4 and xfs i.e. which will be safer in this
particular setup.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

quoted

quoted

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it. The
filesystem will be used as NFS/Samba storage.

Consider also an 'o2' layout (it is probably the same thing for a
3 drive RAID10) or even a RAID5, as 3 drives and this usage seems
one of the few cases where RAID5 may be plausible.

[ ... ] I've run some benchmarks with dd trying the different
chunks and 256k seems like the sweetspot.  dd if=/dev/zero
of=/dev/md0 bs=64k count=655360 oflag=direct

That's for bulk sequential transfers. Random-ish, as in a
fileserver perhaps with many smaller files, may not be the same,
but probably larger chunks are good.

quoted

[ ... ] What kernel version?  This can make a significant
difference in XFS metadata performance.

As an aside, that's a myth that has been propagandaized by DaveC
in his entertaining presentation not long ago.

There have been decent but no major improvements in XFS metadata
*performance*, but weaker implicit *semantics* have been made an
option, and these have a different safety/performance tradeoff
(less implicit safety, somewhat more performance), not "just"
better performance.

http://lwn.net/Articles/476267/
 «In other words, instead of there only being a maximum of 2MB of
  transaction changes not written to the log at any point in time,
  there may be a much greater amount being accumulated in memory.

  Hence the potential for loss of metadata on a crash is much
  greater than for the existing logging mechanism.

  It should be noted that this does not change the guarantee that
  log recovery will result in a consistent filesystem.

  What it does mean is that as far as the recovered filesystem is
  concerned, there may be many thousands of transactions that
  simply did not occur as a result of the crash.

  This makes it even more important that applications that care
  about their data use fsync() where they need to ensure
  application level data integrity is maintained.»

quoted

 Your NFS/Samba workload on 3 slow disks isn't sufficient to
need that much in memory journal buffer space anyway.

That's probably true, but does no harm.

quoted

 XFS uses relatime which is equivalent to noatime WRT IO
reduction performance, so don't specify 'noatime'.

Uhm, not so sure, and 'noatime' does not hurt either.

I just wanted to be explicit about it so that I know what is
set just in case the defaults change

That's what I do as well, because relying on remembering exactly
what the defaults are can cause sometimes confusion. But it is a
matter of taste to a large degree, like 'noatime'.

quoted

In fact, it appears you don't need to specify anything in
mkfs.xfs or fstab, but just use the defaults.  Fancy that.

For NFS/Samba, especially with ACLs (SMB protocol), and
especially if one expects largish directories, and in general I
would recommend a larger inode size, at least 1024B, if not even
2048B.

Also, as a rule I want to make sure that the sector size is set
to 4096B, for future proofing (and recent drives not only have
4096B sectors but usually lie).

quoted

 And the one thing that might actually increase your
performance a little bit you didn't specify--sunit/swidth.

Especially 'sunit', as XFS ideally would align metadata on chunk
boundaries.

quoted

 However, since you're using mdraid, mkfs.xfs will calculate
these for you (which is nice as mdraid10 with odd disk count
can be a tricky calculation).

Ambiguous more than tricky, and not very useful, except the chunk
size.

quoted

quoted

Will my files be safe even on sudden power loss?

The answer is NO, if you mean "absolutely safe". But see the
discussion at the end.

quoted

[ ... ]  Application write behavior does play a role.

Indeed, see the discussion at the end and ways to mitigate.

quoted

 UPS with shutdown scripts, and persistent write cache prevent
this problem. [ ... ]

There is always the problem of system crashes that don't depend
on power....

quoted

quoted

Is barrier=1 enough?  Do i need to disable the write cache?
with: hdparm -W0 /dev/sdb /dev/sdc /dev/sdd

quoted

Disabling drive write caches does decrease the likelihood of
data loss.

quoted

quoted

I tried it but performance is horrendous.

quoted

And this is why you should leave them enabled and use
barriers.  Better yet, use a RAID card with BBWC and disable
the drive caches.

Budget does not allow for RAID card with BBWC

You'd be surprised by how cheap you can get one. But many HW host
adapters with builtin cache have bad performance or horrid bugs,
so you'd have to be careful.

In any case that's not the major problem you have.

quoted

quoted

Am I better of with ext4? Data safety/integrity is the
priority and optimization affecting it is not acceptable.

XFS is the filesystem of the future ;-). I would choose it over
'ext4' in every plausible case.

nightly backups will be stored on an external USB disk

USB is an unreliable, buggy transport, and slow, eSATA is
enormously better and faster.

is xfs going to be prone to more data loss in case the
non-redundant power supply goes out?

That's the wrong question entirely. Data loss can happen for many
other reasons, and XFS is probably one of the safest designs, if
properly used and configured. The problems are elsewhere.

I just updated the kernel to 3.0.0-16.  Did they take out
barrier support in mdraid? or was the implementation replaced
with FUA?  Is there a definitive test to determine if the off
the shelf consumer sata drives honor barrier or cache flush
requests?

Usually they do, but that's the least of your worries. Anyhow a
test that occurs to me is to write a know pattern to a file,
let's say 1GiB, then 'fsync', and as soon as 'fsync' completes,
power off. Then check whether the whole 1GiB is the known pattern.

I think I'd like to go with device cache turned ON and barrier
enabled.

That's how it is supposed to work.

As to general safety issues, there seem to be some misunderstanding, 
and I'll try to be more explicit than "lob the grenade" notion.

It matters a great deal what "safety" means in your mind and that
of your users. As a previous comment pointed out, that usually
involves backups, that is data that has already been stored.

But your insistence on power off and disk caches etc. seems to
indicate that "safety" in your mind means "when I click the
'Save' button it is really saved and not partially".

As to that there quite a lot of qualifiers:

  * Most users don't understand that even in the best scenario a
    file is really saved not when they *click* the 'Save' button,
    but when they get the "Saved!" message. In between anything
    can happen. Also, work in progress (not yet saved explicitly)
    is fair game.

  * "Really saved" is an *application* concern first and foremost.
    The application *must* say (via 'fsync') that it wants the
    data really saved. Unfortunately most applications don't do
    that because "really saved" is a very expensive operation, and
    usually sytems don't crash, so the application writer looks
    like a genius if he has an "optimistic" attitude. If you do a
    web search look for various O_PONIES discussions. Some intros:

      http://lwn.net/Articles/351422/
      http://lwn.net/Articles/322823/

  * XFS (and to a point 'ext4') is designed for applications that
    work correctly and issue 'fsync' appropriately, and if they do
    it is very safe, because it tries hard to ensure that either
    'fsync' means "really saved" or you know that it does not. XFS
    takes advantage of the assumption that applications do the
    right thing to do various latency-based optimizations between
    calls to 'fsync'.

  * Unfortunately most GUI applications don't do the right thing,
    but fortunately you can compensate for that. The key here is
    to make sure that the flusher's parameter are set for rather
    more frequent flushing than the default, which is equivalent
    to issuing 'fsync' systemwide fairly frequently. Ideally set
    'vm/dirty_bytes' to something like 1-3 seconds of IO transfer
    rate (and in reversal on some of my previous advice leave
    'vm/dirty_background_bytes' to something quite large unless
    you *really* want safety), and to shorten significantly
    'vm/dirty_expire_centisecs', 'vm/dirty_writeback_centisecs'.
    This defeats some XFS optimizations, but that's inevitable.

  * In any case you are using NFS/Samba, and that opens a much
    bigger set of issues, because caching happens on the clients
    too: http://www.sabi.co.uk/0707jul.html#070701b

Then Von Neuman help you if your users or you decide to store lots
of messages in MH/Maildir style mailstores, or VM images on
"growable" virtual disks.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Thu, Mar 15, 2012 at 02:07:25PM +0000, Peter Grandi wrote:

quoted

quoted

quoted

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it. The
filesystem will be used as NFS/Samba storage.

Consider also an 'o2' layout (it is probably the same thing for a
3 drive RAID10) or even a RAID5, as 3 drives and this usage seems
one of the few cases where RAID5 may be plausible.

Well, for a file server like NFS/Samba, you could also consider raid10,f2.
I would think you could get about double the read performance compared to n2 and o2
layouts, and also for individual read transfers on a running system
you would get somthing like double the read performance. 
Write performance could be somewhat slower (0 to 10 %) bot as users
are not waiting for writes to complete, they will probably not notice.
 
best regards
keld

Hey Peter,

On Thu, Mar 15, 2012 at 10:07 PM, Peter Grandi [off-list ref] wrote:

quoted

quoted

quoted

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it. The
filesystem will be used as NFS/Samba storage.

Consider also an 'o2' layout (it is probably the same thing for a
3 drive RAID10) or even a RAID5, as 3 drives and this usage seems
one of the few cases where RAID5 may be plausible.

Thanks for reminding me about raid5. I'll probably give it a try and
do some benchmarks. I'd also like to try raid10f2.

quoted

[ ... ] I've run some benchmarks with dd trying the different
chunks and 256k seems like the sweetspot.  dd if=/dev/zero
of=/dev/md0 bs=64k count=655360 oflag=direct

That's for bulk sequential transfers. Random-ish, as in a
fileserver perhaps with many smaller files, may not be the same,
but probably larger chunks are good.

quoted

quoted

[ ... ] What kernel version?  This can make a significant
difference in XFS metadata performance.

As an aside, that's a myth that has been propagandaized by DaveC
in his entertaining presentation not long ago.

There have been decent but no major improvements in XFS metadata
*performance*, but weaker implicit *semantics* have been made an
option, and these have a different safety/performance tradeoff
(less implicit safety, somewhat more performance), not "just"
better performance.

http://lwn.net/Articles/476267/
 «In other words, instead of there only being a maximum of 2MB of
 transaction changes not written to the log at any point in time,
 there may be a much greater amount being accumulated in memory.

 Hence the potential for loss of metadata on a crash is much
 greater than for the existing logging mechanism.

 It should be noted that this does not change the guarantee that
 log recovery will result in a consistent filesystem.

 What it does mean is that as far as the recovered filesystem is
 concerned, there may be many thousands of transactions that
 simply did not occur as a result of the crash.

 This makes it even more important that applications that care
 about their data use fsync() where they need to ensure
 application level data integrity is maintained.»

quoted

quoted

 Your NFS/Samba workload on 3 slow disks isn't sufficient to
need that much in memory journal buffer space anyway.

That's probably true, but does no harm.

quoted

quoted

 XFS uses relatime which is equivalent to noatime WRT IO
reduction performance, so don't specify 'noatime'.

Uhm, not so sure, and 'noatime' does not hurt either.

quoted

I just wanted to be explicit about it so that I know what is
set just in case the defaults change

That's what I do as well, because relying on remembering exactly
what the defaults are can cause sometimes confusion. But it is a
matter of taste to a large degree, like 'noatime'.

quoted

quoted

In fact, it appears you don't need to specify anything in
mkfs.xfs or fstab, but just use the defaults.  Fancy that.

For NFS/Samba, especially with ACLs (SMB protocol), and
especially if one expects largish directories, and in general I
would recommend a larger inode size, at least 1024B, if not even
2048B.

thanks for this tip. will look into adjusting inode size.

Also, as a rule I want to make sure that the sector size is set
to 4096B, for future proofing (and recent drives not only have
4096B sectors but usually lie).

it seems the 1TB drivers that I have still have 512byte sectors

quoted

quoted

 And the one thing that might actually increase your
performance a little bit you didn't specify--sunit/swidth.

Especially 'sunit', as XFS ideally would align metadata on chunk
boundaries.

quoted

quoted

 However, since you're using mdraid, mkfs.xfs will calculate
these for you (which is nice as mdraid10 with odd disk count
can be a tricky calculation).

Ambiguous more than tricky, and not very useful, except the chunk
size.

quoted

quoted

quoted

Will my files be safe even on sudden power loss?

The answer is NO, if you mean "absolutely safe". But see the
discussion at the end.

quoted

quoted

[ ... ]  Application write behavior does play a role.

Indeed, see the discussion at the end and ways to mitigate.

quoted

quoted

 UPS with shutdown scripts, and persistent write cache prevent
this problem. [ ... ]

There is always the problem of system crashes that don't depend
on power....

quoted

quoted

quoted

Is barrier=1 enough?  Do i need to disable the write cache?
with: hdparm -W0 /dev/sdb /dev/sdc /dev/sdd

quoted

quoted

Disabling drive write caches does decrease the likelihood of
data loss.

quoted

quoted

quoted

I tried it but performance is horrendous.

quoted

quoted

And this is why you should leave them enabled and use
barriers.  Better yet, use a RAID card with BBWC and disable
the drive caches.

quoted

Budget does not allow for RAID card with BBWC

You'd be surprised by how cheap you can get one. But many HW host
adapters with builtin cache have bad performance or horrid bugs,
so you'd have to be careful.

could you please suggest a hardware raid card with BBU that's cheap?

In any case that's not the major problem you have.

quoted

quoted

quoted

Am I better of with ext4? Data safety/integrity is the
priority and optimization affecting it is not acceptable.

XFS is the filesystem of the future ;-). I would choose it over
'ext4' in every plausible case.

quoted

nightly backups will be stored on an external USB disk

USB is an unreliable, buggy transport, and slow, eSATA is
enormously better and faster.

quoted

is xfs going to be prone to more data loss in case the
non-redundant power supply goes out?

That's the wrong question entirely. Data loss can happen for many
other reasons, and XFS is probably one of the safest designs, if
properly used and configured. The problems are elsewhere.

Can you please elaborate how xfs can be properly used and configured?

quoted

I just updated the kernel to 3.0.0-16.  Did they take out
barrier support in mdraid? or was the implementation replaced
with FUA?  Is there a definitive test to determine if the off
the shelf consumer sata drives honor barrier or cache flush
requests?

Usually they do, but that's the least of your worries. Anyhow a
test that occurs to me is to write a know pattern to a file,
let's say 1GiB, then 'fsync', and as soon as 'fsync' completes,
power off. Then check whether the whole 1GiB is the known pattern.

quoted

I think I'd like to go with device cache turned ON and barrier
enabled.

That's how it is supposed to work.

As to general safety issues, there seem to be some misunderstanding,
and I'll try to be more explicit than "lob the grenade" notion.

It matters a great deal what "safety" means in your mind and that
of your users. As a previous comment pointed out, that usually
involves backups, that is data that has already been stored.

But your insistence on power off and disk caches etc. seems to
indicate that "safety" in your mind means "when I click the
'Save' button it is really saved and not partially".

let me define safety as needed by the usecase:
fileA is a 2MB open office document file already existing on the file system.
userA opens fileA locally, modifies a lot of lines and attempts to save it.
as the saving operation is proceeding, the PSU goes haywire and power
is cut abruptly.
When the system is turned on, i expect some sort of recovery process
to bring the filesystem to a consistent state.
I expect fileA should be as it was before the save operation and
should not be corrupted in anyway.
Am I asking/expecting too much?

As to that there quite a lot of qualifiers:

 * Most users don't understand that even in the best scenario a
   file is really saved not when they *click* the 'Save' button,
   but when they get the "Saved!" message. In between anything
   can happen. Also, work in progress (not yet saved explicitly)
   is fair game.

 * "Really saved" is an *application* concern first and foremost.
   The application *must* say (via 'fsync') that it wants the
   data really saved. Unfortunately most applications don't do
   that because "really saved" is a very expensive operation, and
   usually sytems don't crash, so the application writer looks
   like a genius if he has an "optimistic" attitude. If you do a
   web search look for various O_PONIES discussions. Some intros:

     http://lwn.net/Articles/351422/
     http://lwn.net/Articles/322823/

 * XFS (and to a point 'ext4') is designed for applications that
   work correctly and issue 'fsync' appropriately, and if they do
   it is very safe, because it tries hard to ensure that either
   'fsync' means "really saved" or you know that it does not. XFS
   takes advantage of the assumption that applications do the
   right thing to do various latency-based optimizations between
   calls to 'fsync'.

 * Unfortunately most GUI applications don't do the right thing,
   but fortunately you can compensate for that. The key here is
   to make sure that the flusher's parameter are set for rather
   more frequent flushing than the default, which is equivalent
   to issuing 'fsync' systemwide fairly frequently. Ideally set
   'vm/dirty_bytes' to something like 1-3 seconds of IO transfer
   rate (and in reversal on some of my previous advice leave
   'vm/dirty_background_bytes' to something quite large unless
   you *really* want safety), and to shorten significantly
   'vm/dirty_expire_centisecs', 'vm/dirty_writeback_centisecs'.
   This defeats some XFS optimizations, but that's inevitable.

 * In any case you are using NFS/Samba, and that opens a much
   bigger set of issues, because caching happens on the clients
   too: http://www.sabi.co.uk/0707jul.html#070701b

Then Von Neuman help you if your users or you decide to store lots
of messages in MH/Maildir style mailstores, or VM images on
"growable" virtual disks.

what's wrong with VM images on "growable" virtual disks. are you
saying not to rely on lvm2 volumes?

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi keld,

On Thu, Mar 15, 2012 at 11:25 PM,  [off-list ref] wrote:

On Thu, Mar 15, 2012 at 02:07:25PM +0000, Peter Grandi wrote:

quoted

quoted

quoted

quoted

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it. The
filesystem will be used as NFS/Samba storage.

Consider also an 'o2' layout (it is probably the same thing for a
3 drive RAID10) or even a RAID5, as 3 drives and this usage seems
one of the few cases where RAID5 may be plausible.

Well, for a file server like NFS/Samba, you could also consider raid10,f2.
I would think you could get about double the read performance compared to n2 and o2
layouts, and also for individual read transfers on a running system
you would get somthing like double the read performance.
Write performance could be somewhat slower (0 to 10 %) bot as users
are not waiting for writes to complete, they will probably not notice.

I also plan to try raid10f2. Did you do your own benchmarks or are you
quoting someone elses?

best regards
keld

thanks for chiming in. have a nice day

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Fri, Mar 16, 2012 at 12:52:19AM +0800, Jessie Evangelista wrote:

Hi keld,

On Thu, Mar 15, 2012 at 11:25 PM,  [off-list ref] wrote:

quoted

On Thu, Mar 15, 2012 at 02:07:25PM +0000, Peter Grandi wrote:

quoted

quoted

quoted

quoted

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it. The
filesystem will be used as NFS/Samba storage.

Consider also an 'o2' layout (it is probably the same thing for a
3 drive RAID10) or even a RAID5, as 3 drives and this usage seems
one of the few cases where RAID5 may be plausible.

Well, for a file server like NFS/Samba, you could also consider raid10,f2.
I would think you could get about double the read performance compared to n2 and o2
layouts, and also for individual read transfers on a running system
you would get somthing like double the read performance.
Write performance could be somewhat slower (0 to 10 %) bot as users
are not waiting for writes to complete, they will probably not notice.

I also plan to try raid10f2. Did you do your own benchmarks or are you
quoting someone elses?

Both, look at our wiki: https://raid.wiki.kernel.org/articles/p/e/r/Performance.html

Best regards
keld

On Thu, Mar 15, 2012 at 06:15:49PM +0100, keld@keldix.com wrote:

On Fri, Mar 16, 2012 at 12:52:19AM +0800, Jessie Evangelista wrote:

quoted

Hi keld,

On Thu, Mar 15, 2012 at 11:25 PM,  [off-list ref] wrote:

quoted

On Thu, Mar 15, 2012 at 02:07:25PM +0000, Peter Grandi wrote:

quoted

quoted

quoted

quoted

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it. The
filesystem will be used as NFS/Samba storage.

Consider also an 'o2' layout (it is probably the same thing for a
3 drive RAID10) or even a RAID5, as 3 drives and this usage seems
one of the few cases where RAID5 may be plausible.

Well, for a file server like NFS/Samba, you could also consider raid10,f2.
I would think you could get about double the read performance compared to n2 and o2
layouts, and also for individual read transfers on a running system
you would get somthing like double the read performance.
Write performance could be somewhat slower (0 to 10 %) bot as users
are not waiting for writes to complete, they will probably not notice.

I also plan to try raid10f2. Did you do your own benchmarks or are you
quoting someone elses?

Both, look at our wiki: https://raid.wiki.kernel.org/articles/p/e/r/Performance.html

I think it would be interesting to include your figures on the wiki page, if
you publish them here on the list. 

Maybe we should rearrange the wiki page a little. I am not so happy about the 
data reported in the section "New benchmarks from 2011" as it only illustrates
what is happening with a 100 % used CPU. I would like to move it to a separate page.
Also the really old data in section "Old performance benchmark" should be moved to
a separate page, IMHO. The text on the wiki page should be gaving info of
general interest for systems running today (still IMHO). Comments?

Best regards
keld

[ ... ]

quoted

Also, as a rule I want to make sure that the sector size is
set to 4096B, for future proofing (and recent drives not only
have 4096B sectors but usually lie).

it seems the 1TB drivers that I have still have 512byte sectors

But usually you can still set the XFS idea of sector size to 4096,
which is probably a good idea in general.

[ ... ]

quoted

quoted

is xfs going to be prone to more data loss in case the
non-redundant power supply goes out?

quoted

That's the wrong question entirely. Data loss can happen for
many other reasons, and XFS is probably one of the safest
designs, if properly used and configured. The problems are
elsewhere.

Can you please elaborate how xfs can be properly used and
configured?

I did that in the following bits of the reply. You must be in a
real hurry if you cannot trim down the quoting or write your
comments after reading through once...

[ ... ]

quoted

But your insistence on power off and disk caches etc. seems to
indicate that "safety" in your mind means "when I click the
'Save' button it is really saved and not partially".

let me define safety as needed by the usecase: fileA is a 2MB
open office document file already existing on the file system.
userA opens fileA locally, modifies a lot of lines and attempts
to save it. as the saving operation is proceeding, the PSU goes
haywire and power is cut abruptly.

To worry you, if the PSU goes haywire, the disk data may become
subtly corrupted:

https://blogs.oracle.com/elowe/entry/zfs_saves_the_day_ta
 «Another user, also running a Tyan 2885 dual-Opteron workstation
  like mine, had experienced data corruption with SATA disks. The
  root cause? A faulty power supply.»

Even if that is not an argument for filesystem provided checksums,
as the ZFS (and other) people say, but for end-to-end (application
level) ones.

When the system is turned on, i expect some sort of recovery
process to bring the filesystem to a consistent state.

The XFS design really cares about that and unless the hardware is
very broken metadata consistency will be good.

I expect fileA should be as it was before the save operation and
should not be corrupted in anyway.  Am I asking/expecting too much?

That is too much to expect of the filesystem and at the same time
too little.

It is too much because it is strictly the responsibility of the
application, and it is very expensive, because it can only happen
by simulating copy-on-write (app makes a copy of the document,
updates the copy, and then atomically renames it, and then makes
another copy). Some applications like OOo/LibreO/VIM instead use a
log file to record updates, and then merge those on save (copy,
merge, rename), which is better. Some filesystems like NILFS2 or
BTRFS or Next3/Next4 use COW to provide builtin versioning, but
that's expensive too. The original UNIX insight to provide a very
simple file abstraction layer should not be lightly discarded (but
I like NILFS2 in particular).

It is too little because of what happens if you have dozens to
thousands of modified but not yet fully persisted files, sych as
newly created mail folders, 'tar' unpacks , source tree checkins,
...

As I tried to show in my previous reply, and in the NFS blog entry
mentioned in it too, on a creduly practical level relying on
applications doing the right thing is optimistic, and it may be
regrettably expedient to complement barriers with frequent system
driven flushing, which partially simulates (at a price) O_PONIES.

[ ... ]

quoted

Then Von Neuman help you if your users or you decide to store
lots of messages in MH/Maildir style mailstores, or VM images
on "growable" virtual disks.

what's wrong with VM images on "growable" virtual disks. are you
saying not to rely on lvm2 volumes?

By "growable" I mean that the virtual disk is allocated sparsely.

As to to LVM2 it is very rarely needed. The only really valuable
feature it has is snapshot LVs, and those are very expensive. XFS,
which can allocate routinely 2GiB (or bigger) files as a single
extents, can be used as a volume manager too.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

But usually you can still set the XFS idea of sector size to 4096,
which is probably a good idea in general.

I'm now running kernel 3.0.0-16-server Ubuntu 10.04LTS
cat /sys/block/sd[b-d]/queue/physical_block_size shows 512
cat /sys/block/sd[b-d]/device/model shows ST31000524AS
looking up the model at seagate, the specs page does not mention 512
byte sectors
but it did mention guaranteed sectors of 1,953,525,168
multiplying by 512bytes we do get 1000204886016(1TBish)

Anyway, I'll have a look at setting the sector size for xfs

I did that in the following bits of the reply. You must be in a
real hurry if you cannot trim down the quoting or write your
comments after reading through once...

I did read thru your comments several times and I really appreciate them.
Will look into setting vm/dirty_bytes, vm/dirty_background_bytes,
vm/dirty_expire_centisecs, vm/dirty_writeback_centisecs.

I'm still scouring the internet for a best practice recipe for
implementing xfs/mdraid.
I am open to writing one and including the inputs everyone is contributing here.
In my search, I also saw some references of alignment issues for partitions.
this is what I used to setup the partitions for the md device

sfdisk /dev/sdb <<EOF
unit: sectors

63,104872257,fd
0,0,0
0,0,0
0,0,0
EOF

I've read a recommendation to start the partition on the 1MB mark.
Does this make sense?

quoted

let me define safety as needed by the usecase: fileA is a 2MB
open office document file already existing on the file system.
userA opens fileA locally, modifies a lot of lines and attempts
to save it. as the saving operation is proceeding, the PSU goes
haywire and power is cut abruptly.

To worry you, if the PSU goes haywire, the disk data may become
subtly corrupted:

https://blogs.oracle.com/elowe/entry/zfs_saves_the_day_ta
 «Another user, also running a Tyan 2885 dual-Opteron workstation
 like mine, had experienced data corruption with SATA disks. The
 root cause? A faulty power supply.»

Even if that is not an argument for filesystem provided checksums,
as the ZFS (and other) people say, but for end-to-end (application
level) ones.

Mmmm, Ive also been reading up on ZFS but haven't put it thru its paces.

quoted

I expect fileA should be as it was before the save operation and
should not be corrupted in anyway.  Am I asking/expecting too much?

That is too much to expect of the filesystem and at the same time
too little.

It is too much because it is strictly the responsibility of the
application, and it is very expensive, because it can only happen
by simulating copy-on-write (app makes a copy of the document,
updates the copy, and then atomically renames it, and then makes
another copy). Some applications like OOo/LibreO/VIM instead use a
log file to record updates, and then merge those on save (copy,
merge, rename), which is better. Some filesystems like NILFS2 or
BTRFS or Next3/Next4 use COW to provide builtin versioning, but
that's expensive too. The original UNIX insight to provide a very
simple file abstraction layer should not be lightly discarded (but
I like NILFS2 in particular).

It is too little because of what happens if you have dozens to
thousands of modified but not yet fully persisted files, sych as
newly created mail folders, 'tar' unpacks , source tree checkins,
...

As I tried to show in my previous reply, and in the NFS blog entry
mentioned in it too, on a creduly practical level relying on
applications doing the right thing is optimistic, and it may be
regrettably expedient to complement barriers with frequent system
driven flushing, which partially simulates (at a price) O_PONIES.

I'd like to read about the NFS blog entry but the link you included
results in a 404.
I forgot to mention in my last reply.

Based on what I understood from your thoughts above, if an
applications issues a flush/fsync
and it does not complete due to some catastrophic crash,
xfs on its own can not roll back to the prev version of the file in
case of unfinished write operation.
disabling the device caches wouldn't help either right?
only filesystems that do COW can do this at the expense of
performance? (btrfs and zfs, please hurry and grow up!)

As to to LVM2 it is very rarely needed. The only really valuable
feature it has is snapshot LVs, and those are very expensive. XFS,
which can allocate routinely 2GiB (or bigger) files as a single
extents, can be used as a volume manager too.

If you were in my place with the resource constraints, you'd go with:
xfs with barriers on top of mdraid10 with device cache ON and setting
vm/dirty_bytes, vm/dirty_background_bytes, vm/dirty_expire_centisecs,
vm/dirty_writeback_centisecs to safe values
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Am Freitag, 16. März 2012, 11:36:07 schrieb Jessie Evangelista:

If you were in my place with the resource constraints, you'd go with:
xfs with barriers on top of mdraid10 with device cache ON and setting
vm/dirty_bytes, vm/dirty_background_bytes, vm/dirty_expire_centisecs,
vm/dirty_writeback_centisecs to safe values

If you ever experienced a crash where lots of sensible and important 
data were lost, you would not even think about "device cache ON".

could you please suggest a hardware raid card with BBU that's cheap?

"Cheap" is a varying definition. How much is your data worth? How much 
does one day of blackout cost? 

I've been very happy with Areca Controllers, like the 12x0 and 1680 
series, and now there's the newer 1882 series like 
http://geizhals.at/eu/721745
plus a BBU for about 100€.

You can even mix RAID levels on the same disks, example of 8x1TB define 
a RAID0 of 500G and the rest a RAID6. Online expansion possible, 
scheduled background verify, e-mail notification on everything, logging, 
ntp times, oob-mgmnt via it's own network interface, ...
Very reliable, I never had a problem. And they have a good support team.

-- 
mit freundlichen Grüssen,
Michael Monnerie, Ing. BSc

it-management Internet Services: Protéger
http://proteger.at [gesprochen: Prot-e-schee]
Tel: +43 660 / 415 6531

[ ... ]

quoted

If you were in my place with the resource constraints, you'd
go with: xfs with barriers on top of mdraid10 with device
cache ON and setting vm/dirty_bytes, vm/dirty_background_bytes,
vm/dirty_expire_centisecs, vm/dirty_writeback_centisecs to
safe values

If you ever experienced a crash where lots of sensible and
important data were lost, you would not even think about
"device cache ON".

It is not as simple as that... *If* hw barriers are implemented
*and* applications do the right things, that is not a concern.
Disabling the device cache is just a way to turn barriers on for
everything. Indeed the whole rationale for having the 'barrier'
option is to let the device caches on, and the OP did ask how to
test that barriers actually work.

Since even most consumers level drives currently implement
barriers correctly, the biggest problem today, as per the
O_PONIES discussions, is applications that don't do the right
thing, and therefore the biggest risk is large amounts of dirty
pages in system memory (either NFS client or server), not in the
drive caches.

Since the Linux flusher parameters are/have been demented, I
have seen one or more GiB of dirty pages in system memory (on
hosts I didn't configure...), which also causes performance
problems.

Again, as a crassly expedient thing, working around the lack of
"do the right thing" in applications by letting only a few
seconds of dirty pages accumulate in system memory seems to fool
enough users (and many system administrators and application
developers) into thinking that stuff is "safe". It worked well
enough for 'ext3' for many years, quite regrettably.

  Note: 'ext3' has also had the "helpful" issue of excessive
  impact of flushing, which made 'fsync' performance terrible,
  but improved the apparent safety for "optimistic" applications.

On 3/15/2012 7:06 AM, Jessie Evangelista wrote:

On Thu, Mar 15, 2012 at 1:38 PM, Stan Hoeppner [off-list ref] wrote:

quoted

Why 256KB for chunk size?

For reference, the machine has 16GB memory

I've run some benchmarks with dd trying the different chunks and 256k
seems like the sweetspot.
dd if=/dev/zero of=/dev/md0 bs=64k count=655360 oflag=direct

Using dd in this manner is precisely analogous to taking your daily
driver Toyota to the local drag strip, making a few runs, and observing
your car can accelerate from 0-92 mph in 1320 ft.  This has no
correlation to daily driving on public roads.

I'll probably forgo setting the journal log file size. It seemed like
a safe optimization from what I've read.

See:
http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E

I just wanted to be explicit about it so that I know what is set just
in case the defaults change

See:
http://xfs.org/index.php/XFS_FAQ#Q:_I_want_to_tune_my_XFS_filesystems_for_.3Csomething.3E

Even if the XFS mount defaults change you won't notice a difference, not
on this server, except for possibly delaylog if you do a lot of 'rm -rf'
operations on directories containing tens of thousands of files.
Delaylog is the only mount default change in many years.  It occurred in
2.6.39, which is why I recommended this rev as the minimum you should
choose.

quoted

In fact, it appears you don't need to specify anything in mkfs.xfs or
fstab, but just use the defaults.  Fancy that.  And the one thing that
might actually increase your performance a little bit you didn't
specify--sunit/swidth.  However, since you're using mdraid, mkfs.xfs
will calculate these for you (which is nice as mdraid10 with odd disk
count can be a tricky calculation).  Again, defaults work for a reason.

The reason I did not set sunit/swidth is because I read somewhere that
mkfs.xfs will calculate based on mdraid.

I guess my stating of the same got lost in the rest of that paragraph. ;)

The server is for a non-profit org that I am helping out.
I think a APC Smart-UPS SC 420VA 230V may fit their shoe string budget.

Given the rough server specs you presented, a 420 (260 watts) should be
fine, assuming you're not running seti@home, folding@home, etc, which
can double average system power draw.  A 420 won't yield much battery
run time but will give more than enough time for a clean shutdown.  Are
you sure you want a 230v model?  If so I'd guess you're outside the
States.  Also:  http://www.apcupsd.com/  APC control daemon with auto
shutdown.

nightly backups will be stored on an external USB disk
is xfs going to be prone to more data loss in case the non-redundant
power supply goes out?

There are bigger issues here WRT XFS and USB connected disks IIRC from
some list posts.  USB is prone to random device/bus disconnections due
to power management in various USB controllers.  XFS design assumes
storage devices are persistently connected, and it does frequent
background reads/writes to the device.  If the USB drive is offline long
enough, lots of errors are logged, and XFS can't access it, it may/will
shutdown the filesystem as a safety precaution.  If you want to use XFS
on that external USB drive, you need to do some research first--I don't
have solid answers for you here.  Or simply use EXT3/4.  XFS isn't going
to yield any advantage with single threaded backup anyway, so maybe just
going with EXT is the smart move.

quoted

I'll appreciate your response stating "Yes, I have a UPS and
tested/working shutdown scripts" or "I'll be implementing a UPS very
soon." :)

I don't have shutdown scripts yet but will look into it.

Again:  http://www.apcupsd.com/  There may be others available.

Meatware would have to do for now as the server will probably be ON
only when there's people at the office. 

I just hope do proper shutdowns when they power it off. ;)

And yes I will be asking them
to not go into production without a UPS

If it's a hard sell, simply explain that every laptop has a built in
UPS, and that the server and its data are obviously as important, if not
more, than any laptop.

Thanks for you input Stan.

You're welcome.

I just updated the kernel to 3.0.0-16.
Did they take out barrier support in mdraid? or was the implementation
replaced with FUA?

Write barriers, in one form or another, are there.  These will never be
removed or broken--too critical.  The implementation may have changed.
Neil can answer this much better than me.

Is there a definitive test to determine if the off the shelf consumer
sata drives honor barrier or cache flush requests?

Just connect the drive and boot up.  You'll see this in dmesg:

sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't
support DPO or FUA

And:
$ hdparm -I /dev/sda
...
Commands/features:
        Enabled Supported:
                ...
           *    Write cache
                ...
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
                ...
These are good indicators that the drive supports barrier operations.

I think I'd like to go with device cache turned ON and barrier enabled.

You just stated the the Linux defaults.  Do note that XFS write barriers
will ensure journal and thus filesystem integrity in a crash/power fail
event.  They do NOT guarantee file data integrity as file data isn't
journaled.  No filesystem (Linux anyway) journals data, only metadata.
To prevent file data loss due to a crash/power fail, you must disable
the drive write caches completely and/or use a BBWC RAID card.

As you know performance is horrible with caches disabled.  With so few
users and so little data writes, you're safe running with barriers and
write cache enabled.  This is how most people on this list with plain
HBAs run their systems.

Am still torn between ext4 and xfs i.e. which will be safer in this
particular setup.

Neither is "safer" than the other.  That's up to your hardware and power
configuration.  Pick the one you are most comfortable working with and
have the most experience supporting.  For this non-prof SOHO workload,
XFS' advanced features will yield little, if any, performance
advantage--you have too few users, disks, and too little IO.

If this box had, say, 24 cores, 128GB RAM, and 192 15k SAS drives across
4 dual port SAS RAID HBAs, 8x24 drive hardware RAID10s in an mdraid
linear array, with a user IO load demanding such a system--multiple GB/s
of concurrent IO, then the only choice is XFS.  EXT4 simply can't scale
close to anything like this.

All things considered, for your system, EXT4 is probably the best choice.

-- 
Stan

On Fri, Mar 16, 2012 at 11:36:07AM +0800, Jessie Evangelista wrote:

I'm still scouring the internet for a best practice recipe for
implementing xfs/mdraid.
I am open to writing one and including the inputs everyone is contributing here.
In my search, I also saw some references of alignment issues for partitions.
this is what I used to setup the partitions for the md device

sfdisk /dev/sdb <<EOF
unit: sectors

63,104872257,fd
0,0,0
0,0,0
0,0,0
EOF

I've read a recommendation to start the partition on the 1MB mark.
Does this make sense?

I would just make the raw disks members of the RAID array, e.g.
/dev/sdb, /dev/sdc etc and not partition them.

On Fri, Mar 16, 2012 at 7:25 AM, Stan Hoeppner [off-list ref] wrote:
..

You just stated the the Linux defaults.  Do note that XFS write barriers
will ensure journal and thus filesystem integrity in a crash/power fail
event.  They do NOT guarantee file data integrity as file data isn't
journaled.  No filesystem (Linux anyway) journals data, only metadata.

..

That's not true, is it? ext3 and ext4 support journal=data.

-- 
Jon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Fri, Mar 16, 2012 at 1:01 PM, Jon Nelson
[off-list ref] wrote:

On Fri, Mar 16, 2012 at 7:25 AM, Stan Hoeppner [off-list ref] wrote:
..

quoted

You just stated the the Linux defaults.  Do note that XFS write barriers
will ensure journal and thus filesystem integrity in a crash/power fail
event.  They do NOT guarantee file data integrity as file data isn't
journaled.  No filesystem (Linux anyway) journals data, only metadata.

..

That's not true, is it? ext3 and ext4 support journal=data.

And btrfs supports COW (as does nilfs2) with "transactions", which
should/could be similar?

-- 
Jon
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[ ... ]

quoted

quoted

write barriers will ensure journal and thus filesystem
integrity in a crash/power fail event.  They do NOT guarantee
file data integrity as file data isn't journaled.

Not well expressed, as XFS barriers do ensure file data integrity,
*if the applications uses them* (and uses them in exactly the
right way).

The difference between metadata and data with XFS is that XFS
itself will use barriers on metadata at the right times, because
that's data to XFS, but it won't use barriers on data, leaving
that entirely to the application.

quoted

quoted

 No filesystem (Linux anyway) journals data, only metadata.

quoted

That's not true, is it? ext3 and ext4 support journal=data.

They do, because they journal blocks, which is not generally a
great choice, but gives the option to journal data blocks too more
easily than other choices. But it is a very special case that few
people use.

Also, there are significant issues with 'ext3' and 'fsync' and
journaling:

http://lwn.net/Articles/328363/
 «There is one other important change needed to get a truly
  quick fsync() with ext3, though: the filesystem must be
  mounted in data=writeback mode. This mode eliminates the
  requirement that data blocks be flushed to disk ahead of
  metadata; in data=ordered mode, instead, the amount of data to
  be written guarantees that fsync() will always be slower.
  Switching to data=writeback eliminates those writes, but, in
  the process, it also turns off the feature which made ext3
  seem more robust than ext4.»

On a more general note, journaling and barriers are sort of
distinct issues.

The real purpose of barriers is to ensure that updates are
actually on the recording medium, whether in the journal or
directly on final destination.
That is barriers are used to ensure that data or metadata on the
persistent layer is current.

The purpose of a journal is not to ensure that the state on the
persistent layer are *current*, but rather *consistent* (at a
lower cost than synchronous updates), without having to be
careful about the order in which the updates are made current.
The updates are made consistent by writing them to the log as
they are needed (not necessarily immediately), and then on
recovery the order gets sorted out spatially.

Currency does not imply consistency (if the updates are made
current in some arbitrary order) and consistency does not imply
currency (if the recording medium is kept consistent but updates
are applied to it infrequently).

The BSD FFS does not need a journal because it is designed to be
very careful as to the order in which updates are made current,
and log file systems don't aim for spatial currency.

And btrfs supports COW (as does nilfs2) with "transactions",
which should/could be similar?

Not quite. They are more like "checkpoints", that is alternate
root inodes that "snapshot" the state of the whole filetree at
some point.

These are not entirely inexpensive, and as a result as I learned
from a talk about some recent updates about the BSD FFS:

  http://www.sabi.co.uk/blog/12-two.html#120222

COW filesystems like ZFS/BTRFS/... need to have a journal too to
support 'fsync' in between checkpoints.

BTW there are now COW versions of 'ext3' and 'ext4', with
snapshotting too:

  http://www.sabi.co.uk/blog/12-two.html#120218b

The 'freeze' features of XFS does not rely on snapshotting, it
relies on suspending all processes that are writing to the
filetree, so updates are avoided for the duration.

As the XFS team have been adding or planning to add various "new"
features like checksums, maybe one day they will add COW to XFS
too (not such an easy task when considering how large XFS extents
can be, but the hole punching code can help there).
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On 3/16/2012 2:28 PM, Peter Grandi wrote:

[ ... ]

quoted

quoted

quoted

write barriers will ensure journal and thus filesystem
integrity in a crash/power fail event.  They do NOT guarantee
file data integrity as file data isn't journaled.

Not well expressed, 

Given the audience, the OP, I was simply avoiding getting too deep in
the weeds Peter.  This thread is on the linux-raid list, not xfs@oss.
You know I have a tendency to get too deep in the weeds.  I think I did
nice job of balance here. ;)

as XFS barriers do ensure file data integrity,
*if the applications uses them* (and uses them in exactly the
right way).

How will the OP know which, if any, of his users' desktop applications
do fsyncs, properly?  He won't.  Which is why I made the general
statement, which is correct, if not elaborate, nor down in the weeds.

The difference between metadata and data with XFS is that XFS
itself will use barriers on metadata at the right times, because
that's data to XFS, but it won't use barriers on data[1], leaving
that entirely to the application.

[1]File data, just to be clear

quoted

quoted

quoted

 No filesystem (Linux anyway) journals data, only metadata.

quoted

quoted

That's not true, is it? ext3 and ext4 support journal=data.

They do, because they journal blocks, which is not generally a
great choice, but gives the option to journal data blocks too more
easily than other choices. But it is a very special case that few
people use.

Few use it because the performance is absolutely horrible.  data=journal
disables delayed allocation (which serious contributes to any modern
filesystem's performance--EXT devs stole/borrowed delayed allocation
from XFS BTW) and it disables O_DIRECT.  It also doubles the number of
data writes to media, once to the journal, once to the filesystem, for
every block of every file written.

On a more general note, journaling and barriers are sort of
distinct issues.

The real purpose of barriers is to ensure that updates are
actually on the recording medium, whether in the journal or
directly on final destination.
That is barriers are used to ensure that data or metadata on the
persistent layer is current.

Correct.  Again, trying to stay out of the weeds.  I'd established that
XFS uses barriers on journal writes for metadata consistency, which
prevents filesystem corruption after a crash, but not necessarily file
corruption.  Making the statement that XFS doesn't journal data gets the
point across more quickly, while staying out of the weeds.

[...]

The 'freeze' features of XFS does not rely on snapshotting, it
relies on suspending all processes that are writing to the
filetree, so updates are avoided for the duration.

xfs_freeze was moved into the VFS in 2.6.29 and is called automatically
when doing an LVM snapshot of any Linux FS supporting such.  Thus,
snapshotting relies on xfs_freeze, not the other way round.  And
xfs_freeze doesn't suspend all processes that are writing to the
filesystem.  All write system calls to the filesystem are simply halted,
and the process blocks on IO until the filesystem is unfrozen.

As the XFS team have been adding or planning to add various "new"
features like checksums, maybe one day they will add COW to XFS
too (not such an easy task when considering how large XFS extents
can be, but the hole punching code can help there).

Not at all an easy rewrite of XFS.  And that's what COW would be, a
massive rewrite.  Copy on write definitely has some advantages for some
usage scenarios, but it's not yet been proven the holy grail of
filesystem design.

-- 
Stan

[ ... ]

I've read a recommendation to start the partition on the 1MB
mark. Does this make sense?

As a general principle it is good, that has almost no cost.
Indeed recent versions of some partitionig tools do that by
default.

I often recommend aligning partitions to 1GiB, also because I
like to have 1GiB or so of empty space at the very beginning and
end of a drive.

I'd like to read about the NFS blog entry but the link you
included results in a 404.  I forgot to mention in my last
reply.

Oops I forgot a bit of the URL:
  http://www.sabi.co.uk/blog/0707jul.html#070701b

Note that currently I suggest different values from:

 «vm/dirty_ratio                  =4
  vm/dirty_background_ratio       =2»

Because:

  * 4% of memory "dirty" today is often a gigantic amount.
    I had provided an elegant patch to specify the same in
    absolute terms in
      http://www.sabi.co.uk/blog/0707jul.html#070701
    but now the official way is the "_bytes" alternative.

  * 2% as the level at which writing becomes uncached is too
    low, and the system become unresposive when that level is
    crossed. Sure it is risky, but, regretfully, I think that
    maintaining responsiveness is usually better than limiting
    outstanding background writes.

Based on what I understood from your thoughts above, if an
applications issues a flush/fsync and it does not complete due
to some catastrophic crash, xfs on its own can not roll back
to the prev version of the file in case of unfinished write
operation. disabling the device caches wouldn't help either
right?

If your goal is to make sure incomplete updates don't get
persisted, disabling device caches might help with that, in a
very perverse way (if the whole partial update is still in the
device cache, it just vanishes). Forget that of course :-).

The main message is that filesystems in UNIX-like system should
not provide atomic transactions, just the means to do them at
the applications level, because they are both difficult and very
expensive.

The secondary message is that some applications and the firmware
of some host adpters and drives don't do the right thing, and
if your really want to make sure about atomic transactions it is
an expensive and difficult system integration challenge.

[ ... ] only filesystems that do COW can do this at the
expense of performance? (btrfs and zfs, please hurry and grow
up!)

Filesystems that do COW sort-of do *global* "rolling" updates,
that is filtree level snapshots, but that's a side effect of a
choice made for other reasons (consistency more than currency).

[ ... ] If you were in my place with the resource constraints,
you'd go with: xfs with barriers on top of mdraid10 with
device cache ON and setting vm/dirty_bytes, [ ... ]

Yes, that seems a reasonable overall tradeoff, because XFS is
implemented to provide well defined (and documented) semantics,
to check whether the underlying storage layer actually does
barriers, and to perform decently even if "delayed" writing is
not that delayed.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Dne 17.3.2012 16:35, Peter Grandi napsal(a):

I often recommend aligning partitions to 1GiB, also because I
like to have 1GiB or so of empty space at the very beginning and
end of a drive.

I'm really curious why do you use such alignment? I can think about few
reasons, but most practical I think you like to slice in gigabyte sizes.

Z.

Dne 15.3.2012 1:30, Jessie Evangelista napsal(a):

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it.
The filesystem will be used as NFS/Samba storage.

mdadm --zero /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm -v --create /dev/md0 --metadata=1.2 --assume-clean
--level=raid10 --chunk 256 --raid-devices=3 /dev/sdb1 /dev/sdc1
/dev/sdd1
mkfs -t xfs -l lazy-count=1,size=128m -f /dev/md0
mount -t xfs -o barrier=1,logbsize=256k,logbufs=8,noatime /dev/md0
/mnt/raid10xfs

Will my files be safe even on sudden power loss? Is barrier=1 enough?
Do i need to disable the write cache?
with: hdparm -W0 /dev/sdb /dev/sdc /dev/sdd

I tried it but performance is horrendous.

Am I better of with ext4? Data safety/integrity is the priority and
optimization affecting it is not acceptable.

Thanks and any advice/guidance would be appreciated
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

I think today you're most safe with old ext3. Maybe data=journal is not
best idea, because it has small user-base. The more aggressive caching
and features leading to awesome performance will bite you harder by
power loss or software bugs.

Limit power loss with UPS unit. You need to cleanly shutdown the system
when UPS reaches it's low level capacity. That's mandatory IMO and still
inexpensive. Next you can think about multiple PSU/UPS units...

But just think in a way that you will never make it 100% safe, so use
damn good backups!

HTH, Z.

quoted

I often recommend aligning partitions to 1GiB, also because I
like to have 1GiB or so of empty space at the very beginning
and end of a drive.

I'm really curious why do you use such alignment? I can think
about few reasons, but most practical I think you like to
slice in gigabyte sizes.

Indeed, and to summarize:

* As mentioned before, I usually leave a chunk of unused space
  at the very start and end of a drive. This also because:
  - Many head landing accidents happen at the start or end of a
    drive.
  - Free space at the start: sometimes it is useful to have a
    few seconds of grace at the start when duplicating a drive
    to realize one has mistyped the name, and many partitioning
    or booting schemes can use a bit of free space at the start.
    Consider XFS and its use of sector 0.
  - Free space at the end: many drives have slightly different
    sizes, and this can cause problems when for example
    rebuilding arrays, or doing backups, and leaving some bit
    unused can avoid a lot of trouble.

* Having even sizes for partitions means that it may be easier
  to image copy them from one drive to another. I often to do
  that. Indeed I usually create partitions of a few "standard"
  sizes, usually tailored to fit drives that tend also to come
  in fairly standard increments, because drive manufacturers in
  each new platter generation usually aim at a fairly standard
  factor of improvement. Standard drive sizes tend to be, in
  gigabytes: 80 160 250 500 1000 1500 2000 3000. Since 80 and
  160 are somewhat old and no longer used, I currently tend to
  do partitions in sizes like 230GiB, 460GiB, 920GiB etc. SSDs
  have somewhat complicated this.

* On a contemporary drive 1GiB is a rather small fraction of the
  capacity of a drive, so why not just align everything to 1GiB,
  even if it seems pretty large in absolute terms? And if you
  align the first partition to start at 1GiB, and leave free
  space at the end, it is farily natural to align everything in
  between on 1GiB boundaries.

In this as in many other cases I like to buy myself some extra
degrees of freedom if they are cheap.

Another example I have written about previously is specifying
advisedly chosen 'sunit' and 'swidth' even on non-RAID volumes,
or non-parity RAID setups, not because they really improve
things, but because the cost is minimal and it might come useful
later.

  Note: I *really* like to be able to do partition image copies,
    because they are so awesomely faster than treewise ones.