Re: raid10n2/xfs setup guidance on write-cache/barrier

On Thu, Mar 15, 2012 at 1:38 PM, Stan Hoeppner [off-list ref] wrote:

On 3/14/2012 7:30 PM, Jessie Evangelista wrote:

quoted

I want to create a raid10,n2 using 3 1TB SATA drives.
I want to create an xfs filesystem on top of it.
The filesystem will be used as NFS/Samba storage.

mdadm --zero /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm -v --create /dev/md0 --metadata=1.2 --assume-clean
--level=raid10 --chunk 256 --raid-devices=3 /dev/sdb1 /dev/sdc1
/dev/sdd1

Why 256KB for chunk size?

For reference, the machine has 16GB memory

I've run some benchmarks with dd trying the different chunks and 256k
seems like the sweetspot.
dd if=/dev/zero of=/dev/md0 bs=64k count=655360 oflag=direct

Looks like you've been reading a very outdated/inaccurate "XFS guide" on
the web...

What kernel version?  This can make a significant difference in XFS
metadata performance.  You should use 2.6.39+ if possible.  What
xfsprogs version?

testing was done with ubuntu 10.04LTS with kernel at 2.6.32-33-server
xfsprogs at 3.1.0ubuntu1

quoted

mkfs -t xfs -l lazy-count=1,size=128m -f /dev/md0

lazy-count=1 is currently the default with recent xfsprogs so no need to
specify it.  Why are you manually specifying the size of the internal
journal log file?  This is unnecessary.  In fact, unless you have
profiled your workload and testing shows that alternate XFS settings
perform better, it is always best to stick with the defaults.  They
exist for a reason, and are well considered.

I'll probably forgo setting the journal log file size. It seemed like
a safe optimization from what I've read.

quoted

mount -t xfs -o barrier=1,logbsize=256k,logbufs=8,noatime /dev/md0
/mnt/raid10xfs

Barrier has no value, it's either on or off.  XFS mounts with barriers
enabled by default so remove 'barrier=1'.  You do not have a RAID card
with persistent write cache (BBWC), so you should leave barriers
enabled.  Barriers mitigate journal log corruption due to power failure
and crashes, which seem seem to be of concern to you.

logbsize=256k and logbufs=8 are the defaults in recent kernels so no
need to specify them.  Your NFS/Samba workload on 3 slow disks isn't
sufficient to need that much in memory journal buffer space anyway.  XFS
uses relatime which is equivalent to noatime WRT IO reduction
performance, so don't specify 'noatime'.

I just wanted to be explicit about it so that I know what is set just
in case the defaults change

In fact, it appears you don't need to specify anything in mkfs.xfs or
fstab, but just use the defaults.  Fancy that.  And the one thing that
might actually increase your performance a little bit you didn't
specify--sunit/swidth.  However, since you're using mdraid, mkfs.xfs
will calculate these for you (which is nice as mdraid10 with odd disk
count can be a tricky calculation).  Again, defaults work for a reason.

The reason I did not set sunit/swidth is because I read somewhere that
mkfs.xfs will calculate based on mdraid.

quoted

Will my files be safe even on sudden power loss?

Are you unwilling to purchase a UPS and implement shutdown scripts?  If
so you have no business running a server, frankly.  Any system will lose
data due to power loss, it's just a matter of how much based on the
quantity of inflight writes at the time the juice dies.  This problem is
mostly filesytem independent.  Application write behavior does play a
role.  UPS with shutdown scripts, and persistent write cache prevent
this problem.  A cheap UPS suitable for this purpose is less money than
a 1TB 7.2k drive, currently.

The server is for a non-profit org that I am helping out.
I think a APC Smart-UPS SC 420VA 230V may fit their shoe string budget.

You say this is an NFS/Samba server.  That would imply that multiple
people or other systems directly rely on it.  Implement a good UPS
solution and eliminate this potential problem.

quoted

Is barrier=1 enough?
Do i need to disable the write cache?
with: hdparm -W0 /dev/sdb /dev/sdc /dev/sdd

Disabling drive write caches does decrease the likelihood of data loss.

quoted

I tried it but performance is horrendous.

And this is why you should leave them enabled and use barriers.  Better
yet, use a RAID card with BBWC and disable the drive caches.

Budget does not allow for RAID card with BBWC

quoted

Am I better of with ext4? Data safety/integrity is the priority and
optimization affecting it is not acceptable.

You're better off using a UPS.  Filesystem makes little difference WRT
data safety/integrity.  All will suffer some damage if you throw a
grenade at them.  So don't throw grenades.  Speaking of which, what is
your backup/restore procedure/hardware for this array?

nightly backups will be stored on an external USB disk
is xfs going to be prone to more data loss in case the non-redundant
power supply goes out?

quoted

Thanks and any advice/guidance would be appreciated

I'll appreciate your response stating "Yes, I have a UPS and
tested/working shutdown scripts" or "I'll be implementing a UPS very
soon." :)

I don't have shutdown scripts yet but will look into it.
Meatware would have to do for now as the server will probably be ON
only when there's people at the office. And yes I will be asking them
to not go into production without a UPS

--
Stan

Thanks for you input Stan.

I just updated the kernel to 3.0.0-16.
Did they take out barrier support in mdraid? or was the implementation
replaced with FUA?
Is there a definitive test to determine if the off the shelf consumer
sata drives honor barrier or cache flush requests?

I think I'd like to go with device cache turned ON and barrier enabled.

Am still torn between ext4 and xfs i.e. which will be safer in this
particular setup.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html