On Sat, 6 Dec 2008, Eric Sandeen wrote:

Justin Piszcz wrote:

quoted

Someone should write a document with XFS and barrier support, if I recall,
in the past, they never worked right on raid1 or raid5 devices, but it
appears now they they work on RAID1, which slows down performance ~12 times!!

quoted

There is some mention of it here:
http://oss.sgi.com/projects/xfs/faq.html#wcache_persistent

But basically I believe it should be noted in the kernel logs, FAQ or somewhere
because just through the process of upgrading the kernel, not changing fstab
or any other part of the system, performance can drop 12x just because the
newer kernels implement barriers.

Perhaps:

printk(KERN_ALERT "XFS is now looking after your metadata very
carefully; if you prefer the old, fast, dangerous way, mount with -o
nobarrier\n");

:)

Really, this just gets xfs on md raid1 in line with how it behaves on
most other devices.

But I agree, some documentation/education is probably in order; if you
choose to disable write caches or you have faith in the battery backup
of your write cache, turning off barriers would be a good idea.  Justin,
it might be interesting to do some tests with:

barrier,   write cache enabled
nobarrier, write cache enabled
nobarrier, write cache disabled

a 12x hit does hurt though...  If you're really motivated, try the same
scenarios on ext3 and ext4 to see what the barrier hit is on those as well.

-Eric

No, I have not forgotten about this I have just been quite busy, I will test
this now, as before, I did not use sync because I was in a hurry and did not
have the ability to test, I am using a different machine/hw type but the
setup is the same, md/raid1 etc.

Since I will only be measuring barriers, per esandeen@ I have changed the mount
options from what I typically use to the defaults.

Here is the /etc/fstab entry:
/dev/md2        /               xfs     defaults        0       1

And the nobarrier entry:
/dev/md2        /               xfs     defaults,nobarrier        0       1

Stop cron and make sure nothing else is using the disk I/O, done:

# /etc/init.d/cron stop
Stopping periodic command scheduler: crond.

The benchmark:
# /usr/bin/time bash -c 'tar xf linux-2.6.27.8.tar; sync'
# echo 1 > /proc/sys/vm/drop_caches # (between tests)

== The tests ==

  KEY:
  barriers = "b"
  write_cache = "w"

  SUMMARY:
   b=on,w=on: 1:19.53 elapsed @ 2% CPU [BENCH_1]
  b=on,w=off: 1:23.59 elapsed @ 2% CPU [BENCH_2]
  b=off,w=on: 0:21.35 elapsed @ 9% CPU [BENCH_3]
b=off,w=off: 0:42.90 elapsed @ 4% CPU [BENCH_4]

So it depends on your settings as far as how slow barriers affect the I/O.

Scheduler used: CFQ.
[    0.168390] io scheduler cfq registered (default)

The raw details:

BENCH_1
# /usr/bin/time bash -c 'tar xf linux-2.6.27.8.tar; sync'
0.16user 1.85system 1:19.53elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+970minor)pagefaults 0swaps

BENCH_2
(turn off write-cache)

# hdparm -W0 /dev/sda

/dev/sda:
  setting drive write-caching to 0 (off)
  write-caching =  0 (off)
# hdparm -W0 /dev/sdb

/dev/sdb:
  setting drive write-caching to 0 (off)
  write-caching =  0 (off)
#

# /usr/bin/time bash -c 'tar xf linux-2.6.27.8.tar; sync'
0.16user 1.86system 1:23.59elapsed 2%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (14major+953minor)pagefaults 0swaps

BENCH_3
(barriers=off; write_cache=on)
# /usr/bin/time bash -c 'tar xf linux-2.6.27.8.tar; sync'
0.18user 1.86system 0:21.35elapsed 9%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (14major+952minor)pagefaults 0swaps

BENCH_4
(turn off write-cache)

# hdparm -W0 /dev/sda

/dev/sda:
  setting drive write-caching to 0 (off)
  write-caching =  0 (off)
# hdparm -W0 /dev/sdb

/dev/sdb:
  setting drive write-caching to 0 (off)
  write-caching =  0 (off)
#

# /usr/bin/time bash -c 'tar xf linux-2.6.27.8.tar; sync'
0.18user 1.76system 0:42.90elapsed 4%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (14major+954minor)pagefaults 0swaps