Lately, I had a serious performance problem on a 1.2 TB ext3 filesystem
placed on LVM, which was placed on Linux RAID-5.
Luckily, it was possible to alleviate much of this problem by removing
internal bitmap from RAID-5 and by using anticipatory IO scheduler.
As suggested earlier by Theodore Tso in "very poor ext3 write
performance on big filesystems" thread on linux-fsdevel list, I should
have used "mke2fs -E stride option to make sure all of the bitmaps
don't get concentrated on one hard drive spindle".
Although the fine mkfs.ext3 manual gives some basic information on how
to do it if you place the whole filesystem on a RAID array, it is not
clear to me how it should be done correctly if I want to create a ext3
filesystem on LVM on RAID-5.
Any helpful hints?
--
Tomasz Chmielewski
http://wpkg.org
On Thu, Feb 28, 2008 at 3:38 AM, Tomasz Chmielewski [off-list ref] wrote:
Lately, I had a serious performance problem on a 1.2 TB ext3 filesystem
placed on LVM, which was placed on Linux RAID-5.
Luckily, it was possible to alleviate much of this problem by removing
internal bitmap from RAID-5 and by using anticipatory IO scheduler.
As suggested earlier by Theodore Tso in "very poor ext3 write
performance on big filesystems" thread on linux-fsdevel list, I should
have used "mke2fs -E stride option to make sure all of the bitmaps
don't get concentrated on one hard drive spindle".
Although the fine mkfs.ext3 manual gives some basic information on how
to do it if you place the whole filesystem on a RAID array, it is not
clear to me how it should be done correctly if I want to create a ext3
filesystem on LVM on RAID-5.
Any helpful hints?
I can't help with the stride stuff, as I use LVM and that makes the
stride stuff utterly useless anyway. However, a suggestion I have for
you which I found helped a great deal was to place the bitmap on some
/other/ device's filesystem (I used a /etc/bitmaps/). Obviously, don't
put it on a FS which is part of the raid, I typically divide each
device into 2 or more raids, but only if I don't have enough devices
for /system/ and /data/ -style pools.
--
Jon
Tomasz Chmielewski wrote:
Although the fine mkfs.ext3 manual gives some basic information on how
to do it if you place the whole filesystem on a RAID array, it is not
clear to me how it should be done correctly if I want to create a ext3
filesystem on LVM on RAID-5.
Any helpful hints?
You should be careful with all the alignemnts. Extents should be properly
aligned with - at the very least - stripes, and preferably with whole stripe
width (not always possible though). To check where lvm extents, you can use
i.e.:
dmsetup table
pvs -o+pe_start --units s
losetup and hexedit experiments (silly, but I used to do it like that in the
past, before I realised about other options :o )
If you don't need default 4M granularity, it's nice to increase it (vgcreate
-s option ; I use 256M and 512M sized extents). You can alter the beginning
of the extents with --metadatasize option of pvcreate (overally it always
rounds up to the next 64K multiple). Be sure to verify with one of the
commands above.
Moreover - if you use partitionable raid and create lvm in one of the
partitions, make sure partition is properly aligned as well.