Severe slowdown with LVM on RAID, alignment problem?

From
Michael Guntsche <hidden>
Date
2008-02-29 00:05:55
Hello list,

I know the subject about lvm on raid and the very popular alignment question have been beaten to death by now.
But whatever I try I cannot seem to work it out, so please bear with me and my lengthy description.

Currently I am testing a SATA RAID-5 for deployment in my little home server. I am interested in space mostly but still do not want to give up all the speed.
So for testing purposes I created a 10GB partition on every disk and created a RAID-5 across them.

mdadm --detail /dev/md0
/dev/md0:
        Version : 01.00.03
  Creation Time : Thu Feb 28 18:56:07 2008
     Raid Level : raid5
     Array Size : 29326080 (27.97 GiB 30.03 GB)
  Used Dev Size : 19550720 (9.32 GiB 10.01 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Feb 29 00:40:44 2008
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 256K

           Name : gibson:0  (local to host gibson)
           UUID : 46cb01e4:ea969e20:535d91a1:234004ed
         Events : 8

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       2       8       65        2      active sync   /dev/sde1
       4       8       81        3      active sync   /dev/sdf1

I set the Cunk Size to 256K as you can see and also used Superblock 1.0, apart from that I changed nothing. I let the RAID resync and then I started testing.

First one was a simple XFS FS sans LVM. I called mkfs.xfs without any parameters and got the following.

 mkfs.xfs /dev/md0
meta-data=/dev/md0               isize=256    agcount=16, agsize=458240 blks
         =                       sectsz=4096  attr=2
data     =                       bsize=4096   blocks=7331520, imaxpct=25
         =                       sunit=64     swidth=192 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=3579, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=786432 blocks=0, rtextents=0

Running a quick bonnie benchmark on this one, I got the following results.

Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1

Well this is not the fastest machine, so I did not think much about the results, in my opinion the look ok, but please correct me if I am totally wrong.

Next I created VG-LV on the raid, formatted it with XFS and ran bonny again.

!! No special switches where used for vgcreate or lvcreate !!


Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1
lvm-nal          1G           28917  21 15344  19           33645  26 136.3   1

We lost a little bit on writing, but what happened to reading? I lost more than half of the performance here.

So I had a look at mkfs.xfs again and saw that sunit and swidth where 0. Looking at the values I got the first time I called mkfs.xfs on the /dev/md0 device I came up with

mkfs.xfs -dsunit=512,swidth=1536 /dev/mapper/lamo-gabo -f
meta-data=/dev/mapper/lamo-gabo  isize=256    agcount=4, agsize=655360 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=64     swidth=192 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Result:

Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1
lvm-nal          1G           28917  21 15344  19           33645  26 136.3   1
xfsopt-lvm-nal   1G           29503  22 15369  19           36604  30 135.0   1

Ok, some improvement but still not really satisfying.

Reading through the archives I also found hints to improve performance by aligning the VG to the stripe/chunk size. Although only talking about writing I tried that.

I came up with.... The LVM header is 64K my chunksize 256, so 256-64=192. I called pvcreate --metadatasize 192k and recreated the VG-LV combo, formatted it again with plain mkfs.xfs (no sunit or swidth) and ran bonnie again.

Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1
lvm-nal          1G           28917  21 15344  19           33645  26 136.3   1
xfsopt-lvm-nal   1G           29503  22 15369  19           36604  30 135.0   1
lvm-al           1G           29561  21 15857  20           35242  28 130.6   1

A little bit better but not really that much.

Finally I reformatted it with the sunit, and swidth values from above.

Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1
lvm-nal          1G           28917  21 15344  19           33645  26 136.3   1
xfsopt-lvm-nal   1G           29503  22 15369  19           36604  30 135.0   1
lvm-al           1G           29561  21 15857  20           35242  28 130.6   1
xfsopt-lvm-al    1G           30948  22 15404  19           33715  26 118.4   1

Nope still not doing great here.

Ok, this is where I am at now.
Do I really loose that much reading speed just by using lvm on RAID? I would have understood some decrease in writing speed but why is reading affected that much?

Once again, sorry for the long E-Mail, but this is really bugging me right now.

Kind regards,
Michael
 

Attachments

Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help