Severe slowdown with LVM on RAID, alignment problem?

Started by Michael Guntsche on Feb 29, 2008

open the first message on its own page

Message details and related content
Hello list,

I know the subject about lvm on raid and the very popular alignment question have been beaten to death by now.
But whatever I try I cannot seem to work it out, so please bear with me and my lengthy description.

Currently I am testing a SATA RAID-5 for deployment in my little home server. I am interested in space mostly but still do not want to give up all the speed.
So for testing purposes I created a 10GB partition on every disk and created a RAID-5 across them.

mdadm --detail /dev/md0
/dev/md0:
        Version : 01.00.03
  Creation Time : Thu Feb 28 18:56:07 2008
     Raid Level : raid5
     Array Size : 29326080 (27.97 GiB 30.03 GB)
  Used Dev Size : 19550720 (9.32 GiB 10.01 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Feb 29 00:40:44 2008
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 256K

           Name : gibson:0  (local to host gibson)
           UUID : 46cb01e4:ea969e20:535d91a1:234004ed
         Events : 8

    Number   Major   Minor   RaidDevice State
       0       8       33        0      active sync   /dev/sdc1
       1       8       49        1      active sync   /dev/sdd1
       2       8       65        2      active sync   /dev/sde1
       4       8       81        3      active sync   /dev/sdf1

I set the Cunk Size to 256K as you can see and also used Superblock 1.0, apart from that I changed nothing. I let the RAID resync and then I started testing.

First one was a simple XFS FS sans LVM. I called mkfs.xfs without any parameters and got the following.

 mkfs.xfs /dev/md0
meta-data=/dev/md0               isize=256    agcount=16, agsize=458240 blks
         =                       sectsz=4096  attr=2
data     =                       bsize=4096   blocks=7331520, imaxpct=25
         =                       sunit=64     swidth=192 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=3579, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=786432 blocks=0, rtextents=0

Running a quick bonnie benchmark on this one, I got the following results.

Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1

Well this is not the fastest machine, so I did not think much about the results, in my opinion the look ok, but please correct me if I am totally wrong.

Next I created VG-LV on the raid, formatted it with XFS and ran bonny again.

!! No special switches where used for vgcreate or lvcreate !!


Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1
lvm-nal          1G           28917  21 15344  19           33645  26 136.3   1

We lost a little bit on writing, but what happened to reading? I lost more than half of the performance here.

So I had a look at mkfs.xfs again and saw that sunit and swidth where 0. Looking at the values I got the first time I called mkfs.xfs on the /dev/md0 device I came up with

mkfs.xfs -dsunit=512,swidth=1536 /dev/mapper/lamo-gabo -f
meta-data=/dev/mapper/lamo-gabo  isize=256    agcount=4, agsize=655360 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=2621440, imaxpct=25
         =                       sunit=64     swidth=192 blks
naming   =version 2              bsize=4096
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=64 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Result:

Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1
lvm-nal          1G           28917  21 15344  19           33645  26 136.3   1
xfsopt-lvm-nal   1G           29503  22 15369  19           36604  30 135.0   1

Ok, some improvement but still not really satisfying.

Reading through the archives I also found hints to improve performance by aligning the VG to the stripe/chunk size. Although only talking about writing I tried that.

I came up with.... The LVM header is 64K my chunksize 256, so 256-64=192. I called pvcreate --metadatasize 192k and recreated the VG-LV combo, formatted it again with plain mkfs.xfs (no sunit or swidth) and ran bonnie again.

Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1
lvm-nal          1G           28917  21 15344  19           33645  26 136.3   1
xfsopt-lvm-nal   1G           29503  22 15369  19           36604  30 135.0   1
lvm-al           1G           29561  21 15857  20           35242  28 130.6   1

A little bit better but not really that much.

Finally I reformatted it with the sunit, and swidth values from above.

Version  1.03c      ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xfs              1G           31359  20 24135  19           79587  29 154.8   1
lvm-nal          1G           28917  21 15344  19           33645  26 136.3   1
xfsopt-lvm-nal   1G           29503  22 15369  19           36604  30 135.0   1
lvm-al           1G           29561  21 15857  20           35242  28 130.6   1
xfsopt-lvm-al    1G           30948  22 15404  19           33715  26 118.4   1

Nope still not doing great here.

Ok, this is where I am at now.
Do I really loose that much reading speed just by using lvm on RAID? I would have understood some decrease in writing speed but why is reading affected that much?

Once again, sorry for the long E-Mail, but this is really bugging me right now.

Kind regards,
Michael
 
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help