Re: raid5 slow
6 messages,
2 authors,
2004-12-02 · open the first message on its own page
On Tuesday November 30, raid@a2000.nu wrote: storage.a2000.nu,500G,23385,97,79328,69,59189,58,25015,91,122493,58,67.7,11,16,1404,76,+++++,+++,+++++,+++,1347,75,+++++,+++,4868,97
it's really slow... :(
is there anything i can do ?
i want to fill the raid asap
for now i have 2 options :
-use the 2.6 and hope someone is going to fix this ?
-use the 2.4 kernel but max 13 devices (there is no LBD support in 2.4?)
or is there a LBD patch so i can create a 2TB+ raid5 array ?
I don't think there is LBD support for 2.4.
I haven't managed to duplicate the degree of slowdown that you report
- I only get about 10%.
However I found an old Email which also reported slow raid5 in 2.6 and
it noted that when the array is degraded, it goes faster ....
I just tried it and got most of the speed back when the array is
degraded!!!
Would you be able to test that option?
I will look into the code and see if I can figure out what is going
on.
NeilBrown
I haven't managed to duplicate the degree of slowdown that you report
- I only get about 10%.
However I found an old Email which also reported slow raid5 in 2.6 and
it noted that when the array is degraded, it goes faster ....
I just tried it and got most of the speed back when the array is
degraded!!!
Would you be able to test that option?
2 tests :
(kernel 2.6.10-rc2 13d r5 ext2)
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storage.a2000.nu 2G 25238 99 116958 98 67987 73 25702 91 142010 70
608.2 3
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 1354 78 +++++ +++ +++++ +++ 1262 74 +++++ +++
4895 95
storage.a2000.nu,2G,25238,99,116958,98,67987,73,25702,91,142010,70,608.2,3,16,1354,78,+++++,+++,+++++,+++,1262,74,+++++,+++,4895,95
]# mdadm --manage --fail /dev/md0 /dev/sdn1
mdadm: set /dev/sdn1 faulty in /dev/md0
]# mdadm --manage --remove /dev/md0 /dev/sdn1
mdadm: hot removed /dev/sdn1
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storage.a2000.nu 2G 25458 99 107295 83 59079 60 26079 93 145775 77
490.2 3
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 1423 83 +++++ +++ +++++ +++ 1453 82 +++++ +++
4872 100
storage.a2000.nu,2G,25458,99,107295,83,59079,60,26079,93,145775,77,490.2,3,16,1423,83,+++++,+++,+++++,+++,1453,82,+++++,+++,4872,100
On Wed, 1 Dec 2004, Neil Brown wrote:
I don't think there is LBD support for 2.4.
found this :
https://www.gelato.unsw.edu.au/archives/lbd/2004-February/000039.html
http://www.gelato.unsw.edu.au/patches/
'Theoretically raid0 and linear should work; I'm certain raid1, raid4 and
raid5 will not work. In any case, RAID is limited to a maximum of 2TB
minus one block per member. '
i don't think i want this on my raid5....
today runned a test on another machine
(dual p3-733 with 4*9gb scsi)
looks like on this system the difference is arround the 10%
maybe you need more disks/faster disks to get a bigger difference
results :
2.6.10-rc2 4d r5 ext3
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
linuxtest.expl 496M 10364 99 39689 55 22184 36 11485 96 52559 46
632.0 4
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 512 99 +++++ +++ 32502 99 528 99 +++++ +++
1938 97
linuxtest.explainerdc.local,496M,10364,99,39689,55,22184,36,11485,96,52559,46,632.0,4,16,512,99,+++++,+++,32502,99,528,99,+++++,+++,1938,97
2.4.20 4d r5 ext3
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
linuxtest.expl 496M 10087 100 53021 63 20934 34 11746 98 58858 34
665.3 4
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 573 99 +++++ +++ 30886 99 592 99 +++++ +++
2264 96
linuxtest.explainerdc.local,496M,10087,100,53021,63,20934,34,11746,98,58858,34,665.3,4,16,573,99,+++++,+++,30886,99,592,99,+++++,+++,2264,96
and another system
P4 2.4Ghz with 8*120gb on 3ware 7805
pci32 so the 2.4 looks like limited by the pci bus :
(2.6.8 8d r5 ext3)
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storage 1G 507 98 64149 20 33240 11 984 97 95124 18
663.0 2
Latency 105ms 1039ms 879ms 29443us 63097us
870ms
Version 1.93c ------Sequential Create------ --------Random
Create--------
storage -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 1849 64 +++++ +++ 10044 9 1570 54 +++++ +++
4406 49
Latency 64064us 184us 188us 133ms 102us
293us
1.93c,1.93c,storage,1,1101940252,1G,,507,98,64149,20,33240,11,984,97,95124,18,663.0,2,16,,,,,1849,64,+++++,+++,10044,9,1570,54,+++++,+++,4406,49,105ms,1039ms,879ms,29443us,63097us,870ms,64064us,184us,188us,133ms,102us,293us
(2.4.28 8d r5 ext3)
Version 1.93c ------Sequential Output------ --Sequential Input-
--Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
storage 1G 98 99 64449 44 42805 21 742 98 113930 29
732.5 4
Latency 129ms 809ms 819ms 23221us 139ms
754ms
Version 1.93c ------Sequential Create------ --------Random
Create--------
storage -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 1526 61 +++++ +++ 8337 20 1940 78 +++++ +++
2441 36
Latency 175ms 39us 83us 78763us 20us
77558us
1.93c,1.93c,storage,1,1101940608,1G,,98,99,64449,44,42805,21,742,98,113930,29,732.5,4,16,,,,,1526,61,+++++,+++,8337,20,1940,78,+++++,+++,2441,36,129ms,809ms,819ms,23221us,139ms,754ms,175ms,39us,83us,78763us,20us,77558us
and another system
p4 2.8Ghz with 2*300gb (normal ide ports)
i created a degraded raid5
25% difference in read performance :
2.4.20 :
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
gx270.a2000.nu 1G 29724 79 41447 15 17575 6 33146 80 55494 12
267.6 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 2539 98 +++++ +++ +++++ +++ 2523 98 +++++ +++
6307 96
gx270.a2000.nu,1G,29724,79,41447,15,17575,6,33146,80,55494,12,267.6,0,16,2539,98
,+++++,+++,+++++,+++,2523,98,+++++,+++,6307,96
2.6.9-ac11 :
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP
gx270.a2000.nu 1G 21190 52 37804 8 10092 2 34726 80 68587 9
263.1 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 3305 99 +++++ +++ +++++ +++ 3372 100 +++++ +++
10428 98
gx270.a2000.nu,1G,21190,52,37804,8,10092,2,34726,80,68587,9,263.1,0,16,3305,99,+
++++,+++,+++++,+++,3372,100,+++++,+++,10428,98