On Mon, Aug 29, 2005 at 06:20:56PM +0000, Holger Kiehl wrote:
quoted
Hello
I have a system with the following setup:
Board is Tyan S4882 with AMD 8131 Chipset
4 Opterons 848 (2.2GHz)
8 GB DDR400 Ram (2GB for each CPU)
1 onboard Symbios Logic 53c1030 dual channel U320 controller
2 SATA disks put together as a SW Raid1 for system, swap and spares
8 SCSI U320 (15000 rpm) disks where 4 disks (sdc, sdd, sde, sdf)
are on one channel and the other four (sdg, sdh, sdi, sdj) on
the other channel.
The U320 SCSI controller has a 64 bit PCI-X bus for itself, there is
no other device on that bus. Unfortunatly I was unable to determine at
what speed it is running, here the output from lspci -vv:
quoted
How does one determine the PCI-X bus speed?
Usually only the card (in your case the Symbios SCSI controller) can
tell. If it does, it'll be most likely in 'dmesg'.
Anyway, I thought with this system I would get theoretically 640 MB/s using
both channels.
You can never use the full theoretical bandwidth of the channel for
data. A lot of overhead remains for other signalling. Similarly for PCI.
quoted
I tested several software raid setups to get the best possible write
speeds for this system. But testing shows that the absolute maximum I
can reach with software raid is only approx. 270 MB/s for writting.
Which is very disappointing.
I'd expect somewhat better (in the 300-400 MB/s range), but this is not
too bad.
To find where the bottleneck is, I'd suggest trying without the
filesystem at all, and just filling a large part of the block device
using the 'dd' command.
Also, trying without the RAID, and just running 4 (and 8) concurrent
dd's to the separate drives could show whether it's the RAID that's
slowing things down.
Ok, I did run the following dd command in different combinations:
dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000
Here the results:
Each disk alone
/dev/sdc1 59.094636 MB/s
/dev/sdd1 58.686592 MB/s
/dev/sde1 55.282807 MB/s
/dev/sdf1 62.271240 MB/s
/dev/sdg1 60.872891 MB/s
/dev/sdh1 62.252781 MB/s
/dev/sdi1 59.145637 MB/s
/dev/sdj1 60.921119 MB/s
sdc + sdd in parallel (2 disks on same channel)
/dev/sdc1 42.512287 MB/s
/dev/sdd1 43.118483 MB/s
sdc + sdg in parallel (2 disks on different channels)
/dev/sdc1 42.938186 MB/s
/dev/sdg1 43.934779 MB/s
sdc + sdd + sde in parallel (3 disks on same channel)
/dev/sdc1 35.043501 MB/s
/dev/sdd1 35.686878 MB/s
/dev/sde1 34.580457 MB/s
Similar results for three disks (sdg + sdh + sdi) on the other channel
/dev/sdg1 36.381137 MB/s
/dev/sdh1 37.541758 MB/s
/dev/sdi1 35.834920 MB/s
sdc + sdd + sde + sdf in parallel (4 disks on same channel)
/dev/sdc1 31.432914 MB/s
/dev/sdd1 32.058752 MB/s
/dev/sde1 31.393455 MB/s
/dev/sdf1 33.208165 MB/s
And here for the four disks on the other channel
/dev/sdg1 31.873028 MB/s
/dev/sdh1 33.277193 MB/s
/dev/sdi1 31.910000 MB/s
/dev/sdj1 32.626744 MB/s
All 8 disks in parallel
/dev/sdc1 24.120545 MB/s
/dev/sdd1 24.419801 MB/s
/dev/sde1 24.296588 MB/s
/dev/sdf1 25.609548 MB/s
/dev/sdg1 24.572617 MB/s
/dev/sdh1 25.552590 MB/s
/dev/sdi1 24.575616 MB/s
/dev/sdj1 25.124165 MB/s
So from these results, I may assume that md is not the cause of the problem.
What comes as a big surprise is that I loose 25% performance with only
two disks and each hanging on its own channel!
Is this normal? I wonder if other people have the same problem with
other controllers or the same.
What can I do next to find out if this is a kernel, driver or hardware
problem?
Thanks,
Holger
To find where the bottleneck is, I'd suggest trying without the
filesystem at all, and just filling a large part of the block device
using the 'dd' command.
Also, trying without the RAID, and just running 4 (and 8) concurrent
dd's to the separate drives could show whether it's the RAID that's
slowing things down.
Ok, I did run the following dd command in different combinations:
dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000
I think a bs of 4k is way too small and will cause huge CPU overhead.
Can you try with something like 4M? Also, you can use /dev/full to avoid
the pre-zeroing.
Here the results:
Each disk alone
/dev/sdc1 59.094636 MB/s
/dev/sdd1 58.686592 MB/s
/dev/sde1 55.282807 MB/s
/dev/sdf1 62.271240 MB/s
/dev/sdg1 60.872891 MB/s
/dev/sdh1 62.252781 MB/s
/dev/sdi1 59.145637 MB/s
/dev/sdj1 60.921119 MB/s
You're saturating some bus. It almost looks like it's the PCI-X,
although that should be able to deliver up (if running at the full speed
of AMD8132) up to 1GB/sec, so it SHOULD not be an issue.
So from these results, I may assume that md is not the cause of the problem.
What comes as a big surprise is that I loose 25% performance with only
two disks and each hanging on its own channel!
Is this normal? I wonder if other people have the same problem with
other controllers or the same.
No, I don't think this is OK.
What can I do next to find out if this is a kernel, driver or hardware
problem?
You need to find where the bottleneck is, by removing one possible
bottleneck at a time in your test.
--
Vojtech Pavlik
SuSE Labs, SuSE CR
To find where the bottleneck is, I'd suggest trying without the
filesystem at all, and just filling a large part of the block device
using the 'dd' command.
Also, trying without the RAID, and just running 4 (and 8) concurrent
dd's to the separate drives could show whether it's the RAID that's
slowing things down.
Ok, I did run the following dd command in different combinations:
dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000
I think a bs of 4k is way too small and will cause huge CPU overhead.
Can you try with something like 4M? Also, you can use /dev/full to avoid
the pre-zeroing.
That was my initial thought as well, but since he's writing the io side
should look correct. I doubt 8 dd's writing 4k chunks will gobble that
much CPU as to make this much difference.
Holger, we need vmstat 1 info while the dd's are running. A simple
profile would be nice as well, boot with profile=2 and do a readprofile
-r; run tests; readprofile > foo and send the first 50 lines of foo to
this list.
--
Jens Axboe
To find where the bottleneck is, I'd suggest trying without the
filesystem at all, and just filling a large part of the block device
using the 'dd' command.
Also, trying without the RAID, and just running 4 (and 8) concurrent
dd's to the separate drives could show whether it's the RAID that's
slowing things down.
Ok, I did run the following dd command in different combinations:
dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000
I think a bs of 4k is way too small and will cause huge CPU overhead.
Can you try with something like 4M? Also, you can use /dev/full to avoid
the pre-zeroing.
That was my initial thought as well, but since he's writing the io side
should look correct. I doubt 8 dd's writing 4k chunks will gobble that
much CPU as to make this much difference.
Holger, we need vmstat 1 info while the dd's are running. A simple
profile would be nice as well, boot with profile=2 and do a readprofile
-r; run tests; readprofile > foo and send the first 50 lines of foo to
this list.
Ok, I did run the following dd command in different combinations:
dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000
I think a bs of 4k is way too small and will cause huge CPU overhead.
Can you try with something like 4M? Also, you can use /dev/full to avoid
the pre-zeroing.
That was my initial thought as well, but since he's writing the io side
should look correct. I doubt 8 dd's writing 4k chunks will gobble that
much CPU as to make this much difference.
Holger, we need vmstat 1 info while the dd's are running. A simple
profile would be nice as well, boot with profile=2 and do a readprofile
-r; run tests; readprofile > foo and send the first 50 lines of foo to
this list.
Here vmstat for 8 dd's still with 4k blocksize:
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
9 2 5244 38272 7738248 10400 0 0 3 11444 390 24 0 5
75 20
5 10 5244 30824 7747680 8684 0 0 0 265672 2582 1917 1 95
0 4
2 12 5244 30948 7747248 8708 0 0 0 222620 2858 292 0 33
0 67
4 10 5244 31072 7747516 8644 0 0 0 236400 3132 326 0 43
0 57
2 12 5244 31320 7747792 8512 0 0 0 250204 3225 285 0 37
0 63
1 13 5244 30948 7747412 8552 0 0 24 227600 3261 312 0 41
0 59
2 12 5244 32684 7746124 8616 0 0 0 235392 3219 274 0 32
0 68
[snip]
Looks as expected, nothing too excessive showing up. About 30-40% sys
time, but it should not bog the machine down that much.
And here the profile output (I assume you meant sorted):
Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).
If we still can't get closer to this, it would be interesting to try my
block tracing stuff so we can see what is going on at the queue level.
But lets gather some more info first, since it requires testing -mm.
--
Jens Axboe
Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
If we still can't get closer to this, it would be interesting to try my
block tracing stuff so we can see what is going on at the queue level.
But lets gather some more info first, since it requires testing -mm.
Ok, please then just tell me what I must do.
Thanks,
Holger
From: Dr. David Alan Gilbert <hidden> Date: 2005-08-31 14:24:07
* Holger Kiehl (Holger.Kiehl@dwd.de) wrote:
On Wed, 31 Aug 2005, Jens Axboe wrote:
Full vmstat session can be found under:
Have you got iostat? iostat -x 10 might be interesting to see
for a period while it is going.
Dave
--
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
\ _________________________|_____ http://www.treblig.org |_______/
There seems to be quite a spread of read performance accross the drives
(pretty consistent accross the run); what makes sdg so much slower than
sdf (which seems to be the slowest and fastest drives respectively).
I guess if everyone was running at sdf's speed you would be pretty happy.
If you physically swap f and g does the performance follow the drive
or the letter?
Dave
--
-----Open up your eyes, open up your mind, open up your code -------
/ Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
\ _________________________|_____ http://www.treblig.org |_______/
Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
I used the following command reading from all 8 disks in parallel:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Here vmstat output (I just cut something out in the middle):
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----^M
r b swpd free buff cache si so bi bo in cs us sy id
wa^M
3 7 4348 42640 7799984 9612 0 0 322816 0 3532 4987 0 22
0 78
1 7 4348 42136 7800624 9584 0 0 322176 0 3526 4987 0 23
4 74
0 8 4348 39912 7802648 9668 0 0 322176 0 3525 4955 0 22
12 66
1 7 4348 38912 7803700 9636 0 0 322432 0 3526 5078 0 23
Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.
quoted
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
I've attached a little sample for you, just run ala
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
--
Jens Axboe
I have seen an 80GB/sec limitation in the kernel unless this value is
changed in the SCSI I/O layer
for 3Ware and other controllers during testing of 2.6.X series kernels.
Change these values in include/linux/blkdev.h and performance goes from
80MB/S to over 670MB/S on the 3Ware controller.
//#define BLKDEV_MIN_RQ 4
//#define BLKDEV_MAX_RQ 128 /* Default maximum */
#define BLKDEV_MIN_RQ 4096
#define BLKDEV_MAX_RQ 8192 /* Default maximum */
Jeff
Jens Axboe wrote:
On Wed, Aug 31 2005, Holger Kiehl wrote:
quoted
On Wed, 31 Aug 2005, Jens Axboe wrote:
quoted
Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
I used the following command reading from all 8 disks in parallel:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Here vmstat output (I just cut something out in the middle):
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----^M
r b swpd free buff cache si so bi bo in cs us sy id
wa^M
3 7 4348 42640 7799984 9612 0 0 322816 0 3532 4987 0 22
0 78
1 7 4348 42136 7800624 9584 0 0 322176 0 3526 4987 0 23
4 74
0 8 4348 39912 7802648 9668 0 0 322176 0 3525 4955 0 22
12 66
1 7 4348 38912 7803700 9636 0 0 322432 0 3526 5078 0 23
Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.
quoted
quoted
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
I've attached a little sample for you, just run ala
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
------------------------------------------------------------------------
#include <stdio.h>
#include <stdlib.h>
#define __USE_GNU
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
#define BS (131072)
#define ALIGN(buf) (char *) (((unsigned long) (buf) + 4095) & ~(4095))
#define BLOCKS (8192)
int main(int argc, char *argv[])
{
char *p;
int fd, i;
if (argc < 2) {
printf("%s: <dev>\n", argv[0]);
return 1;
}
fd = open(argv[1], O_RDONLY | O_DIRECT);
if (fd == -1) {
perror("open");
return 1;
}
p = ALIGN(malloc(BS + 4095));
for (i = 0; i < BLOCKS; i++) {
int r = read(fd, p, BS);
if (r == BS)
continue;
else {
if (r == -1)
perror("read");
break;
}
}
return 0;
}
From: Tom Callahan <hidden> Date: 2005-08-31 16:58:24
From linux-kernel mailing list.....
Don't do this. BLKDEV_MIN_RQ sets the size of the mempool reserved
requests and will only get slightly used in low memory conditions, so
most memory will probably be wasted.....
Change /sys/block/xxx/queue/nr_requests
Tom Callahan
TESSCO Technologies
(443)-506-6216
callahant@tessco.com
jmerkey wrote:
I have seen an 80GB/sec limitation in the kernel unless this value is
changed in the SCSI I/O layer
for 3Ware and other controllers during testing of 2.6.X series kernels.
Change these values in include/linux/blkdev.h and performance goes from
80MB/S to over 670MB/S on the 3Ware controller.
//#define BLKDEV_MIN_RQ 4
//#define BLKDEV_MAX_RQ 128 /* Default maximum */
#define BLKDEV_MIN_RQ 4096
#define BLKDEV_MAX_RQ 8192 /* Default maximum */
Jeff
Jens Axboe wrote:
quoted
On Wed, Aug 31 2005, Holger Kiehl wrote:
quoted
On Wed, 31 Aug 2005, Jens Axboe wrote:
quoted
Nothing sticks out here either. There's plenty of idle time. It
smells
quoted
quoted
quoted
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
I used the following command reading from all 8 disks in parallel:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Here vmstat output (I just cut something out in the middle):
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----^M
r b swpd free buff cache si so bi bo in cs us
Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.
quoted
quoted
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference
though,
quoted
quoted
quoted
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
I've attached a little sample for you, just run ala
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
-----------------------------------------------------------------------
#define BLOCKS (8192)
int main(int argc, char *argv[])
{
char *p;
int fd, i;
if (argc < 2) {
printf("%s: <dev>\n", argv[0]);
return 1;
}
fd = open(argv[1], O_RDONLY | O_DIRECT);
if (fd == -1) {
perror("open");
return 1;
}
p = ALIGN(malloc(BS + 4095));
for (i = 0; i < BLOCKS; i++) {
int r = read(fd, p, BS);
if (r == BS)
continue;
else {
if (r == -1)
perror("read");
break;
}
}
return 0;
}
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I'll try this approach as well. On 2.4.X kernels, I had to change
nr_requests to achieve performance, but
I noticed it didn't seem to work as well on 2.6.X. I'll retry the
change with nr_requests on 2.6.X.
Thanks
Jeff
Tom Callahan wrote:
quoted
From linux-kernel mailing list.....
Don't do this. BLKDEV_MIN_RQ sets the size of the mempool reserved
requests and will only get slightly used in low memory conditions, so
most memory will probably be wasted.....
Change /sys/block/xxx/queue/nr_requests
Tom Callahan
TESSCO Technologies
(443)-506-6216
callahant@tessco.com
jmerkey wrote:
quoted
I have seen an 80GB/sec limitation in the kernel unless this value is
changed in the SCSI I/O layer
for 3Ware and other controllers during testing of 2.6.X series kernels.
Change these values in include/linux/blkdev.h and performance goes from
80MB/S to over 670MB/S on the 3Ware controller.
//#define BLKDEV_MIN_RQ 4
//#define BLKDEV_MAX_RQ 128 /* Default maximum */
#define BLKDEV_MIN_RQ 4096
#define BLKDEV_MAX_RQ 8192 /* Default maximum */
Jeff
Jens Axboe wrote:
quoted
On Wed, Aug 31 2005, Holger Kiehl wrote:
quoted
On Wed, 31 Aug 2005, Jens Axboe wrote:
quoted
Nothing sticks out here either. There's plenty of idle time. It
smells
quoted
quoted
quoted
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
I used the following command reading from all 8 disks in parallel:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Here vmstat output (I just cut something out in the middle):
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----^M
r b swpd free buff cache si so bi bo in cs us
Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.
quoted
quoted
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference
though,
quoted
quoted
quoted
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
I've attached a little sample for you, just run ala
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
-----------------------------------------------------------------------
#define BLOCKS (8192)
int main(int argc, char *argv[])
{
char *p;
int fd, i;
if (argc < 2) {
printf("%s: <dev>\n", argv[0]);
return 1;
}
fd = open(argv[1], O_RDONLY | O_DIRECT);
if (fd == -1) {
perror("open");
return 1;
}
p = ALIGN(malloc(BS + 4095));
for (i = 0; i < BLOCKS; i++) {
int r = read(fd, p, BS);
if (r == BS)
continue;
else {
if (r == -1)
perror("read");
break;
}
}
return 0;
}
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
I have seen an 80GB/sec limitation in the kernel unless this value is
changed in the SCSI I/O layer
for 3Ware and other controllers during testing of 2.6.X series kernels.
Change these values in include/linux/blkdev.h and performance goes from
80MB/S to over 670MB/S on the 3Ware controller.
//#define BLKDEV_MIN_RQ 4
//#define BLKDEV_MAX_RQ 128 /* Default maximum */
#define BLKDEV_MIN_RQ 4096
#define BLKDEV_MAX_RQ 8192 /* Default maximum */
That's insane, you just wasted 1MiB of preallocated requests on each
queue in the system!
Please just do
# echo 512 > /sys/block/dev/queue/nr_requests
after boot for each device you want to increase the queue size too. 512
should be enough with the 3ware.
--
Jens Axboe
512 is not enough. It has to be larger. I just tried 512 and it still
limits the data rates.
Jeff
Jens Axboe wrote:
On Wed, Aug 31 2005, jmerkey wrote:
quoted
I have seen an 80GB/sec limitation in the kernel unless this value is
changed in the SCSI I/O layer
for 3Ware and other controllers during testing of 2.6.X series kernels.
Change these values in include/linux/blkdev.h and performance goes from
80MB/S to over 670MB/S on the 3Ware controller.
//#define BLKDEV_MIN_RQ 4
//#define BLKDEV_MAX_RQ 128 /* Default maximum */
#define BLKDEV_MIN_RQ 4096
#define BLKDEV_MAX_RQ 8192 /* Default maximum */
That's insane, you just wasted 1MiB of preallocated requests on each
queue in the system!
Please just do
# echo 512 > /sys/block/dev/queue/nr_requests
after boot for each device you want to increase the queue size too. 512
should be enough with the 3ware.
512 is not enough. It has to be larger. I just tried 512 and it still
limits the data rates.
Please don't top post.
512 wasn't the point, setting it properly is the point. If you need more
than 512, go ahead. This isn't Holger's problem, though, the reading
would be much faster if it was. If the fusion is using a large queue
depth, increasing nr_requests would likely help the writes (but not to
the extent of where it would suddenly be as fast as it should).
--
Jens Axboe
Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.
I used the following command reading from all 8 disks in parallel:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Here vmstat output (I just cut something out in the middle):
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----^M
r b swpd free buff cache si so bi bo in cs us sy id
wa^M
3 7 4348 42640 7799984 9612 0 0 322816 0 3532 4987 0 22
0 78
1 7 4348 42136 7800624 9584 0 0 322176 0 3526 4987 0 23
4 74
0 8 4348 39912 7802648 9668 0 0 322176 0 3525 4955 0 22
12 66
1 7 4348 38912 7803700 9636 0 0 322432 0 3526 5078 0 23
Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.
quoted
quoted
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
I've attached a little sample for you, just run ala
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
Using kernel 2.6.12.5 again, here the results:
[snip]
Ok, reads as expected, like the buffered io but using less system time.
And you are still 1/3 off the target data rate, hmmm...
With the reads, how does the aggregate bandwidth look when you add
'clients'? Same as with writes, gradually decreasing per-device
throughput?
--
Jens Axboe
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
Using kernel 2.6.12.5 again, here the results:
[snip]
Ok, reads as expected, like the buffered io but using less system time.
And you are still 1/3 off the target data rate, hmmm...
With the reads, how does the aggregate bandwidth look when you add
'clients'? Same as with writes, gradually decreasing per-device
throughput?
I performed the following tests with this command:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Single disk tests:
/dev/sdc1 74.954715 MB/s
/dev/sdg1 74.973417 MB/s
Following disks in parallel:
2 disks on same channel
/dev/sdc1 75.034191 MB/s
/dev/sdd1 74.984643 MB/s
3 disks on same channel
/dev/sdc1 75.027850 MB/s
/dev/sdd1 74.976583 MB/s
/dev/sde1 75.278276 MB/s
4 disks on same channel
/dev/sdc1 58.343166 MB/s
/dev/sdd1 62.993059 MB/s
/dev/sde1 66.940569 MB/s
/dev/sdd1 70.986072 MB/s
2 disks on different channels
/dev/sdc1 74.954715 MB/s
/dev/sdg1 74.973417 MB/s
4 disks on different channels
/dev/sdc1 74.959030 MB/s
/dev/sdd1 74.877703 MB/s
/dev/sdg1 75.009697 MB/s
/dev/sdh1 75.028138 MB/s
6 disks on different channels
/dev/sdc1 49.640743 MB/s
/dev/sdd1 55.935419 MB/s
/dev/sde1 58.795241 MB/s
/dev/sdg1 50.280864 MB/s
/dev/sdh1 54.210705 MB/s
/dev/sdi1 59.413176 MB/s
So this looks different from writting, only as of four disks does the
performance begin to drop.
I just noticed, did you want me to do these test with the oread program?
Thanks,
Holger
From: Michael Tokarev <hidden> Date: 2005-08-31 18:06:05
Holger Kiehl wrote:
On Wed, 31 Aug 2005, Jens Axboe wrote:
quoted
On Wed, Aug 31 2005, Holger Kiehl wrote:
[]
quoted
quoted
I used the following command reading from all 8 disks in parallel:
dd if=/dev/sd?1 of=/dev/null bs=256k count=78125
Here vmstat output (I just cut something out in the middle):
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
3 7 4348 42640 7799984 9612 0 0 322816 0 3532 4987 0 22 0 78
1 7 4348 42136 7800624 9584 0 0 322176 0 3526 4987 0 23 4 74
0 8 4348 39912 7802648 9668 0 0 322176 0 3525 4955 0 22 12 66
Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.
quoted
quoted
You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).
Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?
I've attached a little sample for you, just run ala
# ./oread /dev/sdX
and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.
I went on and did similar tests on our box, which is:
dual Xeon 2.44GHz with HT (so it's like 4 logical CPUs)
dual-channel AIC-7902 U320 controller
8 SEAGATE ST336607LW drives attached to the 2 channels of the
controller, sd[abcd] to channel 0 and sd[efgh] to channel 1
Each drive is capable to get about 60 megabytes/sec.
The kernel is 2.6.13 from kernel.org.
With direct-reading:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 8 12 87 471 1839 0 0 455296 84 1936 3739 0 3 47 50
1 7 12 87 471 1839 0 0 456704 80 1941 3744 0 4 48 48
1 7 12 87 471 1839 0 0 446464 82 1914 3648 0 2 48 50
0 8 12 87 471 1839 0 0 454016 94 1944 3765 0 2 47 50
0 8 12 87 471 1839 0 0 458752 60 1944 3746 0 2 48 50
Without direct:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
8 6 12 80 470 1839 0 0 359966 124 1726 2270 1 89 0 10
2 7 12 80 470 1839 0 0 352813 113 1741 2124 1 88 1 10
8 4 12 80 471 1839 0 0 358990 34 1669 1934 1 94 0 5
7 5 12 79 471 1839 0 0 354065 157 1761 2128 1 90 1 8
6 5 12 80 471 1839 0 0 358062 44 1686 1911 1 93 0 6
So the difference direct vs "indirect" is quite.. significant. And with
direct-reading, all 8 drives are up to their real speed. Note the CPU usage
in case of "indirect" reading too - it's about 90%...
And here's an idle system as well:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 12 89 471 1839 0 0 0 58 151 358 0 0 100 0
0 0 12 89 471 1839 0 0 0 66 157 167 0 0 99 0
Too bad I can't perform write tests on this system...
/mjt
From: Ming Zhang <hidden> Date: 2005-08-31 18:53:03
join the party. ;)
8 400GB SATA disk on same Marvel 8 port PCIX-133 card. P4 CPU.
Supermicro SCT board.
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6]
[raid10] [faulty]
md0 : active raid0 sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1] sda
[0]
3125690368 blocks 64k chunks
8 DISK RAID0 from same slot and card. Stripe size is 512KB.
run oread
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----
cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
1 1 0 533216 330424 11004 0 0 7128 1610 1069 77 0 2
95 3
1 0 0 298464 560828 11004 0 0 230404 0 2595 1389 1
23 0 76
0 1 0 64736 792248 11004 0 0 231420 0 2648 1342 0
26 0 74
1 0 0 8948 848416 9696 0 0 229376 0 2638 1337 0
29 0 71
0 0 0 868896 768 9696 0 0 29696 48 1224 162 0 19
73 8
# time ./oread /dev/md0
real 0m6.595s
user 0m0.004s
sys 0m0.151s
run dd
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----
cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
2 2 0 854008 2932 17108 0 0 7355 1606 1071 80 0 2
95 3
0 2 0 848888 3112 21388 0 0 164332 0 2985 3564 2
7 0 91
0 2 0 844024 3260 25664 0 0 164040 0 2990 3665 1
7 0 92
0 2 0 840328 3380 28920 0 0 164272 0 2932 3791 1
9 0 90
0 2 0 836360 3500 32232 0 0 163688 100 3001 5045 2
7 0 91
0 2 0 831432 3644 36612 0 0 164120 568 2977 3843 0
9 0 91
0 1 0 826056 3752 41688 0 0 7872 0 1267 1474 1 3
0 96
# time dd if=/dev/md0 of=/dev/null bs=131072 count=8192
8192+0 records in
8192+0 records out
real 0m4.771s
user 0m0.005s
sys 0m0.973s
so the reasonable thing here is because of O_DIRECT, the sys time
reduced a lot.
but the time is longer! the reason i found is...
i attached a new oread.c which allow to set block size of each read and
total read count. so i read full strip once a time,
# time ./oread /dev/md0 524288 2048
real 0m4.950s
user 0m0.000s
sys 0m0.131s
compared to
# time ./oread /dev/md0 131072 8192
real 0m6.633s
user 0m0.002s
sys 0m0.191s
but still, I can get linear speed at 4 DISKS, then no speed gain when
adding more disk into the RAID.
Ming
From: Ming Zhang <hidden> Date: 2005-08-31 18:57:13
forgot to attach lspci output.
it is a 133MB PCI-X card but only run at 66MHZ.
quick question, where I can check if it is running at 64bit?
66MHZ * 32Bit /8 * 80% bus utilization ~= 211MB/s then match the upper
speed I meet now...
Ming
02:01.0 SCSI storage controller: Marvell MV88SX5081 8-port SATA I PCI-X
Controller (rev 03)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 128, Cache Line Size 08
Interrupt: pin A routed to IRQ 24
Region 0: Memory at fa000000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 2
Flags: PMEClk+ DSI- D1- D2- AuxCurrent=0mA PME
(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [60] PCI-X non-bridge device.
Command: DPERE- ERO- RBC=0 OST=3
Status: Bus=2 Dev=1 Func=0 64bit+ 133MHz+ SCD- USC-,
DC=simple, DMMRBC=0, DMOST=3, DMCRS=0, RSCEM-
On Wed, 2005-08-31 at 14:52 -0400, Ming Zhang wrote:
join the party. ;)
8 400GB SATA disk on same Marvel 8 port PCIX-133 card. P4 CPU.
Supermicro SCT board.
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath] [raid6]
[raid10] [faulty]
md0 : active raid0 sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2] sdb[1] sda
[0]
3125690368 blocks 64k chunks
8 DISK RAID0 from same slot and card. Stripe size is 512KB.
run oread
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----
cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
1 1 0 533216 330424 11004 0 0 7128 1610 1069 77 0 2
95 3
1 0 0 298464 560828 11004 0 0 230404 0 2595 1389 1
23 0 76
0 1 0 64736 792248 11004 0 0 231420 0 2648 1342 0
26 0 74
1 0 0 8948 848416 9696 0 0 229376 0 2638 1337 0
29 0 71
0 0 0 868896 768 9696 0 0 29696 48 1224 162 0 19
73 8
# time ./oread /dev/md0
real 0m6.595s
user 0m0.004s
sys 0m0.151s
run dd
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----
cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
2 2 0 854008 2932 17108 0 0 7355 1606 1071 80 0 2
95 3
0 2 0 848888 3112 21388 0 0 164332 0 2985 3564 2
7 0 91
0 2 0 844024 3260 25664 0 0 164040 0 2990 3665 1
7 0 92
0 2 0 840328 3380 28920 0 0 164272 0 2932 3791 1
9 0 90
0 2 0 836360 3500 32232 0 0 163688 100 3001 5045 2
7 0 91
0 2 0 831432 3644 36612 0 0 164120 568 2977 3843 0
9 0 91
0 1 0 826056 3752 41688 0 0 7872 0 1267 1474 1 3
0 96
# time dd if=/dev/md0 of=/dev/null bs=131072 count=8192
8192+0 records in
8192+0 records out
real 0m4.771s
user 0m0.005s
sys 0m0.973s
so the reasonable thing here is because of O_DIRECT, the sys time
reduced a lot.
but the time is longer! the reason i found is...
i attached a new oread.c which allow to set block size of each read and
total read count. so i read full strip once a time,
# time ./oread /dev/md0 524288 2048
real 0m4.950s
user 0m0.000s
sys 0m0.131s
compared to
# time ./oread /dev/md0 131072 8192
real 0m6.633s
user 0m0.002s
sys 0m0.191s
but still, I can get linear speed at 4 DISKS, then no speed gain when
adding more disk into the RAID.
Ming
^^^^^^^^^
I don't think this function should be here. This indicates that
lots of writeout is happening due to pages falling off the end
of the LRU.
There was a bug recently causing memory estimates to be wrong
on Opterons that could cause this I think.
Can you send in 2 dumps of /proc/vmstat taken 10 seconds apart
while you're writing at full speed (with 2.6.13 or the latest
-git tree).
A dump of /proc/zoneinfo and /proc/meminfo while the write is
going on would be helpful too.
Thanks,
Nick
--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com
^^^^^^^^^
I don't think this function should be here. This indicates that
lots of writeout is happening due to pages falling off the end
of the LRU.
There was a bug recently causing memory estimates to be wrong
on Opterons that could cause this I think.
Can you send in 2 dumps of /proc/vmstat taken 10 seconds apart
while you're writing at full speed (with 2.6.13 or the latest
-git tree).
Hmm OK, dirty memory is pinned pretty much exactly on dirty_ratio
so maybe I've just led you on a goose chase.
You could
echo 5 > /proc/sys/vm/dirty_background_ratio
echo 10 > /proc/sys/vm/dirty_ratio
To further reduce dirty memory in the system, however this is
a long shot, so please continue your interaction with the
other people in the thread first.
Thanks,
Nick
--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com
Hmm OK, dirty memory is pinned pretty much exactly on dirty_ratio
so maybe I've just led you on a goose chase.
You could
echo 5 > /proc/sys/vm/dirty_background_ratio
echo 10 > /proc/sys/vm/dirty_ratio
To further reduce dirty memory in the system, however this is
a long shot, so please continue your interaction with the
other people in the thread first.
Yes, this does make a difference, here the results of running
dd if=/dev/full of=/dev/sd?1 bs=4M count=4883
on 8 disks at the same time:
34.273340
33.938829
33.598469
32.970575
32.841351
32.723988
31.559880
29.778112
That's 32.710568 MB/s on average per disk with your change and without
it it was 24.958557 MB/s on average per disk.
I will do more tests tomorrow.
Thanks,
Holger
Hmm OK, dirty memory is pinned pretty much exactly on dirty_ratio
so maybe I've just led you on a goose chase.
You could
echo 5 > /proc/sys/vm/dirty_background_ratio
echo 10 > /proc/sys/vm/dirty_ratio
To further reduce dirty memory in the system, however this is
a long shot, so please continue your interaction with the
other people in the thread first.
Yes, this does make a difference, here the results of running
dd if=/dev/full of=/dev/sd?1 bs=4M count=4883
on 8 disks at the same time:
34.273340
33.938829
33.598469
32.970575
32.841351
32.723988
31.559880
29.778112
That's 32.710568 MB/s on average per disk with your change and without
it it was 24.958557 MB/s on average per disk.
I will do more tests tomorrow.
Just rechecked those numbers. Did a fresh boot and run the test several
times. With defaults (dirty_background_ratio=10, dirty_ratio=40) I get
for the dd write tests an average of 24.559491 MB/s (8 disks in parallel)
per disk. With the suggested values (dirty_background_ratio=5, dirty_ratio=10)
32.390659 MB/s per disk.
I then did a SW raid0 over all disks with the following command:
mdadm -C /dev/md3 -l0 -n8 /dev/sd[cdefghij]1
(dirty_background_ratio=10, dirty_ratio=40) 223.955995 MB/s
(dirty_background_ratio=5, dirty_ratio=10) 234.318936 MB/s
So the differnece is not so big anymore.
Something else I notice while doing the dd over 8 disks is the following
(top just before they are finished):
top - 08:39:11 up 2:03, 2 users, load average: 23.01, 21.48, 15.64
Tasks: 102 total, 2 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 17.7% sy, 0.0% ni, 0.0% id, 78.9% wa, 0.2% hi, 3.1% si
Mem: 8124184k total, 8093068k used, 31116k free, 7831348k buffers
Swap: 15631160k total, 13352k used, 15617808k free, 5524k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3423 root 18 0 55204 460 392 R 12.0 0.0 1:15.55 dd
3421 root 18 0 55204 464 392 D 11.3 0.0 1:17.36 dd
3418 root 18 0 55204 464 392 D 10.3 0.0 1:10.92 dd
3416 root 18 0 55200 464 392 D 10.0 0.0 1:09.20 dd
3420 root 18 0 55204 464 392 D 10.0 0.0 1:10.49 dd
3422 root 18 0 55200 460 392 D 9.3 0.0 1:13.58 dd
3417 root 18 0 55204 460 392 D 7.6 0.0 1:13.11 dd
158 root 15 0 0 0 0 D 1.3 0.0 1:12.61 kswapd3
159 root 15 0 0 0 0 D 1.3 0.0 1:08.75 kswapd2
160 root 15 0 0 0 0 D 1.0 0.0 1:07.11 kswapd1
3419 root 18 0 51096 552 476 D 1.0 0.0 1:17.15 dd
161 root 15 0 0 0 0 D 0.7 0.0 0:54.46 kswapd0
1 root 16 0 4876 372 332 S 0.0 0.0 0:01.15 init
2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/2
7 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/3
9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3
A loadaverage of 23 for 8 dd's seems a bit high. Also why is kswapd working
so hard? Is that correct.
Please just tell me if there is anything else I can test or dumps that
could be useful.
Thanks,
Holger
top - 08:39:11 up 2:03, 2 users, load average: 23.01, 21.48, 15.64
Tasks: 102 total, 2 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0% us, 17.7% sy, 0.0% ni, 0.0% id, 78.9% wa, 0.2% hi, 3.1%
si Mem: 8124184k total, 8093068k used, 31116k free, 7831348k
buffers Swap: 15631160k total, 13352k used, 15617808k free, 5524k
cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3423 root 18 0 55204 460 392 R 12.0 0.0 1:15.55 dd
3421 root 18 0 55204 464 392 D 11.3 0.0 1:17.36 dd
3418 root 18 0 55204 464 392 D 10.3 0.0 1:10.92 dd
3416 root 18 0 55200 464 392 D 10.0 0.0 1:09.20 dd
3420 root 18 0 55204 464 392 D 10.0 0.0 1:10.49 dd
3422 root 18 0 55200 460 392 D 9.3 0.0 1:13.58 dd
3417 root 18 0 55204 460 392 D 7.6 0.0 1:13.11 dd
158 root 15 0 0 0 0 D 1.3 0.0 1:12.61 kswapd3
159 root 15 0 0 0 0 D 1.3 0.0 1:08.75 kswapd2
160 root 15 0 0 0 0 D 1.0 0.0 1:07.11 kswapd1
3419 root 18 0 51096 552 476 D 1.0 0.0 1:17.15 dd
161 root 15 0 0 0 0 D 0.7 0.0 0:54.46 kswapd0
A loadaverage of 23 for 8 dd's seems a bit high. Also why is kswapd
working so hard? Is that correct.
Actually, kswapd is another problem. (see "Kswapd Flaw" thread)
Which has little impact on your problem but basically kswapd tries very hard
maybe even to hard to fullfil a request for memory, so when the buffer/cache
pages are full kswapd tries to find some more unused memory. When it finds
none it starts recycling the buffer/cache pages. Which is OK, but it only
does this after searching for swappable memory which wastes CPU cycles.
This can be tuned a little but not much by adjusting /sys(proc)/.../vm/...
Or renicing kswapd to the lowest priority, which may cause other problems.
Things get really bad when procs start asking for more memory than is
available, causing kswapd to take the liberty of paging out running procs in
the hope that these procs won't come back later. So when they do come back
something like a wild goose chase begins. This is also known as OverCommit.
This is closely related to the dreaded OOM-killer, which occurs when the
system cannot satisfy a memory request for a returning proc, causing the VM
to start killing in an unpredictable manner.
Turning OverCommit off should solve this problem but it doesn't.
This is why it is recommended to run the system always with swap enabled even
if you have tons of memory, which really only pushes the problem out of the
way until you hit the dead end and the wild goose chase begins again.
Sadly 2.6.13 did not fix this either.
Although this description only vaguely defines the problem from an end-user
pov, the actual semantics may be quite different.
--
Al
To find where the bottleneck is, I'd suggest trying without the
filesystem at all, and just filling a large part of the block device
using the 'dd' command.
Also, trying without the RAID, and just running 4 (and 8) concurrent
dd's to the separate drives could show whether it's the RAID that's
slowing things down.
Ok, I did run the following dd command in different combinations:
dd if=/dev/zero of=/dev/sd?1 bs=4k count=5000000
I think a bs of 4k is way too small and will cause huge CPU overhead.
Can you try with something like 4M? Also, you can use /dev/full to avoid
the pre-zeroing.
Ok, I now use the following command:
dd if=/dev/full of=/dev/sd?1 bs=4M count=4883
Here the results for all 8 disks in parallel:
/dev/sdc1 24.957257 MB/s
/dev/sdd1 25.290177 MB/s
/dev/sde1 25.046711 MB/s
/dev/sdf1 26.369777 MB/s
/dev/sdg1 24.080695 MB/s
/dev/sdh1 25.008803 MB/s
/dev/sdi1 24.202202 MB/s
/dev/sdj1 24.712840 MB/s
A little bit faster but not much.
Holger