Holger Kiehl wrote:

On Wed, 31 Aug 2005, Jens Axboe wrote:

quoted

On Wed, Aug 31 2005, Holger Kiehl wrote:

[]

quoted

quoted

I used the following command reading from all 8 disks in parallel:

   dd if=/dev/sd?1 of=/dev/null bs=256k count=78125

Here vmstat output (I just cut something out in the middle):

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 3  7   4348  42640 7799984   9612    0    0 322816     0 3532  4987 0 22 0 78
 1  7   4348  42136 7800624   9584    0    0 322176     0 3526  4987 0 23 4 74
 0  8   4348  39912 7802648   9668    0    0 322176     0 3525  4955 0 22 12 66

Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.

quoted

quoted

You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).

Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?

I've attached a little sample for you, just run ala

# ./oread /dev/sdX

and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.

Using kernel 2.6.12.5 again, here the results:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  8      0 8005380   4816  41084    0    0 318848    32 3511  4672  0 1 75 24
 0  8      0 8005380   4816  41084    0    0 320640     0 3512  4877  0 2 75 23
 0  8      0 8005380   4816  41084    0    0 322944     0 3533  5047  0 2 75 24
 0  8      0 8005380   4816  41084    0    0 322816     0 3531  5053  0 1 75 24
 0  8      0 8005380   4816  41084    0    0 322944     0 3531  5048  0 2 75 23
 0  8      0 8005380   4816  41084    0    0 322944     0 3529  5043  0 1 75 24
 0  0      0 8008360   4816  41084    0    0 266880     0 3112  4224  0 2 78 20

I went on and did similar tests on our box, which is:

 dual Xeon 2.44GHz with HT (so it's like 4 logical CPUs)
 dual-channel AIC-7902 U320 controller
 8 SEAGATE ST336607LW drives attached to the 2 channels of the
  controller, sd[abcd] to channel 0 and sd[efgh] to channel 1

Each drive is capable to get about 60 megabytes/sec.
The kernel is 2.6.13 from kernel.org.

With direct-reading:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  8     12     87    471   1839    0    0 455296    84 1936  3739  0  3 47 50
 1  7     12     87    471   1839    0    0 456704    80 1941  3744  0  4 48 48
 1  7     12     87    471   1839    0    0 446464    82 1914  3648  0  2 48 50
 0  8     12     87    471   1839    0    0 454016    94 1944  3765  0  2 47 50
 0  8     12     87    471   1839    0    0 458752    60 1944  3746  0  2 48 50

Without direct:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 8  6     12     80    470   1839    0    0 359966   124 1726  2270  1 89  0 10
 2  7     12     80    470   1839    0    0 352813   113 1741  2124  1 88  1 10
 8  4     12     80    471   1839    0    0 358990    34 1669  1934  1 94  0  5
 7  5     12     79    471   1839    0    0 354065   157 1761  2128  1 90  1  8
 6  5     12     80    471   1839    0    0 358062    44 1686  1911  1 93  0  6

So the difference direct vs "indirect" is quite.. significant.  And with
direct-reading, all 8 drives are up to their real speed.  Note the CPU usage
in case of "indirect" reading too - it's about 90%...

And here's an idle system as well:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0     12     89    471   1839    0    0     0    58  151   358  0  0 100  0
 0  0     12     89    471   1839    0    0     0    66  157   167  0  0 99  0

Too bad I can't perform write tests on this system...

/mjt