On Wed, Aug 31 2005, Holger Kiehl wrote:

On Wed, 31 Aug 2005, Jens Axboe wrote:

quoted

Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.

I used the following command reading from all 8 disks in parallel:

   dd if=/dev/sd?1 of=/dev/null bs=256k count=78125

Here vmstat output (I just cut something out in the middle):

procs -----------memory---------- ---swap-- -----io---- --system-- 
----cpu----^M
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id 
 wa^M
 3  7   4348  42640 7799984   9612    0    0 322816     0 3532  4987  0 22  
 0 78
 1  7   4348  42136 7800624   9584    0    0 322176     0 3526  4987  0 23  
 4 74
 0  8   4348  39912 7802648   9668    0    0 322176     0 3525  4955  0 22 
 12 66
 1  7   4348  38912 7803700   9636    0    0 322432     0 3526  5078  0 23  

Ok, so that's somewhat better than the writes but still off from what
the individual drives can do in total.

quoted

You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).

Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?

I've attached a little sample for you, just run ala

# ./oread /dev/sdX

and it will read 128k chunks direct from that device. Run on the same
drives as above, reply with the vmstat info again.

-- 
Jens Axboe