On Wed, 31 Aug 2005, Jens Axboe wrote:

Nothing sticks out here either. There's plenty of idle time. It smells
like a driver issue. Can you try the same dd test, but read from the
drives instead? Use a bigger blocksize here, 128 or 256k.

I used the following command reading from all 8 disks in parallel:

    dd if=/dev/sd?1 of=/dev/null bs=256k count=78125

Here vmstat output (I just cut something out in the middle):

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----^M
  r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa^M
  3  7   4348  42640 7799984   9612    0    0 322816     0 3532  4987  0 22  0 78
  1  7   4348  42136 7800624   9584    0    0 322176     0 3526  4987  0 23  4 74
  0  8   4348  39912 7802648   9668    0    0 322176     0 3525  4955  0 22 12 66
  1  7   4348  38912 7803700   9636    0    0 322432     0 3526  5078  0 23  7 70
  2  6   4348  37552 7805120   9644    0    0 322432     0 3527  4908  0 23 12 64
  0  8   4348  41152 7801552   9608    0    0 322176     0 3524  5018  0 24  6 70
  1  7   4348  41644 7801044   9572    0    0 322560     0 3530  5175  0 23  0 76
  1  7   4348  37184 7805396   9640    0    0 322176     0 3525  4914  0 24 18 59
  3  7   4348  41704 7800376   9832    0    0 322176    20 3531  5080  0 23  4 73
  1  7   4348  40652 7801700   9732    0    0 323072     0 3533  5115  0 24 13 64
  1  7   4348  40284 7802224   9616    0    0 322560     0 3527  4967  0 23  1 76
  0  8   4348  40156 7802356   9688    0    0 322560     0 3528  5080  0 23  2 75
  6  8   4348  41896 7799984   9816    0    0 322176     0 3530  4945  0 24 20 57
  0  8   4348  39540 7803124   9600    0    0 322560     0 3529  4811  0 24 21 55
  1  7   4348  41520 7801084   9600    0    0 322560     0 3532  4843  0 23 22 55
  0  8   4348  40408 7802116   9588    0    0 322560     0 3527  5010  0 23  4 72
  0  8   4348  38172 7804300   9580    0    0 322176     0 3526  4992  0 24  7 69
  4  7   4348  42264 7799784   9812    0    0 322688     0 3529  5003  0 24  8 68
  1  7   4348  39908 7802520   9660    0    0 322700     0 3529  4963  0 24 14 62
  0  8   4348  37428 7805076   9620    0    0 322420     0 3528  4967  0 23 15 62
  0  8   4348  37056 7805348   9688    0    0 322048     0 3525  4982  0 24 26 50
  1  7   4348  37804 7804456   9696    0    0 322560     0 3528  5072  0 24 16 60
  0  8   4348  38416 7804084   9660    0    0 323200     0 3533  5081  0 24 23 53
  0  8   4348  40160 7802300   9676    0    0 323200    28 3543  5095  0 24 17 59
  1  7   4348  37928 7804612   9608    0    0 323072     0 3532  5175  0 24  7 68
  2  6   4348  38680 7803724   9612    0    0 322944     0 3531  4906  0 25 24 51
  1  7   4348  40408 7802192   9648    0    0 322048     0 3524  4947  0 24 19 57

Full vmstat session can be found under:

   ftp://ftp.dwd.de/pub/afd/linux_kernel_debug/vmstat-256k-read

And here the profile data:

2106577 total                                      0.9469
1638177 default_idle                             34128.6875
179615 copy_user_generic_c                      4726.7105
  27670 end_buffer_async_read                    108.0859
  26055 shrink_zone                                7.1111
  23199 __make_request                            17.2612
  17221 kmem_cache_free                          153.7589
  11796 drop_buffers                              52.6607
  11016 add_to_page_cache                         52.9615
   9470 __wake_up_bit                            197.2917
   8760 buffered_rmqueue                          12.4432
   8646 find_get_page                             90.0625
   8319 __do_page_cache_readahead                 11.0625
   7976 kmem_cache_alloc                         124.6250
   7463 scsi_request_fn                            6.2192
   7208 try_to_free_buffers                       40.9545
   6716 create_empty_buffers                      41.9750
   6432 __end_that_request_first                  11.8235
   6044 test_clear_page_dirty                     25.1833
   5643 scsi_dispatch_cmd                          9.7969
   5588 free_hot_cold_page                        19.4028
   5479 submit_bh                                 18.0230
   3903 __alloc_pages                              3.2965
   3671 file_read_actor                            9.9755
   3425 thread_return                             14.2708
   3333 generic_make_request                       5.6301
   3294 bio_alloc_bioset                           7.6250
   2868 bio_put                                   44.8125
   2851 mpt_interrupt                              2.8284
   2697 mempool_alloc                              8.8717
   2642 block_read_full_page                       3.9315
   2512 do_generic_mapping_read                    2.1216
   2394 set_page_refs                            149.6250
   2235 alloc_page_buffers                         9.9777
   1992 __pagevec_lru_add                          8.3000
   1859 __memset                                   9.6823
   1791 page_waitqueue                            15.9911
   1783 scsi_end_request                           6.9648
   1348 dma_unmap_sg                               6.4808
   1324 bio_endio                                 11.8214
   1306 unlock_page                               20.4062
   1211 mptscsih_freeChainBuffers                  7.5687
   1141 alloc_pages_current                        7.9236
   1136 __mod_page_state                          35.5000
   1116 radix_tree_preload                         8.7188
   1061 __pagevec_release_nonlru                   6.6312
   1043 set_bh_page                                9.3125
   1024 release_pages                              2.9091
   1023 mempool_free                               6.3937
    832 alloc_buffer_head                         13.0000

Full profile data can be found under:

    ftp://ftp.dwd.de/pub/afd/linux_kernel_debug/dd-256k-8disk-read.profile

You might want to try the same with direct io, just to eliminate the
costly user copy. I don't expect it to make much of a difference though,
feels like the problem is elsewhere (driver, most likely).

Sorry, I don't know how to do this. Do you mean using a C program
that sets some flag to do direct io, or how can I do that?

If we still can't get closer to this, it would be interesting to try my
block tracing stuff so we can see what is going on at the queue level.
But lets gather some more info first, since it requires testing -mm.

Ok, please then just tell me what I must do.

Thanks,
Holger