Alan Cox wrote:

Be cautious what you measure. One of he problems until you reach PCI-X
is PCI bandwidth. Thus software md5 can look good but the moment its
combined with other PCI activity goes down the pan entirely.
 

Right, which is why you'd think hardware based RAID would fair better, 
the parity information or mirror'd writes would only require one 
transfer/transaction across the PCI bus.  However, when benchmarking a 
3Ware 4-port ide raid controller (7000) we saw 40-50MB/sec read/write on 
4x160GB drives raid5.  With md raid5 (same controller, HDs, FS, etc)  we 
saw 100MB/sec read, 60MB/sec write.  This was using max 15% CPU on a 
2.4GHz Pentium4, with a 32bit/33MHz PCI bus.

quoted

When the libata Marvell drivers come out, you'll have a cheap upgrade 
path for PCI-X boards if you want fast md raid: 
   

Agreed. PCI-X will change a lot of this for boxes that are not very
cpu/memory limited.
 

 From the testing I've done, interconnect bandwidth has always been the 
limiting factor for the md driver.  Using cheap ($~230) PCI-X 
motherboards--http://www.supermicro.com/products/motherboard/P4/E7210/P4SCT+.cfm--with 
dedicated gigE channels, you can make high density/price and 
performance/price NAS type appliances with no bottlenecks.  For <$3K you 
can build a 2TB NAS server that'll keep a 250MB/s gigE link saturated.

-ryan