[ofa-general] Re: [PATCH 00/10] Implement batching skb API
From: Stephen Hemminger <hidden>
Date: 2007-07-20 07:18:48
On Fri, 20 Jul 2007 12:01:49 +0530 Krishna Kumar [off-list ref] wrote:
Hi Dave, Roland, everyone,
In May, I had proposed creating an API for sending 'n' skbs to a driver to
reduce lock overhead, DMA operations, and specific to drivers that have
completion notification like IPoIB - reduce completion handling ("[RFC] New
driver API to speed up small packets xmits" @
http://marc.info/?l=linux-netdev&m=117880900818960&w=2). I had also sent
initial test results for E1000 which showed minor improvements (but also
got degradations) @http://marc.info/?l=linux-netdev&m=117887698405795&w=2.
After fine-tuning qdisc and other changes, I modified IPoIB to use this API,
and now get good gains. Summary for TCP & No Delay: 1 process improves for
all cases from 1.4% to 49.5%; 4 process has almost identical improvements
from -1.7% to 59.1%; 16 process case also improves in the range of -1.2% to
33.4%; while 64 process doesn't have much improvement (-3.3% to 12.4%). UDP
was tested with 1 process netperf with small increase in BW but big
improvement in Service Demand. Netperf latency tests show small drop in
transaction rate (results in separate attachment).You may see worse performance with batching in the real world when running over WAN's. Like TSO, batching will generate back to back packet trains that are subject to multi-packet synchronized loss. The problem is that intermediate router queues are often close to full, and when a long string of packets arrives back to back only the first ones will get in, the rest get dropped. Normal sends have at least minimal pacing so they are less likely do get synchronized drop.