Linas Vepstas [off-list ref] wrote:
On Fri, Aug 24, 2007 at 03:59:16PM +0200, Jan-Bernd Themann wrote:
quoted
3) On modern systems the incoming packets are processed very fast. Especially
on SMP systems when we use multiple queues we process only a few packets
per napi poll cycle. So NAPI does not work very well here and the interrupt
rate is still high.
I saw this too, on a system that is "modern" but not terribly fast, and
only slightly (2-way) smp. (the spidernet)
I experimented wih various solutions, none were terribly exciting. The
thing that killed all of them was a crazy test case that someone sprung on
me: They had written a worst-case network ping-pong app: send one
packet, wait for reply, send one packet, etc.
If I waited (indefinitely) for a second packet to show up, the test case
completely stalled (since no second packet would ever arrive). And if I
introduced a timer to wait for a second packet, then I just increased
the latency in the response to the first packet, and this was noticed,
and folks complained.
Possible solution / possible brainfart:
Introduce a timer, but don't start to use it to combine packets unless you
receive n packets within the timeframe. If you receive less than m packets
within one timeframe, stop using the timer. The system should now have a
decent response time when the network is idle, and when the network is
busy, nobody will complain about the latency.-)
--
Funny quotes:
22. When everything's going your way, you're in the wrong lane and and going
the wrong way.
Friß, Spammer: rsRxhvmk@CaR.7eggert.dyndns.org m@z3T.7eggert.dyndns.org
On Fri, Aug 24, 2007 at 09:04:56PM +0200, Bodo Eggert wrote:
Linas Vepstas [off-list ref] wrote:
quoted
On Fri, Aug 24, 2007 at 03:59:16PM +0200, Jan-Bernd Themann wrote:
quoted
3) On modern systems the incoming packets are processed very fast. Especially
on SMP systems when we use multiple queues we process only a few packets
per napi poll cycle. So NAPI does not work very well here and the interrupt
rate is still high.
worst-case network ping-pong app: send one
packet, wait for reply, send one packet, etc.
Possible solution / possible brainfart:
Introduce a timer, but don't start to use it to combine packets unless you
receive n packets within the timeframe. If you receive less than m packets
within one timeframe, stop using the timer. The system should now have a
decent response time when the network is idle, and when the network is
busy, nobody will complain about the latency.-)
Ohh, that was inspirational. Let me free-associate some wild ideas.
Suppose we keep a running average of the recent packet arrival rate,
Lets say its 10 per millisecond ("typical" for a gigabit eth runnning
flat-out). If we could poll the driver at a rate of 10-20 per
millisecond (i.e. letting the OS do other useful work for 0.05 millisec),
then we could potentially service the card without ever having to enable
interrupts on the card, and without hurting latency.
If the packet arrival rate becomes slow enough, we go back to an
interrupt-driven scheme (to keep latency down).
The main problem here is that, even for HZ=1000 machines, this amounts
to 10-20 polls per jiffy. Which, if implemented in kernel, requires
using the high-resolution timers. And, umm, don't the HR timers require
a cpu timer interrupt to make them go? So its not clear that this is much
of a win.
The eHEA is a 10 gigabit device, so it can expect 80-100 packets per
millisecond for large packets, and even more, say 1K packets per
millisec, for small packets. (Even the spec for my 1Gb spidernet card
claims its internal rate is 1M packets/sec.)
Another possiblity is to set HZ to 5000 or 20000 or something humongous
... after all cpu's are now faster! But, since this might be wasteful,
maybe we could make HZ be dynamically variable: have high HZ rates when
there's lots of network/disk activity, and low HZ rates when not. That
means a non-constant jiffy.
If all drivers used interrupt mitigation, then the variable-high
frequency jiffy could take thier place, and be more "fair" to everyone.
Most drivers would be polled most of the time when they're busy, and
only use interrupts when they're not.
--linas
Linas Vepstas schrieb:
On Fri, Aug 24, 2007 at 09:04:56PM +0200, Bodo Eggert wrote:
quoted
Linas Vepstas [off-list ref] wrote:
quoted
On Fri, Aug 24, 2007 at 03:59:16PM +0200, Jan-Bernd Themann wrote:
quoted
3) On modern systems the incoming packets are processed very fast. Especially
on SMP systems when we use multiple queues we process only a few packets
per napi poll cycle. So NAPI does not work very well here and the interrupt
rate is still high.
worst-case network ping-pong app: send one
packet, wait for reply, send one packet, etc.
Possible solution / possible brainfart:
Introduce a timer, but don't start to use it to combine packets unless you
receive n packets within the timeframe. If you receive less than m packets
within one timeframe, stop using the timer. The system should now have a
decent response time when the network is idle, and when the network is
busy, nobody will complain about the latency.-)
Ohh, that was inspirational. Let me free-associate some wild ideas.
Suppose we keep a running average of the recent packet arrival rate,
Lets say its 10 per millisecond ("typical" for a gigabit eth runnning
flat-out). If we could poll the driver at a rate of 10-20 per
millisecond (i.e. letting the OS do other useful work for 0.05 millisec),
then we could potentially service the card without ever having to enable
interrupts on the card, and without hurting latency.
If the packet arrival rate becomes slow enough, we go back to an
interrupt-driven scheme (to keep latency down).
The main problem here is that, even for HZ=1000 machines, this amounts
to 10-20 polls per jiffy. Which, if implemented in kernel, requires
using the high-resolution timers. And, umm, don't the HR timers require
a cpu timer interrupt to make them go? So its not clear that this is much
of a win.
That is indeed a good question. At least for 10G eHEA we see
that the average number of packets/poll cycle is very low.
With high precision timers we could control the poll interval
better and thus make sure we get enough packets on the queue in
high load situations to benefit from LRO while keeping the
latency moderate. When the traffic load is low we could just
stick to plain NAPI. I don't know how expensive hp timers are,
we probably just have to test it (when they are available for
POWER in our case). However, having more packets
per poll run would make LRO more efficient and thus the total
CPU utilization would decrease.
I guess on most systems there are not many different network
cards working in parallel. So if the driver could set the poll
interval for its devices, it could be well optimized depending
on the NICs characteristics.
Maybe it would be good enough to have a timer that schedules
the device for NAPI (and thus triggers SoftIRQs, which will
trigger NAPI). Whether this timer would be used via a generic
interface or would be implemented as a proprietary solution
would depend on whether other drivers want / need this feature
as well. Drivers / NICs that work fine with plain NAPI don't
have to use timer :-)
I tried to implement something with "normal" timers, but the result
was everything but great. The timers seem to be far too slow.
I'm not sure if it helps to increase it from 1000HZ to 2500HZ
or more.
Regards,
Jan-Bernd
On Fri, Aug 24, 2007 at 11:11:56PM +0200, Jan-Bernd Themann wrote:
(when they are available for
POWER in our case).
hrtimer worked fine on the powerpc cell arch last summer.
I assume they work on p5 and p6 too, no ??
I tried to implement something with "normal" timers, but the result
was everything but great. The timers seem to be far too slow.
I'm not sure if it helps to increase it from 1000HZ to 2500HZ
or more.
Heh. Do the math. Even on 1gigabit cards, that's not enough:
(1gigabit/sec) x (byte/8 bits) x (packet/1500bytes) x (sec/1000 jiffy)
is 83 packets a jiffy (for big packets, even more for small packets,
and more again for 10 gigabit cards). So polling once per jiffy is a
latency disaster.
--linas