Re: linux-3.0.18+r8169+ipv4/tcp forwarding = tso/gso weirdness and performance degration

On Thu, 2012-03-15 at 17:11 +0200, Timo Teras wrote:

On Thu, 15 Mar 2012 08:06:35 +0200 Timo Teras [off-list ref] wrote:

quoted

On Wed, 14 Mar 2012 21:53:19 +0100 Francois Romieu
[off-list ref] wrote:

quoted

Timo Teras [off-list ref] :
[...]

quoted

# ethtool -S eth2
NIC statistics:
     tx_packets: 2069391193
     rx_packets: 3245815642
     tx_errors: 0
     rx_errors: 645238
     rx_missed: 31414

It does not look like stuff for the higher layers guys.

Can you tshark -w foobar on the sender side and
'while : ; do sleep 1; ethtool -S eth2 >> glop; done' on the
receiver during a bad wget (a big zero filled file should compress
well).

Indeed.

It seems that my earlier test about the "GRO off" effect were mistaken
(I used accidentally proxy, and that gave the illusion that things are
working. Whoops.)

So far I changed the cross-over cable and it didn't help. However,
forcing the NIC to 100mbit/full-duplex mode fixes the rx_errors. It
seems that something bad is happening in the gigabit mode.

I wonder if it's using pause frames and that's messing things up.
Seems that I can't turn it off, though.

I can also double check my cables, though it is factory made Cat-5E
cross-over cable; and happens with two different cables.

Ok. So far I have two of these boxes with same r8169 hardware. Both
generate bad packets on transmit only; and on both 3 nic systems it's
the middle eth1 nic. The symptoms are identical: in 1GB mode I have
minor packet loss, where as 100Mbit/s mode seems to work just fine.

The first box, that I've been talking so far about, is as mentioned
connected to another similar box. The r8169 there reports rx_errors.
The cable is ok; I've tried with two different ones.

The other broken box is connected to a HP ProCurve 4202vl-48G, and the
switch is reporting drops due to FCS Rx errors.

So I have two broken pieces of hardware, or there is a driver bug.

I'll try upgrading my kernel to 3.0.x series on the sender box and see
if it's fixing anything. Suggestions for further testing would be
appreciated.

r8169 has to make an additional copy of incoming frames, because of
hardware flaw and security requirements.

This was added in 2.6.37 or 2.6.38, dont remember exactly.

So your cpu might be to slow to handle the load at 1Gb speed.

If you have one flow, there is nothing to do, but if your workload has
several flows and your machine is SMP, you can try RPS/RFS as documented
in Documentation/networking/scaling.txt