Re: debugging TCP stalls on high-speed wifi

On 12/12/19 1:11 PM, Johannes Berg wrote:

Hi Eric,

Thanks for looking :)

quoted

quoted

I'm not sure how to do headers-only, but I guess -s100 will work.

https://johannes.sipsolutions.net/files/he-tcp.pcap.xz

Lack of GRO on receiver is probably what is killing performance,
both for receiver (generating gazillions of acks) and sender
(to process all these acks)

Yes, I'm aware of this, to some extent. And I'm not saying we should see
even close to 1800 Mbps like we have with UDP...

Mind you, the biggest thing that kills performance with many ACKs isn't
the load on the system - the sender system is only moderately loaded at
~20-25% of a single core with TSO, and around double that without TSO.
The thing that kills performance is eating up all the medium time with
small non-aggregated packets, due to the the half-duplex nature of WiFi.
I know you know, but in case somebody else is reading along :-)

But unless somehow you think processing the (many) ACKs on the sender
will cause it to stop transmitting, or something like that, I don't
think I should be seeing what I described earlier: we sometimes (have
to?) reclaim the entire transmit queue before TCP starts pushing data
again. That's less than 2MB split across at least two TCP streams, I
don't see why we should have to get to 0 (which takes about 7ms) until
more packets come in from TCP?

Or put another way - if I free say 400kB worth of SKBs, what could be
the reason we don't see more packets be sent out of the TCP stack within
the few ms or so? I guess I have to correlate this somehow with the ACKs
so I know how much data is outstanding for ACKs. (*)

The sk_pacing_shift is set to 7, btw, which should give us 8ms of
outstanding data. For now in this setup that's enough(**), and indeed
bumping the limit up (setting sk_pacing_shift to say 5) doesn't change
anything. So I think this part we actually solved - I get basically the
same performance and behaviour with two streams (needed due to GBit LAN
on the other side) as with 20 streams.

quoted

I had a plan about enabling compressing ACK as I did for SACK
in commit 
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d9f4262b7ea41ca9981cc790e37cca6e37c789e

But I have not done it yet.
It is a pity because this would tremendously help wifi I am sure.

Nice :-)

But that is something the *receiver* would have to do.

Yes, this is the plan. Eventually the receiver gets smarter.

The dirty secret here is that we're getting close to 1700 Mbps TCP with
Windows in place of Linux in the setup, with the same receiver on the
other end (which is actually a single Linux machine with two GBit
network connections to the AP). So if we had this I'm sure it'd increase
performance, but it still wouldn't explain why we're so much slower than
Windows :-)

I presume you could hack TCP to no longer care about bufferbloat and you'll
probably match Windows 'performance' on a single flow and a lossless network.

Ie always send ~64KB TSO packets and fill the queues, inflating RTT.

Then, in presence of losses, you get a problem because the retransmit packets
can only be sent _after_ the huge queue that has been put on the sender.

If only TCP could predict the future ;)