From: Ben Greear <hidden> Date: 2012-06-04 18:29:11
I'm going some TCP performance testing on wifi -> LAN interface connections. With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.
I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per second in
upload direction, 2000 pps download. And the vast majority of the download pkts
are 66 byte ACK pkts from what I can tell.)
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a
bit longer and send more ACK data in fewer packets?
Packet traces and other info available if anyone wants to take a look.
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Daniel Baluta <hidden> Date: 2012-06-04 19:22:02
On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear [off-list ref] wrote:
I'm going some TCP performance testing on wifi -> LAN interface connections.
With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is
about 80Mbps.
I think the problem is that there are way too many ACK packets, and
bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per
second in
upload direction, 2000 pps download. And the vast majority of the download
pkts
are 66 byte ACK pkts from what I can tell.)
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket
wait a
bit longer and send more ACK data in fewer packets?
An ACK is generated after every second full sized segment or a timeout
expires.
Currently, there is no way to tune these parameters. Here is an experimental
patch [1]. If anyone, thinks that this patch has a chance to get accepted
I will be happily try to further improve it.
Packet traces and other info available if anyone wants to take a look.
From: Eric Dumazet <hidden> Date: 2012-06-04 19:32:38
On Mon, 2012-06-04 at 11:29 -0700, Ben Greear wrote:
I'm going some TCP performance testing on wifi -> LAN interface connections. With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.
I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per second in
upload direction, 2000 pps download. And the vast majority of the download pkts
are 66 byte ACK pkts from what I can tell.)
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a
bit longer and send more ACK data in fewer packets?
Well, thats half duplex days...
There is the ACK every 2 packets rule of thumb, so that tcp sender can
increase its cwnd.
Then, some folks tried submitting patches to make '2' more like 10 or
15, but this went nowhere.
Other idea is to arm a timer and defer ACK sending, in the hope we
receive another packet very soon.
That could be done with a special qdisc, sort of netem...
From: Ben Greear <hidden> Date: 2012-06-04 20:09:14
On 06/04/2012 12:22 PM, Daniel Baluta wrote:
On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear[off-list ref] wrote:
quoted
I'm going some TCP performance testing on wifi -> LAN interface connections.
With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is
about 80Mbps.
I think the problem is that there are way too many ACK packets, and
bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per
second in
upload direction, 2000 pps download. And the vast majority of the download
pkts
are 66 byte ACK pkts from what I can tell.)
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket
wait a
bit longer and send more ACK data in fewer packets?
An ACK is generated after every second full sized segment or a timeout
expires.
Currently, there is no way to tune these parameters. Here is an experimental
patch [1]. If anyone, thinks that this patch has a chance to get accepted
I will be happily try to further improve it.
It looks like it could be useful for my case, but I would also want
per-socket options to set the min/max ack delay so that the settings
are not just system-wide.
I can at least test this, and perhaps even hack on the code if
you are not interested in the per-socket settings...
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Ben Greear <hidden> Date: 2012-06-04 20:12:21
On 06/04/2012 12:32 PM, Eric Dumazet wrote:
On Mon, 2012-06-04 at 11:29 -0700, Ben Greear wrote:
quoted
I'm going some TCP performance testing on wifi -> LAN interface connections. With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is about 80Mbps.
I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per second in
upload direction, 2000 pps download. And the vast majority of the download pkts
are 66 byte ACK pkts from what I can tell.)
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket wait a
bit longer and send more ACK data in fewer packets?
Well, thats half duplex days...
WiFi is still half-duplex, and isn't changing any time soon I think.
There is the ACK every 2 packets rule of thumb, so that tcp sender can
increase its cwnd.
Then, some folks tried submitting patches to make '2' more like 10 or
15, but this went nowhere.
Other idea is to arm a timer and defer ACK sending, in the hope we
receive another packet very soon.
From a quick read of Daniel's patch..it seems there is already a
timer that could be used for this?
That could be done with a special qdisc, sort of netem...
That sounds like a horrible hack :)
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Daniel Baluta <hidden> Date: 2012-06-04 20:15:48
On Mon, Jun 4, 2012 at 11:09 PM, Ben Greear [off-list ref] wrote:
On 06/04/2012 12:22 PM, Daniel Baluta wrote:
quoted
On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear[off-list ref]
wrote:
quoted
I'm going some TCP performance testing on wifi -> LAN interface
connections.
With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is
about 80Mbps.
I think the problem is that there are way too many ACK packets, and
bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts
per
second in
upload direction, 2000 pps download. And the vast majority of the
download
pkts
are 66 byte ACK pkts from what I can tell.)
Kernel is 3.3.7+
Anyone know of any tuning parameters that would let the receiving socket
wait a
bit longer and send more ACK data in fewer packets?
An ACK is generated after every second full sized segment or a timeout
expires.
Currently, there is no way to tune these parameters. Here is an
experimental
patch [1]. If anyone, thinks that this patch has a chance to get accepted
I will be happily try to further improve it.
It looks like it could be useful for my case, but I would also want
per-socket options to set the min/max ack delay so that the settings
are not just system-wide.
I can at least test this, and perhaps even hack on the code if
you are not interested in the per-socket settings...
Yes, indeed this should be a per socket option. I will be
back with a patch in the next days.
Meanwhile, it would be great if you could test my previous patch.
thanks,
Daniel.
Anyone know of any tuning parameters that would let the receiving socket wait a
bit longer and send more ACK data in fewer packets?
It's perhaps better to think of Acks as offering an opportunity for the transmitting TCP algorithm to sample the congestion of the path. That's particularly important in Slow Start (when the transmitting algorithm is trying to send data exponentially faster) and both traditional TCP and Linux's CUBIC+Hystart will not ramp up transmission as quickly as they otherwise would when starved of samples.
Of course you've got to weigh slower ramp up in transmission rate against the loss in throughput due to the turn-around time on the half-duplex wireless media. But I don't see the proposed patch doing any weighing of considerations at all, but punting that to user space which has even less information about the condition of TCP (particularly the receiver's guess as to the slow-start/congestion-avoidance mode of the transmitter) and less information about the attributes of the media (particularly the turn-around time of half-duplex media on the path) to make a decision with.
There's a nice summary of the tradeoffs of delayed Acks in RFC2525 section 2.13, although the discussion is dated (pre-ABC, pre-HyStart).
Note the firm requirement in RFC2581 section 4.2 that an Ack be sent within 500ms. There are implementations in the field which have delayed Acks out to 2s, these are notoriously painful and many modern algorithms drop back into a low performance mode when facing those receivers.
- Glen
From: Ben Greear <hidden> Date: 2012-06-07 00:26:44
On 06/04/2012 12:22 PM, Daniel Baluta wrote:
Currently, there is no way to tune these parameters. Here is an experimental
patch [1]. If anyone, thinks that this patch has a chance to get accepted
I will be happily try to further improve it.
quoted
Packet traces and other info available if anyone wants to take a look.
I tried your patch in 3.5.0-rc1.
Doesn't seem to help any...do you have suggested settings for
the proc values? (I tried increasing tcp_delack_segs up to 10,
but no significant increase...)
I'll keep poking around.
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: Ben Greear <hidden> Date: 2012-06-07 00:40:59
On 06/06/2012 05:26 PM, Ben Greear wrote:
On 06/04/2012 12:22 PM, Daniel Baluta wrote:
quoted
Currently, there is no way to tune these parameters. Here is an experimental
patch [1]. If anyone, thinks that this patch has a chance to get accepted
I will be happily try to further improve it.
quoted
Packet traces and other info available if anyone wants to take a look.
I tried your patch in 3.5.0-rc1.
Doesn't seem to help any...do you have suggested settings for
the proc values? (I tried increasing tcp_delack_segs up to 10,
but no significant increase...)
I'll keep poking around.
Ahh, forcing send-buffer to be large helps significantly..wonder
if that was my original problem.
Will get some more useful results tomorrow...
Thanks,
Ben
From: Ben Greear <hidden> Date: 2012-06-07 04:15:59
On 06/04/2012 12:22 PM, Daniel Baluta wrote:
On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear[off-list ref] wrote:
quoted
I'm going some TCP performance testing on wifi -> LAN interface connections.
With
UDP, we can get around 250Mbps of payload throughput. With TCP, max is
about 80Mbps.
I think the problem is that there are way too many ACK packets, and
bi-directional
traffic on wifi interfaces really slows things down. (About 7000 pkts per
second in
upload direction, 2000 pps download. And the vast majority of the download
pkts
are 66 byte ACK pkts from what I can tell.)
After a bit more playing, I did notice a reliable 5% increase in
traffic (200Mbps -> 210Mbps) from changing the delack segments
to 20 from the default of 1. That is enough to be useful to me,
and there may be more significant gains to be found...
I haven't done a full matrix of testing yet.
I read through the original thread, and to summarize:
* Need to make the values per-socket.
* No multiplication in hot path.
* Would be nice to make it automatic.
The first seems fairly trivial..just add a new set of socket-opts (or
maybe just one that can take all 3 values?) and store the settings in
the socket structs.
As for getting rid of the multiply..I think you cannot just use shifts.
That does not give the needed granularity. An alternative is
to update a cached computation every time the mss or socket-opt changes.
As for a magic heuristic to figure this out, I think that would be
quite tricky to do right. So, maybe add that in the future, but
for the present, just allowing applications to set the value seems
enough.
We could support configurable system-wide defaults so that users of
programs that do not know this new sockopt can still take advantage
of the feature.
Does this sound like a reasonable solution?
Thanks,
Ben
--
Ben Greear [off-list ref]
Candela Technologies Inc http://www.candelatech.com
From: David Laight <hidden> Date: 2012-06-07 12:29:36
-----Original Message-----
From: netdev-owner@vger.kernel.org
[mailto:netdev-owner@vger.kernel.org] On Behalf Of Ben Greear
Sent: 07 June 2012 05:16
To: Daniel Baluta
Cc: netdev
Subject: Re: tcp wifi upload performance and lots of ACKs
On 06/04/2012 12:22 PM, Daniel Baluta wrote:
quoted
On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear[off-list ref]
wrote:
quoted
quoted
I'm going some TCP performance testing on wifi -> LAN interface
connections.
quoted
quoted
With
UDP, we can get around 250Mbps of payload throughput. With TCP,
max is
quoted
quoted
about 80Mbps.
I think the problem is that there are way too many ACK packets, and
bi-directional
traffic on wifi interfaces really slows things down. (About 7000
pkts per
quoted
quoted
second in
upload direction, 2000 pps download. And the vast majority of the
After a bit more playing, I did notice a reliable 5% increase in
traffic (200Mbps -> 210Mbps) from changing the delack segments
to 20 from the default of 1. That is enough to be useful to me,
and there may be more significant gains to be found...
I haven't done a full matrix of testing yet.
Does this delaying of acks have a detrimental effect on the
sending end?
I've seen very bad interactions between delayed acks and
(I believe) the 'slow start' code on connections with
one-directional data, Nagle disabled and very low RTT.
What I saw was the sender sending 4 data packets, then
sitting waiting for an ack - in spite of accumulating
several kB of data to send.
Delaying acks further will only make this worse.
David
From: Ben Greear <hidden> Date: 2012-06-07 14:41:57
On 06/07/2012 05:20 AM, David Laight wrote:
quoted
-----Original Message-----
From: netdev-owner@vger.kernel.org
[mailto:netdev-owner@vger.kernel.org] On Behalf Of Ben Greear
Sent: 07 June 2012 05:16
To: Daniel Baluta
Cc: netdev
Subject: Re: tcp wifi upload performance and lots of ACKs
On 06/04/2012 12:22 PM, Daniel Baluta wrote:
quoted
On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear[off-list ref]
wrote:
quoted
quoted
quoted
I'm going some TCP performance testing on wifi -> LAN interface
connections.
quoted
quoted
quoted
With
UDP, we can get around 250Mbps of payload throughput. With TCP,
max is
quoted
quoted
quoted
about 80Mbps.
I think the problem is that there are way too many ACK packets, and
bi-directional
traffic on wifi interfaces really slows things down. (About 7000
pkts per
quoted
quoted
quoted
second in
upload direction, 2000 pps download. And the vast majority of the
After a bit more playing, I did notice a reliable 5% increase in
traffic (200Mbps -> 210Mbps) from changing the delack segments
to 20 from the default of 1. That is enough to be useful to me,
and there may be more significant gains to be found...
I haven't done a full matrix of testing yet.
Does this delaying of acks have a detrimental effect on the
sending end?
I've seen very bad interactions between delayed acks and
(I believe) the 'slow start' code on connections with
one-directional data, Nagle disabled and very low RTT.
What I saw was the sender sending 4 data packets, then
sitting waiting for an ack - in spite of accumulating
several kB of data to send.
Delaying acks further will only make this worse.
I'm sure it's not for everyone in all cases. In my case, I'm
sending long-term bulk transfer, at high speeds, over wifi network
which has some latency. Tested one-way traffic so far.
With the patch and delayed acks, I get more sender throughput than
without (200Mbps -> 210Mbps).
Thanks,
Ben
From: Rick Jones <hidden> Date: 2012-06-07 17:51:22
Does this delaying of acks have a detrimental effect on the
sending end?
I've seen very bad interactions between delayed acks and
(I believe) the 'slow start' code on connections with
one-directional data, Nagle disabled and very low RTT.
What I saw was the sender sending 4 data packets, then
sitting waiting for an ack - in spite of accumulating
several kB of data to send.
Delaying acks further will only make this worse.
At least two stacks have a reasonable ACK avoidance heuristic. Those would be HP-UX and Solaris (Mac OS 9 had one as well, IIRC). The heuristics are rather similar because the two TCP stacks share a common ancestor. I used to interact with HP-UX's regularly, my statements will be based on that, and an assumption Solaris is similar.
Both attempt to divine what the senders' congestion window happens to be and be certain to send an ACK before that is exhausted. So, at the start of a connection, there will be the usual, more rapid ACKnowledgement. As things happen "normally" then the number of segments per ACK increases until it hits a configurable limit. There are conditions which will cut the limit in half on a given connection - one is the sending of an ACK via the standalone ACK timer (this is from memory, so may be a bit off). There are probably a few other conditions that drop the limit by half. The heuristic attempts to learn in each connection what the reasonable limit on ACK avoidance might be so there isn't a per-connection control, just the global controls via ndd. As conditions causing the limit to be cut in half arise, the connection naturally and irrevocably falls back to the usual "ack-every-other" behaviour.
When there is little to no packet loss, and a rather regular stream of data, this works rather well indeed. For example in a LAN or Data Center. You can run netperf TCP_STREAM with the limit at the default, and with the limit set to two and see the considerable difference in service demand on either side.
This may not work well when the sender has a congestion window growth heuristic different from what the ACK avoidance heuristic assumes. If I recall correctly, the heuristic in HP-UX assumes the sender grows cwnd by the number of bytes/segments ACKnowledged. If the sender grows the cwnd by only one segment per ACK rather than by the bytes ACKnowldeged by the ACK the growth of the cwnd will be slowed. In a LAN that may be papered-over a bit, but it will become quite noticable in a higher RTT environment. Probably not as noticable for a sufficiently short connection, or a long one, but will be for ones in the middle. The short connection doesn't need much cwnd in the first place, and the heuristic works its way up to avoiding ACKs, and the long one will be long enough to have the ACK avoidance heuristic gravitate down to ack-every-other.
rick jones
From: David Miller <davem@davemloft.net> Date: 2012-06-07 18:10:14
From: Rick Jones <redacted>
Date: Thu, 07 Jun 2012 10:51:20 -0700
This may not work well when the sender has a congestion window growth
heuristic different from what the ACK avoidance heuristic assumes. If
I recall correctly, the heuristic in HP-UX assumes the sender grows
cwnd by the number of bytes/segments ACKnowledged.