Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

9 messages, 3 authors, 2015-02-09 · open the first message on its own page

Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Avery Fay <hidden>
Date: 2015-02-04 06:50:55 

Hello,

Let me say first: if there's a better place to ask this, please point
me in that direction.

We've been having huge packets / sec spikes in the past few days.
After some investigation, it looks like single connections are getting
stuck in a loop (see tcpdump below). Each "stuck" connection will
generate about 200kpps. It looks like our side is rejecting packets
with "packets rejects in established connections because of timestamp"
from netstat -s (internally PAWSEstab counter) and then generating an
additional packet that we send out. All of these connections originate
from georgia tech, but so far (not completely verified) it doesn't
seem like there's any pattern to the client/os other than the fact
that they're trying to make an https request to us.

As a temporary countermeasure, we've disabled net.ipv4.tcp_timestamps,
which solves the immediate problem.

Our server is 174.36.240.86 running Ubuntu 12.04 with kernel 3.13.0-35-generic

The client is 128.61.57.205 and in this case almost certainly has user
agent (we found successful requests 10 seconds before the tcpdump with
same ip): Dalvik/2.1.0 (Linux; U; Android 5.0; XT1095
Build/LXE22.46-11)

Beginning of tcpdump:

05:36:10.723423 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [S],
seq 4192140517, win 65535, options [mss 1380,sackOK,TS val 3985707 ecr
0,nop,wscale 8], length 0
05:36:10.723431 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags
[S.], seq 1191654135, ack 4192140518, win 28960, options [mss
1460,sackOK,TS val 3311875276 ecr 3985707,nop,wscale 7], length 0
05:36:10.742480 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [.],
ack 1, win 343, options [nop,nop,TS val 3985710 ecr 3311875276],
length 0
05:36:10.742534 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags [.],
ack 1, win 343, options [nop,nop,TS val 3985710 ecr 3311875276],
length 0
05:36:10.761060 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], seq 1:227, ack 1, win 343, options [nop,nop,TS val 3985712 ecr
3311875276], length 226
05:36:10.761067 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 227, win 235, options [nop,nop,TS val 3311875285 ecr 3985712],
length 0
05:36:10.761122 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
seq 1:2737, ack 227, win 235, options [nop,nop,TS val 3311875285 ecr
3985712], length 2736
05:36:10.761129 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags
[P.], seq 2737:3256, ack 227, win 235, options [nop,nop,TS val
3311875285 ecr 3985712], length 519
05:36:10.798133 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags
[P.], seq 2737:3256, ack 227, win 235, options [nop,nop,TS val
3311875295 ecr 3985712], length 519
05:36:11.018130 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311875350 ecr
3985712], length 1368
05:36:11.458134 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311875460 ecr
3985712], length 1368
05:36:12.338130 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311875680 ecr
3985712], length 1368
05:36:14.102128 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311876121 ecr
3985712], length 1368
05:36:17.634130 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
seq 1:1369, ack 227, win 235, options [nop,nop,TS val 3311877004 ecr
3985712], length 1368
05:36:20.820850 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[F.], seq 227, ack 1, win 343, options [nop,nop,TS val 3986717 ecr
3311875285], length 0
05:36:20.820879 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags
[F.], seq 3256, ack 228, win 235, options [nop,nop,TS val 3311877800
ecr 3986717], length 0
05:36:20.835368 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr
3311875276,nop,nop,sack 1 {3256:3256}], length 0
05:36:20.835373 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877804 ecr 3986717],
length 0
05:36:20.835375 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.835378 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877804 ecr 3986717],
length 0
05:36:20.849735 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr
3311875276,nop,nop,sack 2 {3257:3257}{3256:3256}], length 0
05:36:20.849740 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877807 ecr 3986717],
length 0
05:36:20.849783 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.849785 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877807 ecr 3986717],
length 0
05:36:20.849787 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.849788 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877807 ecr 3986717],
length 0
05:36:20.849789 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.849790 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877807 ecr 3986717],
length 0
05:36:20.864132 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.864137 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717],
length 0
05:36:20.864185 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.864189 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717],
length 0
05:36:20.864226 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.864228 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717],
length 0
05:36:20.864229 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.864230 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717],
length 0
05:36:20.864231 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.864233 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717],
length 0
05:36:20.864282 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.864285 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877811 ecr 3986717],
length 0
05:36:20.878523 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878530 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878532 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878534 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878620 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878624 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878625 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878627 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878628 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878629 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878630 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878632 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878632 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878634 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878635 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878636 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878672 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878674 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878678 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878679 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878724 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878726 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0
05:36:20.878816 IP 128.61.57.205.34574 > 174.36.240.86.443: Flags
[P.], ack 1, win 343, options [nop,nop,TS val 3985712 ecr 3311875276],
length 0
05:36:20.878818 IP 174.36.240.86.443 > 128.61.57.205.34574: Flags [.],
ack 228, win 235, options [nop,nop,TS val 3311877815 ecr 3986717],
length 0

At this point, it just repeats until some timeout is hit. I haven't
timed it, but probably one or two minutes.

I guess I have a few questions:

1.) What's going on here? It looks like maybe there's some packet loss
and then connection termination gets stuck in a loop because the
client timestamp went down?
2.) Is there a better way to mitigate this other than disabling
tcp_timestamps or blocking gatech ips?
3.) Is this our problem (ok, obviously our problem since we're
affected but...), a kernel problem, or a gatech problem?

I'd really appreciate any help on this,

Avery

Re: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Eric Dumazet <hidden>
Date: 2015-02-04 08:03:07 

On Tue, 2015-02-03 at 22:50 -0800, Avery Fay wrote:
Hello,

Let me say first: if there's a better place to ask this, please point
me in that direction.

We've been having huge packets / sec spikes in the past few days.
After some investigation, it looks like single connections are getting
stuck in a loop (see tcpdump below). Each "stuck" connection will
generate about 200kpps. It looks like our side is rejecting packets
with "packets rejects in established connections because of timestamp"
from netstat -s (internally PAWSEstab counter) and then generating an
additional packet that we send out. All of these connections originate
from georgia tech, but so far (not completely verified) it doesn't
seem like there's any pattern to the client/os other than the fact
that they're trying to make an https request to us.

As a temporary countermeasure, we've disabled net.ipv4.tcp_timestamps,
which solves the immediate problem.

Our server is 174.36.240.86 running Ubuntu 12.04 with kernel 3.13.0-35-generic

The client is 128.61.57.205 and in this case almost certainly has user
agent (we found successful requests 10 seconds before the tcpdump with
same ip): Dalvik/2.1.0 (Linux; U; Android 5.0; XT1095
Build/LXE22.46-11)

Beginning of tcpdump:
...

At this point, it just repeats until some timeout is hit. I haven't
timed it, but probably one or two minutes.

I guess I have a few questions:

1.) What's going on here? It looks like maybe there's some packet loss
and then connection termination gets stuck in a loop because the
client timestamp went down?
2.) Is there a better way to mitigate this other than disabling
tcp_timestamps or blocking gatech ips?
3.) Is this our problem (ok, obviously our problem since we're
affected but...), a kernel problem, or a gatech problem?

I'd really appreciate any help on this,
Would you have a pcap file instead ?

It looks a middlebox is broken, I dont think Android could possibly send
a frame with no payload, but with Push flag.

Neal has some patches that add a rate limiting on DACKS, that we might
upstream. (per socket rate limiting of 2 DACK per second)

Thanks

Re: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Avery Fay <hidden>
Date: 2015-02-04 08:35:39 

Sure, https://dl.dropboxusercontent.com/u/9777748/loop.pcap.gz

Also, no idea if it helps, but here's the traceroute:

HOST: apibalancer-wdc-05                          Loss%   Snt   Last
Avg  Best  Wrst StDev
  1.|-- 184.173.130.1-static.reverse.softlayer.com   0.0%    10    0.2
  1.9   0.2   9.6   3.1
  2.|-- 208.43.118.164-static.reverse.softlayer.com  0.0%    10    0.2
  0.2   0.2   0.3   0.0
  3.|-- ae8.bbr02.eq01.wdc02.networklayer.com        0.0%    10    1.2
  1.3   1.1   2.6   0.5
  4.|-- ash-b1-link.telia.net                        0.0%    10    1.2
  4.3   1.1  12.7   5.0
  5.|-- ash-bb3-link.telia.net                       0.0%    10    1.2
  2.6   1.1  15.0   4.4
  6.|-- atl-bb1-link.telia.net                       0.0%    10   13.4
 13.3  13.3  13.4   0.0
  7.|-- 213.248.94.220                               0.0%    10   14.2
 14.2  14.2  14.4   0.1
  8.|-- 130.207.254.6                                0.0%    10   14.2
 14.2  14.1  14.3   0.1
    |  `|-- 130.207.254.185
  9.|-- gateway2-rtr.gatech.edu                      0.0%    10   14.2
 14.3  14.1  14.8   0.2
    |  `|-- 143.215.254.97
 10.|-- 143.215.254.97                               0.0%    10   14.6
 14.6  14.4  14.7   0.1
    |  `|-- 143.215.253.114
 11.|-- 143.215.253.114                              0.0%    10   15.1
 14.7  14.5  15.1   0.2
 12.|-- ???                                         100.0    10    0.0
  0.0   0.0   0.0   0.0

On Wed, Feb 4, 2015 at 12:03 AM, Eric Dumazet [off-list ref] wrote:
On Tue, 2015-02-03 at 22:50 -0800, Avery Fay wrote:
quoted
Hello,

Let me say first: if there's a better place to ask this, please point
me in that direction.

We've been having huge packets / sec spikes in the past few days.
After some investigation, it looks like single connections are getting
stuck in a loop (see tcpdump below). Each "stuck" connection will
generate about 200kpps. It looks like our side is rejecting packets
with "packets rejects in established connections because of timestamp"
from netstat -s (internally PAWSEstab counter) and then generating an
additional packet that we send out. All of these connections originate
from georgia tech, but so far (not completely verified) it doesn't
seem like there's any pattern to the client/os other than the fact
that they're trying to make an https request to us.

As a temporary countermeasure, we've disabled net.ipv4.tcp_timestamps,
which solves the immediate problem.

Our server is 174.36.240.86 running Ubuntu 12.04 with kernel 3.13.0-35-generic

The client is 128.61.57.205 and in this case almost certainly has user
agent (we found successful requests 10 seconds before the tcpdump with
same ip): Dalvik/2.1.0 (Linux; U; Android 5.0; XT1095
Build/LXE22.46-11)

Beginning of tcpdump:
...
quoted
At this point, it just repeats until some timeout is hit. I haven't
timed it, but probably one or two minutes.

I guess I have a few questions:

1.) What's going on here? It looks like maybe there's some packet loss
and then connection termination gets stuck in a loop because the
client timestamp went down?
2.) Is there a better way to mitigate this other than disabling
tcp_timestamps or blocking gatech ips?
3.) Is this our problem (ok, obviously our problem since we're
affected but...), a kernel problem, or a gatech problem?

I'd really appreciate any help on this,
Would you have a pcap file instead ?

It looks a middlebox is broken, I dont think Android could possibly send
a frame with no payload, but with Push flag.

Neal has some patches that add a rate limiting on DACKS, that we might
upstream. (per socket rate limiting of 2 DACK per second)

Thanks

Re: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Avery Fay <hidden>
Date: 2015-02-04 08:37:47 

Sorry, forgot to mention: the packet capture has been filtered to only
the ip/port combo from above. I also stopped the capture early before
the traffic would naturally stop because I wanted to disable
timestamps again as soon as possible.

Avery

On Wed, Feb 4, 2015 at 12:35 AM, Avery Fay [off-list ref] wrote:
Sure, https://dl.dropboxusercontent.com/u/9777748/loop.pcap.gz

Also, no idea if it helps, but here's the traceroute:

HOST: apibalancer-wdc-05                          Loss%   Snt   Last
Avg  Best  Wrst StDev
  1.|-- 184.173.130.1-static.reverse.softlayer.com   0.0%    10    0.2
  1.9   0.2   9.6   3.1
  2.|-- 208.43.118.164-static.reverse.softlayer.com  0.0%    10    0.2
  0.2   0.2   0.3   0.0
  3.|-- ae8.bbr02.eq01.wdc02.networklayer.com        0.0%    10    1.2
  1.3   1.1   2.6   0.5
  4.|-- ash-b1-link.telia.net                        0.0%    10    1.2
  4.3   1.1  12.7   5.0
  5.|-- ash-bb3-link.telia.net                       0.0%    10    1.2
  2.6   1.1  15.0   4.4
  6.|-- atl-bb1-link.telia.net                       0.0%    10   13.4
 13.3 13.3 13.4 0.0
  7.|-- 213.248.94.220                               0.0%    10   14.2
 14.2  14.2  14.4   0.1
  8.|-- 130.207.254.6                                0.0%    10   14.2
 14.2  14.1  14.3   0.1
    |  `|-- 130.207.254.185
  9.|-- gateway2-rtr.gatech.edu                      0.0%    10   14.2
 14.3 14.1 14.8 0.2
    |  `|-- 143.215.254.97
 10.|-- 143.215.254.97                               0.0%    10   14.6
 14.6  14.4  14.7   0.1
    |  `|-- 143.215.253.114
 11.|-- 143.215.253.114                              0.0%    10   15.1
 14.7  14.5  15.1   0.2
 12.|-- ???                                         100.0    10    0.0
  0.0   0.0   0.0   0.0

On Wed, Feb 4, 2015 at 12:03 AM, Eric Dumazet [off-list ref] wrote:
quoted
On Tue, 2015-02-03 at 22:50 -0800, Avery Fay wrote:
quoted
Hello,

Let me say first: if there's a better place to ask this, please point
me in that direction.

We've been having huge packets / sec spikes in the past few days.
After some investigation, it looks like single connections are getting
stuck in a loop (see tcpdump below). Each "stuck" connection will
generate about 200kpps. It looks like our side is rejecting packets
with "packets rejects in established connections because of timestamp"
from netstat -s (internally PAWSEstab counter) and then generating an
additional packet that we send out. All of these connections originate
from georgia tech, but so far (not completely verified) it doesn't
seem like there's any pattern to the client/os other than the fact
that they're trying to make an https request to us.

As a temporary countermeasure, we've disabled net.ipv4.tcp_timestamps,
which solves the immediate problem.

Our server is 174.36.240.86 running Ubuntu 12.04 with kernel 3.13.0-35-generic

The client is 128.61.57.205 and in this case almost certainly has user
agent (we found successful requests 10 seconds before the tcpdump with
same ip): Dalvik/2.1.0 (Linux; U; Android 5.0; XT1095
Build/LXE22.46-11)

Beginning of tcpdump:
...
quoted
At this point, it just repeats until some timeout is hit. I haven't
timed it, but probably one or two minutes.

I guess I have a few questions:

1.) What's going on here? It looks like maybe there's some packet loss
and then connection termination gets stuck in a loop because the
client timestamp went down?
2.) Is there a better way to mitigate this other than disabling
tcp_timestamps or blocking gatech ips?
3.) Is this our problem (ok, obviously our problem since we're
affected but...), a kernel problem, or a gatech problem?

I'd really appreciate any help on this,
Would you have a pcap file instead ?

It looks a middlebox is broken, I dont think Android could possibly send
a frame with no payload, but with Push flag.

Neal has some patches that add a rate limiting on DACKS, that we might
upstream. (per socket rate limiting of 2 DACK per second)

Thanks

Re: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Eric Dumazet <hidden>
Date: 2015-02-04 12:20:32 

On Wed, 2015-02-04 at 00:35 -0800, Avery Fay wrote:
Sure, https://dl.dropboxusercontent.com/u/9777748/loop.pcap.gz
Nice, something is really broken on this peer (not a linux one for
sure), or some really buggy middlebox.

The strange [P.] frame with no payload has a wrong sequence number
anyway, so we send back a duplicate ack.

  1   0.000000 128.61.57.205 -> 174.36.240.86 TCP 74 34574 > https [SYN] Seq=0 Win=65535 Len=0 MSS=1380 SACK_PERM=1 TSval=3985707 TSecr=0 WS=256
  2   0.000008 174.36.240.86 -> 128.61.57.205 TCP 74 https > 34574 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=3311875276 TSecr=3985707 WS=128
  3   0.019057 128.61.57.205 -> 174.36.240.86 TCP 66 34574 > https [ACK] Seq=1 Ack=1 Win=87808 Len=0 TSval=3985710 TSecr=3311875276
  4   0.019111 128.61.57.205 -> 174.36.240.86 TCP 66 [TCP Dup ACK 3#1] 34574 > https [ACK] Seq=1 Ack=1 Win=87808 Len=0 TSval=3985710 TSecr=3311875276
  5   0.037637 128.61.57.205 -> 174.36.240.86 SSL 292 Client Hello
  6   0.037644 174.36.240.86 -> 128.61.57.205 TCP 66 https > 34574 [ACK] Seq=1 Ack=227 Win=30080 Len=0 TSval=3311875285 TSecr=3985712
  7   0.037699 174.36.240.86 -> 128.61.57.205 TLSv1.2 2802 Server Hello
  8   0.037706 174.36.240.86 -> 128.61.57.205 TLSv1.2 585 Certificate
  9   0.074710 174.36.240.86 -> 128.61.57.205 TCP 585 [TCP Retransmission] https > 34574 [PSH, ACK] Seq=2737 Ack=227 Win=30080 Len=519 TSval=3311875295 TSecr=3985712[Reassembly error, protocol TCP: New fragment overlaps ol
d data (retransmission?)]
 10   0.294707 174.36.240.86 -> 128.61.57.205 TLSv1.2 1434 [TCP Retransmission] Server Hello
 11   0.734711 174.36.240.86 -> 128.61.57.205 TLSv1.2 1434 [TCP Retransmission] Server Hello
 12   1.614707 174.36.240.86 -> 128.61.57.205 TLSv1.2 1434 [TCP Retransmission] Server Hello
 13   3.378705 174.36.240.86 -> 128.61.57.205 TLSv1.2 1434 [TCP Retransmission] Server Hello
 14   6.910707 174.36.240.86 -> 128.61.57.205 TLSv1.2 1434 [TCP Retransmission] Server Hello
 15  10.097427 128.61.57.205 -> 174.36.240.86 TCP 66 34574 > https [FIN, ACK] Seq=227 Ack=1 Win=87808 Len=0 TSval=3986717 TSecr=3311875285
 16  10.097456 174.36.240.86 -> 128.61.57.205 TCP 66 https > 34574 [FIN, ACK] Seq=3256 Ack=228 Win=30080 Len=0 TSval=3311877800 TSecr=3986717
 17  10.111945 128.61.57.205 -> 174.36.240.86 TCP 78 [TCP Dup ACK 15#1] 34574 > https [PSH, ACK] Seq=228 Ack=1 Win=87808 Len=0 TSval=3985712 TSecr=3311875276 SLE=3256 SRE=3256
 18  10.111950 174.36.240.86 -> 128.61.57.205 TCP 66 [TCP Dup ACK 16#1] https > 34574 [ACK] Seq=3257 Ack=228 Win=30080 Len=0 TSval=3311877804 TSecr=3986717
 19  10.111952 128.61.57.205 -> 174.36.240.86 TCP 66 [TCP Dup ACK 15#2] 34574 > https [PSH, ACK] Seq=228 Ack=1 Win=87808 Len=0 TSval=3985712 TSecr=3311875276
 20  10.111955 174.36.240.86 -> 128.61.57.205 TCP 66 [TCP Dup ACK 16#2] https > 34574 [ACK] Seq=3257 Ack=228 Win=30080 Len=0 TSval=3311877804 TSecr=3986717
 21  10.126312 128.61.57.205 -> 174.36.240.86 TCP 86 [TCP Dup ACK 15#3] 34574 > https [PSH, ACK] Seq=228 Ack=1 Win=87808 Len=0 TSval=3985712 TSecr=3311875276 SLE=3257 SRE=3257 SLE=3256 SRE=3256
 22  10.126317 174.36.240.86 -> 128.61.57.205 TCP 66 [TCP Dup ACK 16#3] https > 34574 [ACK] Seq=3257 Ack=228 Win=30080 Len=0 TSval=3311877807 TSecr=3986717
 23  10.126360 128.61.57.205 -> 174.36.240.86 TCP 66 [TCP Dup ACK 15#4] 34574 > https [PSH, ACK] Seq=228 Ack=1 Win=87808 Len=0 TSval=3985712 TSecr=3311875276
 24  10.126362 174.36.240.86 -> 128.61.57.205 TCP 66 [TCP Dup ACK 16#4] https > 34574 [ACK] Seq=3257 Ack=228 Win=30080 Len=0 TSval=3311877807 TSecr=3986717
 25  10.126364 128.61.57.205 -> 174.36.240.86 TCP 66 [TCP Dup ACK 15#5] 34574 > https [PSH, ACK] Seq=228 Ack=1 Win=87808 Len=0 TSval=3985712 TSecr=3311875276
 26  10.126365 174.36.240.86 -> 128.61.57.205 TCP 66 [TCP Dup ACK 16#5] https > 34574 [ACK] Seq=3257 Ack=228 Win=30080 Len=0 TSval=3311877807 TSecr=3986717
 27  10.126366 128.61.57.205 -> 174.36.240.86 TCP 66 [TCP Dup ACK 15#6] 34574 > https [PSH, ACK] Seq=228 Ack=1 Win=87808 Len=0 TSval=3985712 TSecr=3311875276
 28  10.126367 174.36.240.86 -> 128.61.57.205 TCP 66 [TCP Dup ACK 16#6] https > 34574 [ACK] Seq=3257 Ack=228 Win=30080 Len=0 TSval=3311877807 TSecr=3986717
 29  10.140709 128.61.57.205 -> 174.36.240.86 TCP 66 [TCP Dup ACK 15#7] 34574 > https [PSH, ACK] Seq=228 Ack=1 Win=87808 Len=0 TSval=3985712 TSecr=3311875276
 30  10.140714 174.36.240.86 -> 128.61.57.205 TCP 66 [TCP Dup ACK 16#7] https > 34574 [ACK] Seq=3257 Ack=228 Win=30080 Len=0 TSval=3311877811 TSecr=3986717

Neal patches definitely would solve this issue.

Re: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Avery Fay <hidden>
Date: 2015-02-05 23:25:07 

On Wed, Feb 4, 2015 at 4:20 AM, Eric Dumazet [off-list ref] wrote:
Nice, something is really broken on this peer (not a linux one for
sure), or some really buggy middlebox.

The strange [P.] frame with no payload has a wrong sequence number
anyway, so we send back a duplicate ack.
Thanks for this analysis. I've tried to reach out to the gatech NOC to
identify the device, but I haven't been able to get a response from
anyone (except general support, but they don't want to put me in touch
with someone who can help).

Neal patches definitely would solve this issue.

Are these available anywhere? Or could they be made available? This
seems like a much better solution than disabling timestamps.

Avery

Re: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Eric Dumazet <hidden>
Date: 2015-02-06 00:17:54 

On Thu, 2015-02-05 at 15:25 -0800, Avery Fay wrote:

Are these available anywhere? Or could they be made available? This
seems like a much better solution than disabling timestamps.
Neal is currently polishing a serie of 4 patches.

This should be land to netdev shortly ;)

Thanks

Re: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Neal Cardwell <ncardwell@google.com>
Date: 2015-02-06 21:16:26 

On Thu, Feb 5, 2015 at 7:17 PM, Eric Dumazet [off-list ref] wrote:
On Thu, 2015-02-05 at 15:25 -0800, Avery Fay wrote:
quoted
Are these available anywhere? Or could they be made available? This
seems like a much better solution than disabling timestamps.
Neal is currently polishing a serie of 4 patches.

This should be land to netdev shortly ;)
I emailed the patch series to the list. There is a a "[PATCH net-next
0/4] tcp: mitigate TCP ACK loops due to out-of-window validation
dupacks" cover letter with an overview. And you can find the patches
here:

http://patchwork.ozlabs.org/patch/437462/
http://patchwork.ozlabs.org/patch/437463/
http://patchwork.ozlabs.org/patch/437464/
http://patchwork.ozlabs.org/patch/437465/

Please feel free to try these out and let us know how they work for you.

Thanks!

neal

Re: Invalid timestamp? causing tight ack loop (hundreds of thousands of packets / sec)

From: Neal Cardwell <ncardwell@google.com>
Date: 2015-02-09 15:42:25 

On Fri, Feb 6, 2015 at 4:16 PM, Neal Cardwell [off-list ref] wrote:
On Thu, Feb 5, 2015 at 7:17 PM, Eric Dumazet [off-list ref] wrote:
quoted
On Thu, 2015-02-05 at 15:25 -0800, Avery Fay wrote:
quoted
Are these available anywhere? Or could they be made available? This
seems like a much better solution than disabling timestamps.
These are now in the "net-next" tree:

f06535c Merge branch 'tcp_ack_loops'
4fb17a6 tcp: mitigate ACK loops for connections as tcp_timewait_sock
f2b2c58 tcp: mitigate ACK loops for connections as tcp_sock
a9b2c06 tcp: mitigate ACK loops for connections as tcp_request_sock
032ee42 tcp: helpers to mitigate ACK loops by rate-limiting
out-of-window dupacks

neal
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help