Hi Zhigang,

Thanks for help looking into the issue.

My answer to your analysis is of course there won't the third dup-ack, because the server only sends TWO NEW data packets every time.  Clearly this is server's problem and not the client's problem.

Thanks,
Joe

--- On Fri, 9/25/09, zhigang gong <zhigang.gong@gmail.com> wrote:

From: zhigang gong <redacted>
Subject: Re: TCP stack bug related to F-RTO?
To: "Joe Cao" <redacted>
Cc: linux-kernel@vger.kernel.org, jcaoco2002@yahoo.com, netdev@vger.kernel.org
Date: Friday, September 25, 2009, 1:55 AM
Oh, I see, so I spoke too quickly in
last mail. You just ignore some packets
in the trace. I have analysed the traffic flow  and
have some findings as below,
hope it's helpful.

quoted

quoted

quoted

1. The client opens up a big window,
2. the server sends 19 packets in a row (pkt

#14- #32

quoted

quoted

in the trace), but all of them are dropped due to

some

quoted

quoted

congestion.

quoted

3. The server hits RTO and retransmits pkt

#14 in #33
This retransmission timer expiring indicate the server's
tcp/ip
stack to enter slow start mode, as a result we can see the
server's sending window will be reduced to one.

quoted

quoted

quoted

4. The client immediately acks #33 (=#14),

and the

quoted

quoted

server (seems like to enter F-RTO) expends the

window and

quoted

quoted

sends *NEW* pkt #35 & #36.=A0 Timeoute is

doubled to

quoted

quoted

2*RTO; The client immediately sends two Dup-ack to

#35 and

quoted

quoted

#36.

Server is still in slow start mode, and extend window to
2.

quoted

quoted

quoted

5. after 2*RTO, pkt #15 is retransmitted in

#39.

Here , the second retransmission timer expiring ocur.
Server's sending
window reduce to one again and continue in slow start
mode.

quoted

quoted

quoted

6.. The client immediately acks #39 (=#15) in

#40, and

quoted

quoted

the server continues to expand the window and

sends two

quoted

quoted

*NEW* pkt #41 & #42. Now the timeoute is

doubled to 4

quoted

quoted

*RTO.

Here you ignore two duplicate acks #37 and #38 sent by the
client. As I know
the server must receive three or even more duplcate acks
before it enter fast
retransmit mode, otherwise it will still in slow start mode
and  it
will wait until next
time retransmission timer expiring before retransmit the
lost packets.
And this is
actually what you got.

I'm not an kernel expert, I just analyse from the TCP
protocol standard. From my
view, I think there is no problem in the server's network
stack. But
there maybe
some problem in the client (or some intermediate network
appliance) side, as it
always just sends two duplicate acks at the same time, and
never send the third
one no matter how long the interval is. In my opinion, if
the client
can send the third
duplicate acks then the server will enter fast retransmit
mode and
then fast recovery
then every thing will be ok.

quoted

quoted

quoted

8. After 4*RTO timeout, #16 is

retransmitted.

quoted

quoted

quoted

9....
10. The above steps repeats for

retransmitting pkt

quoted

quoted

#16-#32 and each time the timeout is doubled.

quoted

11. It takes a long long time to retransmit

all the

quoted

quoted

lost packets and before that is done, the client

sends a RST

quoted

quoted

because of timeout.

On Fri, Sep 25, 2009 at 2:42 PM, Joe Cao [off-list ref]
wrote:

quoted

Hi,

On the wrong tcp checksum, that's because of hardware

checksum offload.

quoted

As for the seq/ack number, because the trace is long,

I deliberately removed those irrelevant packets between
after the three-way handshake and when the problem happens.
 That can be seen from the timestamps.

quoted

Please also note that I intentionally replaced the IP

addresses and mac addresses in the trace to hide proprietary
information in the trace.

quoted

Anyway, the problem is not related to the checksum, or

seq/ack number, otherwise, you won't see the behavior shown
in the trace.

quoted

Thanks,
Joe

--- On Thu, 9/24/09, zhigang gong <zhigang.gong@gmail.com>

wrote:

quoted