Hi,
it's not easy to change the initial RTO of TCP as right now you need to
recompile your kernel. In order to make it easier to tune this setting,
I was wondering whether you would consider turning it into a sysctl. I
attached a first attempt at a patch that does this -- this is my first
patch to the Linux kernel so although I've read SubmitChecklist and
SubmittingPatches, and I've run checkpatch.pl, please let me know if I'm
doing something wrong.
I am doing this because I work in a high-throughput low-latency environment
(line-rate GbE with submillisecond RTT) and some of our clients are negatively
affected by the high initial RTO when the servers are unable to accept() new
connections fast enough. While we're working on fixing these servers and/or
giving them larger backlog queues when they listen(), being able to tune
the initial RTO at runtime would be very useful as quick workaround for the
server-side issues.
Some large Internet websites are also running with a more aggressive initial
RTO, for instance Google documented some of what they're doing here:
http://www.ietf.org/proceedings/75/slides/tcpm-1.pdf
While I'm not arguing to change the default value at this time, I believe
that this patch would also come in handy for those who wish to experiment
with various values in their environment.
If you're willing to consider this patch, bear in mind I only compiled it,
I didn't test it yet (not knowing whether you'd want something like that or
not). I would also appreciate if anyone had any insight on what I did with
`COUNTER_TRIES' in `syncookies.c' as this magic constant is rather mysterious
and the comment didn't help me figure out how it had been derived. I couldn't
find anything online and git blame didn't help me either (it pre-dates Git).
@@ -384,6 +384,12 @@ tcp_retries2 - INTEGER RFC 1122 recommends at least 100 seconds for the timeout, which corresponds to a value of at least 8.+tcp_initial_rto - INTEGER+ This value sets the initial retransmit timeout, that is how long+ the kernel will wait before retransmitting the initial SYN packet.++ RFC 1122 says that this SHOULD be 3 seconds, which is the default.+ tcp_rfc1337 - BOOLEAN If set, the TCP stack behaves conforming to RFC1337. If unset, we are not conforming to RFC, but prevent TCP TIME_WAIT
@@ -30,6 +30,8 @@ static int tcp_adv_win_scale_min = -31;staticinttcp_adv_win_scale_max=31;staticintip_ttl_min=1;staticintip_ttl_max=255;+staticinttcp_initial_rto_min=TCP_RTO_MIN;+staticinttcp_initial_rto_max=TCP_RTO_MAX;/* Update system visible IP port range */staticvoidset_local_port_range(intrange[2])
@@ -2352,7 +2352,7 @@ static int do_tcp_setsockopt(struct sock *sk, int level,caseTCP_DEFER_ACCEPT:/* Translate value in seconds to number of retransmits */icsk->icsk_accept_queue.rskq_defer_accept=-secs_to_retrans(val,TCP_TIMEOUT_INIT/HZ,+secs_to_retrans(val,sysctl_tcp_initial_rto/HZ,TCP_RTO_MAX/HZ);break;
@@ -2539,7 +2539,7 @@ static int do_tcp_getsockopt(struct sock *sk, int level,break;caseTCP_DEFER_ACCEPT:val=retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept,-TCP_TIMEOUT_INIT/HZ,TCP_RTO_MAX/HZ);+sysctl_tcp_initial_rto/HZ,TCP_RTO_MAX/HZ);break;caseTCP_WINDOW_CLAMP:val=tp->window_clamp;
@@ -890,7 +890,7 @@ static void tcp_init_metrics(struct sock *sk)if(dst_metric(dst,RTAX_RTT)==0)gotoreset;-if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(TCP_TIMEOUT_INIT<<3))+if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(sysctl_tcp_initial_rto<<3))gotoreset;/* Initial rtt is determined from SYN,SYN-ACK.
@@ -916,7 +916,7 @@ static void tcp_init_metrics(struct sock *sk)tp->mdev_max=tp->rttvar=max(tp->mdev,tcp_rto_min(sk));}tcp_set_rto(sk);-if(inet_csk(sk)->icsk_rto<TCP_TIMEOUT_INIT&&!tp->rx_opt.saw_tstamp){+if(inet_csk(sk)->icsk_rto<sysctl_tcp_initial_rto&&!tp->rx_opt.saw_tstamp){reset:/* Play conservative. If timestamps are not*supported,TCPwillfailtorecalculatecorrect
@@ -1834,8 +1834,8 @@ static int tcp_v4_init_sock(struct sock *sk)tcp_init_xmit_timers(sk);tcp_prequeue_init(tp);-icsk->icsk_rto=TCP_TIMEOUT_INIT;-tp->mdev=TCP_TIMEOUT_INIT;+icsk->icsk_rto=sysctl_tcp_initial_rto;+tp->mdev=sysctl_tcp_initial_rto;/* So many TCP implementations out there (incorrectly) count the*initialSYNframeintheirdelayed-ACKandcongestioncontrol
@@ -29,6 +29,7 @@ int sysctl_tcp_keepalive_probes __read_mostly = TCP_KEEPALIVE_PROBES;intsysctl_tcp_keepalive_intvl__read_mostly=TCP_KEEPALIVE_INTVL;intsysctl_tcp_retries1__read_mostly=TCP_RETR1;intsysctl_tcp_retries2__read_mostly=TCP_RETR2;+intsysctl_tcp_initial_rto__read_mostly=TCP_TIMEOUT_INIT;intsysctl_tcp_orphan_retries__read_mostly;intsysctl_tcp_thin_linear_timeouts__read_mostly;
@@ -135,8 +136,8 @@ static void tcp_mtu_probing(struct inet_connection_sock *icsk, struct sock *sk)/* This function calculates a "timeout" which is equivalent to the timeout of a*TCPconnectionafter"boundary"unsuccessful,exponentiallybacked-off-*retransmissionswithaninitialRTOofTCP_RTO_MINorTCP_TIMEOUT_INITif-*syn_setflagisset.+*retransmissionswithaninitialRTOofTCP_RTO_MINor+*sysctl_tcp_initial_rtoifsyn_setflagisset.*/staticboolretransmits_timed_out(structsock*sk,unsignedintboundary,
@@ -1957,8 +1957,8 @@ static int tcp_v6_init_sock(struct sock *sk)tcp_init_xmit_timers(sk);tcp_prequeue_init(tp);-icsk->icsk_rto=TCP_TIMEOUT_INIT;-tp->mdev=TCP_TIMEOUT_INIT;+icsk->icsk_rto=sysctl_tcp_initial_rto;+tp->mdev=sysctl_tcp_initial_rto;/* So many TCP implementations out there (incorrectly) count the*initialSYNframeintheirdelayed-ACKandcongestioncontrol
From: Eric Dumazet <hidden> Date: 2011-05-17 08:08:07
Le mardi 17 mai 2011 à 00:40 -0700, Benoit Sigoure a écrit :
Instead of hardcoding the initial RTO to 3s and requiring
the kernel to be recompiled to change it, expose it as a
sysctl that can be tuned at runtime. Leave the default
value unchanged.
I wont discuss if introducing a new sysctl is welcomed, only on patch
issues. I believe some work in IETF is done to reduce the 3sec value to
1sec anyway.
@@ -384,6 +384,12 @@ tcp_retries2 - INTEGER RFC 1122 recommends at least 100 seconds for the timeout, which corresponds to a value of at least 8.+tcp_initial_rto - INTEGER+ This value sets the initial retransmit timeout, that is how long+ the kernel will wait before retransmitting the initial SYN packet.++ RFC 1122 says that this SHOULD be 3 seconds, which is the default.+
units ? seconds ? ms ? jiffies ? I suggest using ms as external
interface.
quoted hunk
tcp_rfc1337 - BOOLEAN
If set, the TCP stack behaves conforming to RFC1337. If unset,
we are not conforming to RFC, but prevent TCP TIME_WAIT
@@ -2352,7 +2352,7 @@ static int do_tcp_setsockopt(struct sock *sk, int level,caseTCP_DEFER_ACCEPT:/* Translate value in seconds to number of retransmits */icsk->icsk_accept_queue.rskq_defer_accept=-secs_to_retrans(val,TCP_TIMEOUT_INIT/HZ,+secs_to_retrans(val,sysctl_tcp_initial_rto/HZ,
Here you assume sysctl_tcp_initial_rto is expressed in jiffies ?
Oh well...
quoted hunk
TCP_RTO_MAX / HZ);
break;
@@ -2539,7 +2539,7 @@ static int do_tcp_getsockopt(struct sock *sk, int level, break; case TCP_DEFER_ACCEPT: val = retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept,- TCP_TIMEOUT_INIT / HZ, TCP_RTO_MAX / HZ);+ sysctl_tcp_initial_rto / HZ, TCP_RTO_MAX / HZ); break; case TCP_WINDOW_CLAMP: val = tp->window_clamp;
From: Alexander Zimmermann <hidden> Date: 2011-05-17 08:31:38
Hi Benoit,
Am 17.05.2011 um 09:40 schrieb Benoit Sigoure:
Instead of hardcoding the initial RTO to 3s and requiring
the kernel to be recompiled to change it, expose it as a
sysctl that can be tuned at runtime. Leave the default
value unchanged.
regardless of netdev will accept this patch or not, the
upcoming initRTO is 1s. See
http://tools.ietf.org/id/draft-paxson-tcpm-rfc2988bis-02.txt
The draft is IESG approved and will become an RFC soon.
Alex
//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22222
// email: zimmermann@cs.rwth-aachen.de
// web: http://www.umic-mesh.net
//
From: Hagen Paul Pfeifer <hidden> Date: 2011-05-17 11:02:49
On Tue, 17 May 2011 10:07:57 +0200, Eric Dumazet wrote:
I wont discuss if introducing a new sysctl is welcomed, only on patch
issues. I believe some work in IETF is done to reduce the 3sec value to
1sec anyway.
Why not? I though all new knobs in this area should be done on a per route
metric so it can be controlled on a per path basis. RTO should be
adjustable on a per path basis, because it depends on the path.
Some months back [1] I posted a patch to enable/disable TCP quick ack
mode, which has nothing to do with network paths, just with a local server
policy. But David rejected the patch with the argument that I should use a
per path knob (this is a little bit inapprehensible for me, but David has
the last word).
Hagen
[1] http://kerneltrap.org/mailarchive/linux-netdev/2010/8/23/6283640
From: Eric Dumazet <hidden> Date: 2011-05-17 12:20:31
Le mardi 17 mai 2011 à 13:02 +0200, Hagen Paul Pfeifer a écrit :
On Tue, 17 May 2011 10:07:57 +0200, Eric Dumazet wrote:
quoted
I wont discuss if introducing a new sysctl is welcomed, only on patch
issues. I believe some work in IETF is done to reduce the 3sec value to
1sec anyway.
Why not?
Just because I let this point to David and others. I personally dont
care that much.
I though all new knobs in this area should be done on a per route
metric so it can be controlled on a per path basis. RTO should be
adjustable on a per path basis, because it depends on the path.
Adding many knobs to each clone had a huge cost on previous kernels.
(Think some machines have millions entries in IP route cache), this used
quite a lot of memory.
With latest David work, we'll consume less ram, because we can now share
settings, instead of copying them on each dst entry.
Some months back [1] I posted a patch to enable/disable TCP quick ack
mode, which has nothing to do with network paths, just with a local server
policy. But David rejected the patch with the argument that I should use a
per path knob (this is a little bit inapprehensible for me, but David has
the last word).
Instead of hardcoding the initial RTO to 3s and requiring
the kernel to be recompiled to change it, expose it as a
sysctl that can be tuned at runtime. Leave the default
value unchanged.
Signed-off-by: Benoit Sigoure <redacted>
---
v2 of the patch to address Eric's comments. Of course I had to forget
to convert things back and forth between jiffies and ms -- /me n00b.
Code compiles. It seems like no one is opposed to this change, but if
one of you guys could express explicit interest in merging this change,
I'd be happy to spend a bit more time to test it.
The new sysctl is exposed in milliseconds but internally the value remains
in jiffies to avoid having to convert back / and forth between jiffies and
ms in most places.
I'm glad to hear that the default value will be tuned down to 1s. This
change will help people play with this value and easily revert it back at
runtime if they feel like they preferred the current value.
Thank you for your time.
Documentation/networking/ip-sysctl.txt | 8 ++++++++
include/net/tcp.h | 3 ++-
net/ipv4/syncookies.c | 2 +-
net/ipv4/sysctl_net_ipv4.c | 11 +++++++++++
net/ipv4/tcp.c | 4 ++--
net/ipv4/tcp_input.c | 8 ++++----
net/ipv4/tcp_ipv4.c | 6 +++---
net/ipv4/tcp_minisocks.c | 6 +++---
net/ipv4/tcp_output.c | 2 +-
net/ipv4/tcp_timer.c | 9 +++++----
net/ipv6/syncookies.c | 2 +-
net/ipv6/tcp_ipv6.c | 6 +++---
12 files changed, 44 insertions(+), 23 deletions(-)
@@ -384,6 +384,14 @@ tcp_retries2 - INTEGER RFC 1122 recommends at least 100 seconds for the timeout, which corresponds to a value of at least 8.+tcp_initial_rto - INTEGER+ This value sets the initial retransmit timeout (in milliseconds),+ that is how long the kernel will wait before retransmitting the+ initial SYN packet.++ RFC 1122 says that this SHOULD be 3000 milliseconds, which is the+ default.+ tcp_rfc1337 - BOOLEAN If set, the TCP stack behaves conforming to RFC1337. If unset, we are not conforming to RFC, but prevent TCP TIME_WAIT
@@ -213,6 +213,7 @@ extern int sysctl_tcp_syn_retries;externintsysctl_tcp_synack_retries;externintsysctl_tcp_retries1;externintsysctl_tcp_retries2;+externintsysctl_tcp_initial_rto;/* in jiffies */externintsysctl_tcp_orphan_retries;externintsysctl_tcp_syncookies;externintsysctl_tcp_retrans_collapse;
@@ -30,6 +30,8 @@ static int tcp_adv_win_scale_min = -31;staticinttcp_adv_win_scale_max=31;staticintip_ttl_min=1;staticintip_ttl_max=255;+staticinttcp_initial_rto_min=TCP_RTO_MIN;+staticinttcp_initial_rto_max=TCP_RTO_MAX;/* Update system visible IP port range */staticvoidset_local_port_range(intrange[2])
@@ -2352,7 +2352,7 @@ static int do_tcp_setsockopt(struct sock *sk, int level,caseTCP_DEFER_ACCEPT:/* Translate value in seconds to number of retransmits */icsk->icsk_accept_queue.rskq_defer_accept=-secs_to_retrans(val,TCP_TIMEOUT_INIT/HZ,+secs_to_retrans(val,sysctl_tcp_initial_rto/HZ,TCP_RTO_MAX/HZ);break;
@@ -2539,7 +2539,7 @@ static int do_tcp_getsockopt(struct sock *sk, int level,break;caseTCP_DEFER_ACCEPT:val=retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept,-TCP_TIMEOUT_INIT/HZ,TCP_RTO_MAX/HZ);+sysctl_tcp_initial_rto/HZ,TCP_RTO_MAX/HZ);break;caseTCP_WINDOW_CLAMP:val=tp->window_clamp;
@@ -890,7 +890,7 @@ static void tcp_init_metrics(struct sock *sk)if(dst_metric(dst,RTAX_RTT)==0)gotoreset;-if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(TCP_TIMEOUT_INIT<<3))+if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(sysctl_tcp_initial_rto<<3))gotoreset;/* Initial rtt is determined from SYN,SYN-ACK.
@@ -916,7 +916,7 @@ static void tcp_init_metrics(struct sock *sk)tp->mdev_max=tp->rttvar=max(tp->mdev,tcp_rto_min(sk));}tcp_set_rto(sk);-if(inet_csk(sk)->icsk_rto<TCP_TIMEOUT_INIT&&!tp->rx_opt.saw_tstamp){+if(inet_csk(sk)->icsk_rto<sysctl_tcp_initial_rto&&!tp->rx_opt.saw_tstamp){reset:/* Play conservative. If timestamps are not*supported,TCPwillfailtorecalculatecorrect
@@ -1834,8 +1834,8 @@ static int tcp_v4_init_sock(struct sock *sk)tcp_init_xmit_timers(sk);tcp_prequeue_init(tp);-icsk->icsk_rto=TCP_TIMEOUT_INIT;-tp->mdev=TCP_TIMEOUT_INIT;+icsk->icsk_rto=sysctl_tcp_initial_rto;+tp->mdev=sysctl_tcp_initial_rto;/* So many TCP implementations out there (incorrectly) count the*initialSYNframeintheirdelayed-ACKandcongestioncontrol
@@ -29,6 +29,7 @@ int sysctl_tcp_keepalive_probes __read_mostly = TCP_KEEPALIVE_PROBES;intsysctl_tcp_keepalive_intvl__read_mostly=TCP_KEEPALIVE_INTVL;intsysctl_tcp_retries1__read_mostly=TCP_RETR1;intsysctl_tcp_retries2__read_mostly=TCP_RETR2;+intsysctl_tcp_initial_rto__read_mostly=TCP_TIMEOUT_INIT;intsysctl_tcp_orphan_retries__read_mostly;intsysctl_tcp_thin_linear_timeouts__read_mostly;
@@ -135,8 +136,8 @@ static void tcp_mtu_probing(struct inet_connection_sock *icsk, struct sock *sk)/* This function calculates a "timeout" which is equivalent to the timeout of a*TCPconnectionafter"boundary"unsuccessful,exponentiallybacked-off-*retransmissionswithaninitialRTOofTCP_RTO_MINorTCP_TIMEOUT_INITif-*syn_setflagisset.+*retransmissionswithaninitialRTOofTCP_RTO_MINor+*sysctl_tcp_initial_rtoifsyn_setflagisset.*/staticboolretransmits_timed_out(structsock*sk,unsignedintboundary,
@@ -1957,8 +1957,8 @@ static int tcp_v6_init_sock(struct sock *sk)tcp_init_xmit_timers(sk);tcp_prequeue_init(tp);-icsk->icsk_rto=TCP_TIMEOUT_INIT;-tp->mdev=TCP_TIMEOUT_INIT;+icsk->icsk_rto=sysctl_tcp_initial_rto;+tp->mdev=sysctl_tcp_initial_rto;/* So many TCP implementations out there (incorrectly) count the*initialSYNframeintheirdelayed-ACKandcongestioncontrol
Instead of hardcoding the initial RTO to 3s and requiring
the kernel to be recompiled to change it, expose it as a
sysctl that can be tuned at runtime. Leave the default
value unchanged.
Signed-off-by: Benoit Sigoure <redacted>
If you read the ietf draft that reduces the initial RTO down to 1
second, it states that if we take a timeout during the initial
connection handshake then we have to revert the RTO back up to 3
seconds.
This fallback logic conflicts with being able to only change the
initial RTO via sysctl, I think. Because there are actually two
values at stake and they depend upon eachother, the initial RTO and
the value we fallback to on initial handshake retransmissions.
So I'd rather get a patch that implements the 1 second initial
RTO with the 3 second fallback on SYN retransmit, than this patch.
We already have too many knobs.
On Wed, May 18, 2011 at 12:26 PM, David Miller [off-list ref] wrote:
If you read the ietf draft that reduces the initial RTO down to 1
second, it states that if we take a timeout during the initial
connection handshake then we have to revert the RTO back up to 3
seconds.
This fallback logic conflicts with being able to only change the
initial RTO via sysctl, I think. Because there are actually two
values at stake and they depend upon eachother, the initial RTO and
the value we fallback to on initial handshake retransmissions.
So I'd rather get a patch that implements the 1 second initial
RTO with the 3 second fallback on SYN retransmit, than this patch.
We already have too many knobs.
I was hoping this knob would be accepted because this is such an
important issue that it even warrants an IETF draft to attempt to
change the standard. I'm not sure how long it will take for this
draft to be accepted and then implemented, so I thought adding this
simple knob today would really help in the future.
Plus, should the draft be accepted, this knob will still be just as
useful (e.g. to revert back to today's behavior), and people might
want to consider adding another knob for the fallback initRTO (this is
debatable). I don't believe this knob conflicts with the proposed
change to the standard, it actually goes along with it pretty well and
helps us prepare better for this upcoming change.
I agree that there are too many knobs, and I hate feature creep too,
but I've found many of these knobs to be really useful, and the degree
to which Linux's TCP stack can be tuned is part of what makes it so
versatile.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
From: David Miller <davem@davemloft.net> Date: 2011-05-18 19:55:51
From: tsuna <redacted>
Date: Wed, 18 May 2011 12:40:21 -0700
I was hoping this knob would be accepted because this is such an
important issue that it even warrants an IETF draft to attempt to
change the standard. I'm not sure how long it will take for this
draft to be accepted and then implemented, so I thought adding this
simple knob today would really help in the future.
I've already changed the initial TCP congestion window in Linux to 10
without some stupid draft being fully accepted.
I'll just as easily accept right now a patch right now which lowers
the initial RTO to 1 second and adds the 3 second RTO fallback.
From: Hagen Paul Pfeifer <hidden> Date: 2011-05-18 20:20:31
* David Miller | 2011-05-18 15:52:00 [-0400]:
I've already changed the initial TCP congestion window in Linux to 10
without some stupid draft being fully accepted.
I'll just as easily accept right now a patch right now which lowers
the initial RTO to 1 second and adds the 3 second RTO fallback.
I like the idea to make the initial RTO a knob because we in a isolated MANET
environment have a RTT larger then 1 second. Especially the link layer setup
procedure over several hops demand some time-costly setup time. After that the
RTT is <1 second. The current algorithm works great for us. So this RTO change
will be counterproductive: it will always trigger a needless timeout.
The main problem for us is that Google at all pushing their view of Internet
with a lot of pressure. The same is true for the IETF IW adjustments, which is
unsuitable for networks which operates at a bandwidth characteristic some
years ago. The _former_ conservative principle "TCP over everything" is
forgotten.
Hagen
From: Hagen Paul Pfeifer <hidden> Date: 2011-05-18 20:27:18
* David Miller | 2011-05-18 16:23:06 [-0400]:
Then this gets back to the fact that this is a network
attribute and thus more suitable as a route metric not
a global system-wide sysctl.
Yes, in an Email response to Eric I mentioned this already. The initial RTO is
a perfect candidate for route metric. I waiting for a patch to test it! ;-)
Hagen
Prior to this patch, Linux would always use 3 seconds (compile-time
constant) as the initial RTO. Draft RFC 2988bis-02 proposes to tune
this down to 1 second and, in case of a timeout during the TCP 3WHS,
revert the RTO back up to 3 seconds when data transmission begins.
This patch implements this behavior but retains default values for
the initial RTO of 3 seconds, instead of 1 second as is suggested
in the draft RFC. This way, in a default configuration, the behavior
of Linux's TCP is unchanged.
This patch also adds 2 knobs to tweak the initial RTO:
- tcp_initial_rto: initial RTO used during the 3WHS (default remains
unchanged: 3 seconds). This was previously a compile-time constant.
- tcp_initial_fallback_rto: the RTO to fallback to if a timeout occurs
during the 3WHS, with a default value of 3 seconds too, as per the
draft RFC.
Signed-off-by: Benoit Sigoure <redacted>
---
On Wed, May 18, 2011 at 12:52 PM, David Miller [off-list ref] wrote:
I'll just as easily accept right now a patch right now which lowers
the initial RTO to 1 second and adds the 3 second RTO fallback.
Here's a first attempt at a patch that implements the behavior described in
the draft RFC. I only compiled it so far, if you would like to move forward
with this approach, I'll go ahead and test it on a real server.
I'm not sure whether COUNTER_TRIES in syncookies.c should be based off
sysctl_tcp_initial_rto or sysctl_tcp_initial_fallback_rto, if we're going
to take the first one down to 1s...
Documentation/networking/ip-sysctl.txt | 19 +++++++++++++++++++
include/net/tcp.h | 4 +++-
net/ipv4/syncookies.c | 2 +-
net/ipv4/sysctl_net_ipv4.c | 20 ++++++++++++++++++++
net/ipv4/tcp.c | 4 ++--
net/ipv4/tcp_input.c | 13 +++++++++----
net/ipv4/tcp_ipv4.c | 6 +++---
net/ipv4/tcp_minisocks.c | 6 +++---
net/ipv4/tcp_output.c | 2 +-
net/ipv4/tcp_timer.c | 10 ++++++----
net/ipv6/syncookies.c | 2 +-
net/ipv6/tcp_ipv6.c | 6 +++---
12 files changed, 71 insertions(+), 23 deletions(-)
@@ -384,6 +384,25 @@ tcp_retries2 - INTEGER RFC 1122 recommends at least 100 seconds for the timeout, which corresponds to a value of at least 8.+tcp_initial_rto - INTEGER+ This value sets the initial retransmit timeout (in milliseconds),+ that is how long the kernel will wait before retransmitting the+ initial SYN packet.++ RFC 1122 says that this SHOULD be 3000 milliseconds, which is the+ default. Note that draft RFC 2988bis-02 says that this SHOULD be+ 1000 milliseconds, which might become the default value in future+ versions.++tcp_initial_fallback_rto - INTEGER+ This value sets the initial retransmit timeout (in milliseconds)+ to use after completing a three-way handshake during which the+ initial SYN packet had to be retransmitted after waiting for+ tcp_initial_rto milliseconds.++ Draft RFC 2988bis-02 says that this MUST be 3000 milliseconds,+ which is the default.+ tcp_rfc1337 - BOOLEAN If set, the TCP stack behaves conforming to RFC1337. If unset, we are not conforming to RFC, but prevent TCP TIME_WAIT
@@ -213,6 +213,8 @@ extern int sysctl_tcp_syn_retries;externintsysctl_tcp_synack_retries;externintsysctl_tcp_retries1;externintsysctl_tcp_retries2;+externintsysctl_tcp_initial_rto;/* in jiffies */+externintsysctl_tcp_initial_fallback_rto;/* in jiffies */externintsysctl_tcp_orphan_retries;externintsysctl_tcp_syncookies;externintsysctl_tcp_retrans_collapse;
@@ -30,6 +30,8 @@ static int tcp_adv_win_scale_min = -31;staticinttcp_adv_win_scale_max=31;staticintip_ttl_min=1;staticintip_ttl_max=255;+staticinttcp_min_rto=TCP_RTO_MIN;+staticinttcp_max_rto=TCP_RTO_MAX;/* Update system visible IP port range */staticvoidset_local_port_range(intrange[2])
@@ -2352,7 +2352,7 @@ static int do_tcp_setsockopt(struct sock *sk, int level,caseTCP_DEFER_ACCEPT:/* Translate value in seconds to number of retransmits */icsk->icsk_accept_queue.rskq_defer_accept=-secs_to_retrans(val,TCP_TIMEOUT_INIT/HZ,+secs_to_retrans(val,sysctl_tcp_initial_rto/HZ,TCP_RTO_MAX/HZ);break;
@@ -2539,7 +2539,7 @@ static int do_tcp_getsockopt(struct sock *sk, int level,break;caseTCP_DEFER_ACCEPT:val=retrans_to_secs(icsk->icsk_accept_queue.rskq_defer_accept,-TCP_TIMEOUT_INIT/HZ,TCP_RTO_MAX/HZ);+sysctl_tcp_initial_rto/HZ,TCP_RTO_MAX/HZ);break;caseTCP_WINDOW_CLAMP:val=tp->window_clamp;
@@ -868,6 +868,11 @@ static void tcp_init_metrics(struct sock *sk){structtcp_sock*tp=tcp_sk(sk);structdst_entry*dst=__sk_dst_get(sk);+/* If we had to retransmit anything during the 3WHS,+*usetheinitialfallbackRTO.+*/+intinit_rto=inet_csk(sk)->icsk_retransmits?+sysctl_tcp_initial_fallback_rto:sysctl_tcp_initial_rto;if(dst==NULL)gotoreset;
@@ -890,7 +895,7 @@ static void tcp_init_metrics(struct sock *sk)if(dst_metric(dst,RTAX_RTT)==0)gotoreset;-if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(TCP_TIMEOUT_INIT<<3))+if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(init_rto<<3))gotoreset;/* Initial rtt is determined from SYN,SYN-ACK.
@@ -916,7 +921,7 @@ static void tcp_init_metrics(struct sock *sk)tp->mdev_max=tp->rttvar=max(tp->mdev,tcp_rto_min(sk));}tcp_set_rto(sk);-if(inet_csk(sk)->icsk_rto<TCP_TIMEOUT_INIT&&!tp->rx_opt.saw_tstamp){+if(inet_csk(sk)->icsk_rto<init_rto&&!tp->rx_opt.saw_tstamp){reset:/* Play conservative. If timestamps are not*supported,TCPwillfailtorecalculatecorrect
@@ -1834,8 +1834,8 @@ static int tcp_v4_init_sock(struct sock *sk)tcp_init_xmit_timers(sk);tcp_prequeue_init(tp);-icsk->icsk_rto=TCP_TIMEOUT_INIT;-tp->mdev=TCP_TIMEOUT_INIT;+icsk->icsk_rto=sysctl_tcp_initial_rto;+tp->mdev=sysctl_tcp_initial_rto;/* So many TCP implementations out there (incorrectly) count the*initialSYNframeintheirdelayed-ACKandcongestioncontrol
@@ -29,6 +29,8 @@ int sysctl_tcp_keepalive_probes __read_mostly = TCP_KEEPALIVE_PROBES;intsysctl_tcp_keepalive_intvl__read_mostly=TCP_KEEPALIVE_INTVL;intsysctl_tcp_retries1__read_mostly=TCP_RETR1;intsysctl_tcp_retries2__read_mostly=TCP_RETR2;+intsysctl_tcp_initial_rto__read_mostly=TCP_TIMEOUT_INIT;+intsysctl_tcp_initial_fallback_rto__read_mostly=TCP_TIMEOUT_INIT;intsysctl_tcp_orphan_retries__read_mostly;intsysctl_tcp_thin_linear_timeouts__read_mostly;
@@ -135,8 +137,8 @@ static void tcp_mtu_probing(struct inet_connection_sock *icsk, struct sock *sk)/* This function calculates a "timeout" which is equivalent to the timeout of a*TCPconnectionafter"boundary"unsuccessful,exponentiallybacked-off-*retransmissionswithaninitialRTOofTCP_RTO_MINorTCP_TIMEOUT_INITif-*syn_setflagisset.+*retransmissionswithaninitialRTOofTCP_RTO_MINor+*sysctl_tcp_initial_rtoifsyn_setflagisset.*/staticboolretransmits_timed_out(structsock*sk,unsignedintboundary,
@@ -1957,8 +1957,8 @@ static int tcp_v6_init_sock(struct sock *sk)tcp_init_xmit_timers(sk);tcp_prequeue_init(tp);-icsk->icsk_rto=TCP_TIMEOUT_INIT;-tp->mdev=TCP_TIMEOUT_INIT;+icsk->icsk_rto=sysctl_tcp_initial_rto;+tp->mdev=sysctl_tcp_initial_rto;/* So many TCP implementations out there (incorrectly) count the*initialSYNframeintheirdelayed-ACKandcongestioncontrol
Prior to this patch, Linux would always use 3 seconds (compile-time
constant) as the initial RTO. Draft RFC 2988bis-02 proposes to tune
this down to 1 second and, in case of a timeout during the TCP 3WHS,
revert the RTO back up to 3 seconds when data transmission begins.
We just had a discussion where it was determined that changes to
these settings are "network specific" and therefore that if it
is appropriate at all (I'm still not convinced) it is only suitable
as a routing metric.
Prior to this patch, Linux would always use 3 seconds (compile-time
constant) as the initial RTO. Draft RFC 2988bis-02 proposes to tune
this down to 1 second and, in case of a timeout during the TCP 3WHS,
revert the RTO back up to 3 seconds when data transmission begins.
We just had a discussion where it was determined that changes to
these settings are "network specific" and therefore that if it
is appropriate at all (I'm still not convinced) it is only suitable
as a routing metric.
Fair enough. I'll take another stab at it and see if I can change
this to be on a per network basis. Do I need any patch that's not yet
in Linus' tree? I'm referring to this:
On Tue, May 17, 2011 at 5:20 AM, Eric Dumazet [off-list ref] wrote:
Adding many knobs to each clone had a huge cost on previous kernels.
(Think some machines have millions entries in IP route cache), this used
quite a lot of memory.
With latest David work, we'll consume less ram, because we can now share
settings, instead of copying them on each dst entry.
If this has already been merged then it sounds like I should have
everything I need..?
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
Prior to this patch, Linux would always use 3 seconds (compile-time
constant) as the initial RTO. Draft RFC 2988bis-02 proposes to tune
this down to 1 second and, in case of a timeout during the TCP 3WHS,
revert the RTO back up to 3 seconds when data transmission begins.
We just had a discussion where it was determined that changes to
these settings are "network specific" and therefore that if it
is appropriate at all (I'm still not convinced) it is only suitable
as a routing metric.
Fair enough. I'll take another stab at it and see if I can change
this to be on a per network basis. Do I need any patch that's not yet
in Linus' tree? I'm referring to this:
Keep in mind another thing I do not like about this knob.
The IETF draft has a requirement that we fallback to 3 seconds if the
initial RTO is 1 second.
Nothing in your facilities ensure this, or provide a way for the
kernel to make sure this is the case.
And for other values of initial RTO, what fallback is appropriate?
As a result of all of this, I do not really think this is something
the user should control at all.
I really would rather see the initial RTO be static and be set to 1
with fallback RTO of 3.
On Wed, May 18, 2011 at 9:14 PM, David Miller [off-list ref] wrote:
The IETF draft has a requirement that we fallback to 3 seconds if the
initial RTO is 1 second.
Nothing in your facilities ensure this, or provide a way for the
kernel to make sure this is the case.
Not sure to understand what you're saying. If tcp_initial_rto = 1000
and tcp_initial_fallback_rto = 3000, then you get exactly the behavior
the draft describes. The knobs simply allow you to either revert to
today's behavior or use other settings that would make more sense in
your environment (e.g. very high RTT). Are you concerned about cases
where, say, tcp_initial_fallback_rto < tcp_initial_rto?
And for other values of initial RTO, what fallback is appropriate?
Presumably if the user decides to tweak these knobs, they'll know
what's appropriate for their environment. Or are you suggesting that
one value be derived from the other? (e.g. tcp_initial_fallback_rto =
3 * tcp_initial_rto)
As a result of all of this, I do not really think this is something
the user should control at all.
I really would rather see the initial RTO be static and be set to 1
with fallback RTO of 3.
I can also provide a simple patch for this if you want to start from
there. And then maybe we can discuss having a runtime knob some more
:-)
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
From: Alexander Zimmermann <hidden> Date: 2011-05-19 06:10:46
Hi,
Am 19.05.2011 um 06:33 schrieb tsuna:
Presumably if the user decides to tweak these knobs, they'll know
what's appropriate for their environment.
Are you sure? I'm not. I fully agree with David that minRTO is
something that a user shout not control at all
Or are you suggesting that
one value be derived from the other? (e.g. tcp_initial_fallback_rto =
3 * tcp_initial_rto)
quoted
As a result of all of this, I do not really think this is something
the user should control at all.
I really would rather see the initial RTO be static and be set to 1
with fallback RTO of 3.
I can also provide a simple patch for this if you want to start from
there. And then maybe we can discuss having a runtime knob some more
:-)
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22222
// email: zimmermann@cs.rwth-aachen.de
// web: http://www.umic-mesh.net
//
On Wed, May 18, 2011 at 11:10 PM, Alexander Zimmermann
[off-list ref] wrote:
Am 19.05.2011 um 06:33 schrieb tsuna:
quoted
Presumably if the user decides to tweak these knobs, they'll know
what's appropriate for their environment.
Are you sure? I'm not. I fully agree with David that minRTO is
s/minRTO/initRTO/, right?
something that a user shout not control at all
I personally don't like to hold the hand and spoon feed users too
much, I want to trust them to be responsible and know what they're
doing. Yes, there will always be people who will act stupid and do
stupid things with whatever knobs you expose. The web is full of
people who advise to tune up all the TCP rmem/wmem parameters to crazy
high level based on the voodoo belief that they're going to improve
their TCP performance, but then as long as you have knobs in your
system, these people will misuse them anyway and shoot themselves in
the foot, what can we do about that.
There's also a good chunk of people who know what they're doing, and
for them compile-time constants are annoying because it's inconvenient
to experiment and iterate quickly when you need to recompile your
kernel to change a value. If turning the compile time constant into a
knob leaves the code reasonably straightforward and doesn't incur too
much overhead, then why not do it?
Regarding this knob in particular, I can imagine that people who are
in environment where RTT easily gets around 1s will be upset by the
change in the default value, and doubly upset that they have to
recompile their kernel to change the value back to 3s. I'm in favor
of the reduction of initRTO, for the same reason Google is, but I can
also understand that the direction we're taking might not be
appropriate for everyone.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
From: Alexander Zimmermann <hidden> Date: 2011-05-19 06:36:15
Am 19.05.2011 um 08:25 schrieb tsuna:
On Wed, May 18, 2011 at 11:10 PM, Alexander Zimmermann
[off-list ref] wrote:
quoted
Am 19.05.2011 um 06:33 schrieb tsuna:
quoted
Presumably if the user decides to tweak these knobs, they'll know
what's appropriate for their environment.
Are you sure? I'm not. I fully agree with David that minRTO is
s/minRTO/initRTO/, right?
Yes of course :-)
quoted
something that a user shout not control at all
I personally don't like to hold the hand and spoon feed users too
much, I want to trust them to be responsible and know what they're
doing. Yes, there will always be people who will act stupid and do
stupid things with whatever knobs you expose. The web is full of
people who advise to tune up all the TCP rmem/wmem parameters to crazy
high level based on the voodoo belief that they're going to improve
their TCP performance, but then as long as you have knobs in your
system, these people will misuse them anyway and shoot themselves in
the foot, what can we do about that.
But if you tune rmen/wmen to crazy level, it's only your TCP performance
that hurts (and maybe the receiver's one).
If you set the initRTO=0.1s, it's good for me but bad for the rest of the
world. That's the difference.
Or do you want to implement a lower barrier of 1sec so that you can ensure
that nobody set the initRTO lower than 1s?
There's also a good chunk of people who know what they're doing, and
for them compile-time constants are annoying because it's inconvenient
to experiment and iterate quickly when you need to recompile your
kernel to change a value. If turning the compile time constant into a
knob leaves the code reasonably straightforward and doesn't incur too
much overhead, then why not do it?
Regarding this knob in particular, I can imagine that people who are
in environment where RTT easily gets around 1s will be upset by the
change in the default value, and doubly upset that they have to
recompile their kernel to change the value back to 3s. I'm in favor
of the reduction of initRTO, for the same reason Google is, but I can
also understand that the direction we're taking might not be
appropriate for everyone.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22222
// email: zimmermann@cs.rwth-aachen.de
// web: http://www.umic-mesh.net
//
From: Benoit Sigoure <redacted>
Draft RFC 2988bis-02 recommends that the initial RTO be lowered
from 3 seconds down to 1 second, and that in case of a timeout
during the TCP 3WHS, the RTO should fallback to 3 seconds when
data transmission begins.
---
On Wed, May 18, 2011 at 10:46 PM, David Miller [off-list ref] wrote:
From: tsuna <redacted>
Date: Wed, 18 May 2011 21:33:21 -0700
quoted
On Wed, May 18, 2011 at 9:14 PM, David Miller [off-list ref] wrote:
quoted
I really would rather see the initial RTO be static and be set to 1
with fallback RTO of 3.
I can also provide a simple patch for this if you want to start from
there. And then maybe we can discuss having a runtime knob some more
:-)
Yeah why don't we do that :-)
Alright, here we go.
include/net/tcp.h | 5 ++++-
net/ipv4/tcp_input.c | 13 +++++++++----
2 files changed, 13 insertions(+), 5 deletions(-)
@@ -868,6 +868,11 @@ static void tcp_init_metrics(struct sock *sk){structtcp_sock*tp=tcp_sk(sk);structdst_entry*dst=__sk_dst_get(sk);+/* If we had to retransmit anything during the 3WHS, use+*theinitialfallbackRTOasperdraftRFC2988bis-02.+*/+intinit_rto=inet_csk(sk)->icsk_retransmits?+TCP_TIMEOUT_INIT_FALLBACK:TCP_TIMEOUT_INIT;if(dst==NULL)gotoreset;
@@ -890,7 +895,7 @@ static void tcp_init_metrics(struct sock *sk)if(dst_metric(dst,RTAX_RTT)==0)gotoreset;-if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(TCP_TIMEOUT_INIT<<3))+if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(init_rto<<3))gotoreset;/* Initial rtt is determined from SYN,SYN-ACK.
@@ -916,7 +921,7 @@ static void tcp_init_metrics(struct sock *sk)tp->mdev_max=tp->rttvar=max(tp->mdev,tcp_rto_min(sk));}tcp_set_rto(sk);-if(inet_csk(sk)->icsk_rto<TCP_TIMEOUT_INIT&&!tp->rx_opt.saw_tstamp){+if(inet_csk(sk)->icsk_rto<init_rto&&!tp->rx_opt.saw_tstamp){reset:/* Play conservative. If timestamps are not*supported,TCPwillfailtorecalculatecorrect
On Wed, May 18, 2011 at 11:36 PM, Alexander Zimmermann
[off-list ref] wrote:
If you set the initRTO=0.1s, it's good for me but bad for the rest of the
world. That's the difference.
Or do you want to implement a lower barrier of 1sec so that you can ensure
that nobody set the initRTO lower than 1s?
Oh, I see. Yes, there is a lower bound (and an upper bound) on what
values the kernel will accept as initRTO. In the patch "Implement a
two-level initial RTO as per draft RFC 2988bis-02" above, I re-used
TCP_RTO_MIN and TCP_RTO_MAX in net/ipv4/sysctl_net_ipv4.c in order to
prevent users from setting a minRTO that's outside this range. They
are defined as follows in tcp.h:
#define TCP_RTO_MAX ((unsigned)(120*HZ))
#define TCP_RTO_MIN ((unsigned)(HZ/5))
So we're talking about a [200ms ; 120s] range no matter what.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
Draft RFC 2988bis-02 recommends that the initial RTO be lowered
from 3 seconds down to 1 second, and that in case of a timeout
during the TCP 3WHS, the RTO should fallback to 3 seconds when
data transmission begins.
Signed-off-by: Benoit Sigoure <redacted>
---
Apologies for the spam, I sent this patch from the wrong address and without
sob'ing it. I build the Linux kernel in a 15G tmpfs (it's faster this way :D)
and I lost my .git/config after a reboot.
include/net/tcp.h | 5 ++++-
net/ipv4/tcp_input.c | 13 +++++++++----
2 files changed, 13 insertions(+), 5 deletions(-)
@@ -868,6 +868,11 @@ static void tcp_init_metrics(struct sock *sk){structtcp_sock*tp=tcp_sk(sk);structdst_entry*dst=__sk_dst_get(sk);+/* If we had to retransmit anything during the 3WHS, use+*theinitialfallbackRTOasperdraftRFC2988bis-02.+*/+intinit_rto=inet_csk(sk)->icsk_retransmits?+TCP_TIMEOUT_INIT_FALLBACK:TCP_TIMEOUT_INIT;if(dst==NULL)gotoreset;
@@ -890,7 +895,7 @@ static void tcp_init_metrics(struct sock *sk)if(dst_metric(dst,RTAX_RTT)==0)gotoreset;-if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(TCP_TIMEOUT_INIT<<3))+if(!tp->srtt&&dst_metric_rtt(dst,RTAX_RTT)<(init_rto<<3))gotoreset;/* Initial rtt is determined from SYN,SYN-ACK.
@@ -916,7 +921,7 @@ static void tcp_init_metrics(struct sock *sk)tp->mdev_max=tp->rttvar=max(tp->mdev,tcp_rto_min(sk));}tcp_set_rto(sk);-if(inet_csk(sk)->icsk_rto<TCP_TIMEOUT_INIT&&!tp->rx_opt.saw_tstamp){+if(inet_csk(sk)->icsk_rto<init_rto&&!tp->rx_opt.saw_tstamp){reset:/* Play conservative. If timestamps are not*supported,TCPwillfailtorecalculatecorrect
From: Alexander Zimmermann <hidden> Date: 2011-05-19 06:52:15
Am 19.05.2011 um 08:42 schrieb tsuna:
On Wed, May 18, 2011 at 11:36 PM, Alexander Zimmermann
[off-list ref] wrote:
quoted
If you set the initRTO=0.1s, it's good for me but bad for the rest of the
world. That's the difference.
Or do you want to implement a lower barrier of 1sec so that you can ensure
that nobody set the initRTO lower than 1s?
Oh, I see. Yes, there is a lower bound (and an upper bound) on what
values the kernel will accept as initRTO. In the patch "Implement a
two-level initial RTO as per draft RFC 2988bis-02" above, I re-used
TCP_RTO_MIN and TCP_RTO_MAX in net/ipv4/sysctl_net_ipv4.c in order to
prevent users from setting a minRTO that's outside this range. They
are defined as follows in tcp.h:
#define TCP_RTO_MAX ((unsigned)(120*HZ))
#define TCP_RTO_MIN ((unsigned)(HZ/5))
So we're talking about a [200ms ; 120s] range no matter what.
Why is 200ms a valid lower bound for initRTO? I'm aware of
measurements that 1s is save for Internet, but I don't know of any
studies that 200ms is save...
On Wed, May 18, 2011 at 11:52 PM, Alexander Zimmermann
[off-list ref] wrote:
quoted
So we're talking about a [200ms ; 120s] range no matter what.
Why is 200ms a valid lower bound for initRTO? I'm aware of
measurements that 1s is save for Internet, but I don't know of any
studies that 200ms is save...
The constants that are quoted aren't specific to the initRTO. They're
used to bound the RTO as it gets adjusted during the TCP session. See
`tcp_set_rto' in tcp_input.c for reference.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
From: Hagen Paul Pfeifer <hidden> Date: 2011-05-19 08:02:11
On Thu, 19 May 2011 08:52:10 +0200, Alexander Zimmermann wrote:
quoted
#define TCP_RTO_MAX ((unsigned)(120*HZ))
quoted
#define TCP_RTO_MIN ((unsigned)(HZ/5))
quoted
quoted
So we're talking about a [200ms ; 120s] range no matter what.
Why is 200ms a valid lower bound for initRTO? I'm aware of
measurements that 1s is save for Internet, but I don't know of any
studies that 200ms is save...
TCP_RTO_MAX and TCP_RTO_MIN is the lower/upper bound for the RTO in
general, not for the initial RTO. RFC 2988 specify a lower bound of 1
second but all operating system choose a lower one because at the time
where RFC 2988 was written the clock granularity was not that accurate. The
minimum RTO for FreeBSD is even 30ms! Furthermore, analysis had
demonstrated that a minimum RTO of 1 second badly breaks throughput in
environments faster then 33kB with minor packet loss rate (e.g. 1%).
So yes, it CAN be wise to choose other lower/upper bounds. But keep in
mind that we should NOT artificial limit ourself. I can image data center
scenarios where a initial RTO of <1 match perfectly.
Hagen
On Thu, May 19, 2011 at 1:02 AM, Hagen Paul Pfeifer [off-list ref] wrote:
So yes, it CAN be wise to choose other lower/upper bounds. But keep in
mind that we should NOT artificial limit ourself. I can image data center
scenarios where a initial RTO of <1 match perfectly.
Yes that's exactly the point I was trying to make when talking to
Alexander offline. On today's Internet, RTTs are easily in the
hundreds of ms, and initRTO is 3s, so there's 2 orders of magnitude of
difference. In my environment, if my RTT is ~2µs, an initRTO of 200ms
means that there's a gap of 6 orders of magnitude (!). And yes,
although I don't work for High Frequency Trading companies in Wall
Street, I'm already buying switches full of line-rate 10Gb ports with
a port-to-port latency of 500ns for L2/L3 forwarding/switching. I
expect this kind of network gear will quickly become prevalent in
datacenter/backend environments.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
From: Alexander Zimmermann <hidden> Date: 2011-05-19 16:56:05
Am 19.05.2011 um 18:40 schrieb tsuna:
On Thu, May 19, 2011 at 1:02 AM, Hagen Paul Pfeifer [off-list ref] wrote:
quoted
So yes, it CAN be wise to choose other lower/upper bounds. But keep in
mind that we should NOT artificial limit ourself. I can image data center
scenarios where a initial RTO of <1 match perfectly.
Yes that's exactly the point I was trying to make when talking to
Alexander offline. On today's Internet, RTTs are easily in the
hundreds of ms, and initRTO is 3s, so there's 2 orders of magnitude of
difference. In my environment,
Exactly. This is the point. It's *your* environment. However, TCP is
general purpose. And for the wider internet 1s is know to be save. See the
measurements in the draft that Mark Allman run.
if my RTT is ~2µs, an initRTO of 200ms
means that there's a gap of 6 orders of magnitude (!).
Currently, initRTO is 3s. So you the gap is even larger.
And yes,
although I don't work for High Frequency Trading companies in Wall
Street, I'm already buying switches full of line-rate 10Gb ports with
a port-to-port latency of 500ns for L2/L3 forwarding/switching. I
expect this kind of network gear will quickly become prevalent in
datacenter/backend environments.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22222
// email: zimmermann@cs.rwth-aachen.de
// web: http://www.umic-mesh.net
//
On Thu, May 19, 2011 at 9:55 AM, Alexander Zimmermann
[off-list ref] wrote:
Exactly. This is the point. It's *your* environment. However, TCP is
general purpose. And for the wider internet 1s is know to be save. See the
measurements in the draft that Mark Allman run.
That's right, there's no one-size-fits-all solution. That's why I'm
in favor of keeping a reasonably conservative default (say 1s to 3s,
so we don't break the Internets) and giving people a knob to adjust it
to whatever makes sense for them.
Looking through the kernel, I see that SCTP already has knobs for
this: sctp_rto_initial, sctp_rto_min, sctp_rto_max. You can even
control the constants used to update rttvar and srtt: sctp_rto_alpha,
sctp_rto_beta
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
Hi Benoit,
AFAICT, the passive open side would not fall back the
RTO to 3sec in this change because SYNACK timeouts are not
recorded in icsk_retransmits but reqsk->retrans?
Yuchung
On Wed, May 18, 2011 at 11:36 PM, Benoit Sigoure [off-list ref] wrote:
quoted hunk
From: Benoit Sigoure <redacted>
Draft RFC 2988bis-02 recommends that the initial RTO be lowered
from 3 seconds down to 1 second, and that in case of a timeout
during the TCP 3WHS, the RTO should fallback to 3 seconds when
data transmission begins.
---
On Wed, May 18, 2011 at 10:46 PM, David Miller [off-list ref] wrote:
quoted
From: tsuna <redacted>
Date: Wed, 18 May 2011 21:33:21 -0700
quoted
On Wed, May 18, 2011 at 9:14 PM, David Miller [off-list ref] wrote:
quoted
I really would rather see the initial RTO be static and be set to 1
with fallback RTO of 3.
I can also provide a simple patch for this if you want to start from
there. And then maybe we can discuss having a runtime knob some more
:-)
Yeah why don't we do that :-)
Alright, here we go.
include/net/tcp.h | 5 ++++-
net/ipv4/tcp_input.c | 13 +++++++++----
2 files changed, 13 insertions(+), 5 deletions(-)
@@ -122,7 +122,10 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);
#endif
#define TCP_RTO_MAX ((unsigned)(120*HZ))
#define TCP_RTO_MIN ((unsigned)(HZ/5))
-#define TCP_TIMEOUT_INIT ((unsigned)(3*HZ)) /* RFC 1122 initial RTO value */
+/* The next 2 values come from Draft RFC 2988bis-02. */
+#define TCP_TIMEOUT_INIT ((unsigned)(1*HZ)) /* initial RTO value */
+#define TCP_TIMEOUT_INIT_FALLBACK ((unsigned)(3*HZ)) /* initial RTO to fallback to when
+ * a timeout happens during the 3WHS. */
#define TCP_RESOURCE_PROBE_INTERVAL ((unsigned)(HZ/2U)) /* Maximal interval between probes
* for local resources.
{
struct tcp_sock *tp = tcp_sk(sk);
struct dst_entry *dst = __sk_dst_get(sk);
+ /* If we had to retransmit anything during the 3WHS, use
+ * the initial fallback RTO as per draft RFC 2988bis-02.
+ */
+ int init_rto = inet_csk(sk)->icsk_retransmits ?
+ TCP_TIMEOUT_INIT_FALLBACK : TCP_TIMEOUT_INIT;
if (dst == NULL)
goto reset;
tp->mdev_max = tp->rttvar = max(tp->mdev, tcp_rto_min(sk));
}
tcp_set_rto(sk);
- if (inet_csk(sk)->icsk_rto < TCP_TIMEOUT_INIT && !tp->rx_opt.saw_tstamp) {
+ if (inet_csk(sk)->icsk_rto < init_rto && !tp->rx_opt.saw_tstamp) {
reset:
/* Play conservative. If timestamps are not
* supported, TCP will fail to recalculate correct
@@ -924,8 +929,8 @@ reset:
*/
if (!tp->rx_opt.saw_tstamp && tp->srtt) {
tp->srtt = 0;
- tp->mdev = tp->mdev_max = tp->rttvar = TCP_TIMEOUT_INIT;
- inet_csk(sk)->icsk_rto = TCP_TIMEOUT_INIT;
+ tp->mdev = tp->mdev_max = tp->rttvar = init_rto;
+ inet_csk(sk)->icsk_rto = init_rto;
}
}
tp->snd_cwnd = tcp_init_cwnd(tp, dst);
--
1.7.0.4
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: David Miller <davem@davemloft.net> Date: 2011-05-19 19:31:04
From: tsuna <redacted>
Date: Thu, 19 May 2011 10:11:50 -0700
Looking through the kernel, I see that SCTP already has knobs for
this: sctp_rto_initial, sctp_rto_min, sctp_rto_max. You can even
control the constants used to update rttvar and srtt: sctp_rto_alpha,
sctp_rto_beta
SCTP is 1) not even a sliver of deployment compared to TCP and 2)
doesn't get nearly the same scrutiny on patch review that TCP
changes do.
I basically let the SCTP folks play in their own sandbox, because
frankly SCTP doesn't matter.
The only time I care about an SCTP change is when it has an impact on
the rest of the networking code.
So using SCTP as an example of "see we do this already over here" is a
non-starter. Don't do it.
@@ -868,6 +868,11 @@ static void tcp_init_metrics(struct sock *sk) { struct tcp_sock *tp = tcp_sk(sk); struct dst_entry *dst = __sk_dst_get(sk);+ /* If we had to retransmit anything during the 3WHS, use+ * the initial fallback RTO as per draft RFC 2988bis-02.+ */+ int init_rto = inet_csk(sk)->icsk_retransmits ?+ TCP_TIMEOUT_INIT_FALLBACK : TCP_TIMEOUT_INIT;
Please do not put comments in the middle of a set of function
local variable declarations.
Also, as mentioned already, icsk_retransmits is not where SYN
retransmissions are counted.
It is stored in the TCP minisocket ->retrans field.
On Thu, May 19, 2011 at 12:27 PM, David Miller [off-list ref] wrote:
So using SCTP as an example of "see we do this already over here" is a
non-starter. Don't do it.
Fair enough. I hope that the "there's no one-size-fits-all solution"
argument has more weight than "hey SCTP does it". :)
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
From: H.K. Jerry Chu <hidden> Date: 2011-05-20 02:02:00
On Wed, May 18, 2011 at 12:40 PM, tsuna [off-list ref] wrote:
On Wed, May 18, 2011 at 12:26 PM, David Miller [off-list ref] wrote:
quoted
If you read the ietf draft that reduces the initial RTO down to 1
second, it states that if we take a timeout during the initial
connection handshake then we have to revert the RTO back up to 3
seconds.
This fallback logic conflicts with being able to only change the
initial RTO via sysctl, I think. Because there are actually two
values at stake and they depend upon eachother, the initial RTO and
the value we fallback to on initial handshake retransmissions.
So I'd rather get a patch that implements the 1 second initial
RTO with the 3 second fallback on SYN retransmit, than this patch.
We already have too many knobs.
I was hoping this knob would be accepted because this is such an
important issue that it even warrants an IETF draft to attempt to
change the standard. I'm not sure how long it will take for this
draft to be accepted and then implemented, so I thought adding this
simple knob today would really help in the future.
As one of the co-authors of rfc2988bis I was planning to provide a patch
as soon as the draft gets approved but it looks like you have beaten
me to it :)
Personally I'm in favor of a knob too. We at Google has added such a
knob for years.
Jerry
Plus, should the draft be accepted, this knob will still be just as
useful (e.g. to revert back to today's behavior), and people might
want to consider adding another knob for the fallback initRTO (this is
debatable). I don't believe this knob conflicts with the proposed
change to the standard, it actually goes along with it pretty well and
helps us prepare better for this upcoming change.
I agree that there are too many knobs, and I hate feature creep too,
but I've found many of these knobs to be really useful, and the degree
to which Linux's TCP stack can be tuned is part of what makes it so
versatile.
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: H.K. Jerry Chu <hidden> Date: 2011-05-20 10:27:40
On Wed, May 18, 2011 at 1:20 PM, Hagen Paul Pfeifer [off-list ref] wrote:
* David Miller | 2011-05-18 15:52:00 [-0400]:
quoted
I've already changed the initial TCP congestion window in Linux to 10
without some stupid draft being fully accepted.
I'll just as easily accept right now a patch right now which lowers
the initial RTO to 1 second and adds the 3 second RTO fallback.
I like the idea to make the initial RTO a knob because we in a isolated MANET
environment have a RTT larger then 1 second. Especially the link layer setup
procedure over several hops demand some time-costly setup time. After that the
RTT is <1 second. The current algorithm works great for us. So this RTO change
will be counterproductive: it will always trigger a needless timeout.
The main problem for us is that Google at all pushing their view of Internet
with a lot of pressure. The same is true for the IETF IW adjustments, which is
unsuitable for networks which operates at a bandwidth characteristic some
years ago. The _former_ conservative principle "TCP over everything" is
forgotten.
Not sure how our various parameter tuning proposals deviate from the "TCP over
everything" principle?
Note that the design goal of rfc2988bis is to try to benefit 98% of
traffic while
keeping any negative impact to the remaining 2% at a minimum. This is why we
limit the use of < 3sec initRTO to at most once. This way the negative impact
of the 1sec initRTO to a path with RTT > 1sec is limited mostly to one
additional,
small, spuriously retransmitted SYN or SYN-ACK pkt, and the unnecessary
reduction of IW to 1 segment.
We actually thought about removing the IW reduction part but unfortunately the
text belongs to a different rfc5681, which is at a higher maturity
level ("draft-standard")
than rfc2988 hence can't be done as part of rfc2988bis. Anyway I have
since added
the recommendation to the IW10 draft. See draft-ietf-tcpm-initcwnd-01.txt.
The bottom line is the damage of rfc2988bis to any network with
initRTT > 1sec is
limited to one spurious retransmitted SYN/SYN-ACK. In the current Linux code,
the SYN/SYN-ACK retransmit is forgotten on the passive open side by the time
3WHS is completed so there is nothing needed to be done. But for the active
open side SYN retransmit will cause not long IW to be reduced to 1, but also
reduction of ssthresh, which is not part of rfc5681 so some more work is needed.
I can provide a patch (or work with tsuna) to ensure a correct fix is made.
Jerry
Hagen
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Hagen Paul Pfeifer <hidden> Date: 2011-05-20 11:00:05
On Fri, 20 May 2011 03:27:37 -0700, "H.K. Jerry Chu" wrote:
Hi Jerry
Not sure how our various parameter tuning proposals deviate from the
"TCP
over everything" principle?
For our environment it hurts because we _always_ have an initial RTO >1. I
understand and accept that 98% will benefit of this modification, no doubt
Jerry! Try to put yourself in our situation: imaging a proposal of an init
RTO modification to 0.5 seconds. Maybe because 98% of Internet traffic is
now localized and the RTO is average now 0.2 seconds. Anyway, this will
penalize your network always and this will be the situation for one of my
customer. I can live with that, I see the benefits for the rest of the
world. But I am happy to see a knob where I can restore the old behavior.
Maybe some other environments will benefit from a even lower or higher
initial RTO.
Hagen
For our environment it hurts because we _always_ have an initial RTO >1. I
understand and accept that 98% will benefit of this modification, no doubt
Jerry! Try to put yourself in our situation: imaging a proposal of an init
RTO modification to 0.5 seconds. Maybe because 98% of Internet traffic is
now localized and the RTO is average now 0.2 seconds. Anyway, this will
penalize your network always and this will be the situation for one of my
customer. I can live with that, I see the benefits for the rest of the
world. But I am happy to see a knob where I can restore the old behavior.
Maybe some other environments will benefit from a even lower or higher
initial RTO.
AX.25 is definitely happier with a multi-second round trip but it's a
special case. Some X.25 networks are going to have similar behaviour.
It shouldn't be penalising each connection (and it's worse than that of
course because each node on a shared media network gets in the way of the
rest, plus the queueing effect of all the extra blockages) because done
right multiple connections to the same host can use the previous
connections as estimates (and indeed for the initial RTO there's a good
argument for treating estimates as 'host, then x.y.z.* match, then
average of previous except the x.y.z.* match, then unknown')
The latter would fix an awful lot of the weird cases pretty effectively.
Alan
From: H.K. Jerry Chu <hidden> Date: 2011-05-21 00:06:15
Hey Hagen,
On Fri, May 20, 2011 at 4:00 AM, Hagen Paul Pfeifer [off-list ref] wrote:
On Fri, 20 May 2011 03:27:37 -0700, "H.K. Jerry Chu" wrote:
Hi Jerry
quoted
Not sure how our various parameter tuning proposals deviate from the
"TCP
quoted
over everything" principle?
For our environment it hurts because we _always_ have an initial RTO >1. I
understand and accept that 98% will benefit of this modification, no doubt
Jerry! Try to put yourself in our situation: imaging a proposal of an init
Understood but my point was none of the parameter tuning proposals break
"TCP over everything", although they may not help solving "TCP optimized for
everything", but we never had the latter anyway.
We've tried hard to keep the penalty to those initRTT > 1sec paths at a minimum,
i.e., just one extra tinygram. This is important also for us because it may take
1sec for many Android clients to establish connections over a radio channel that
has been put into power saving mode.
RTO modification to 0.5 seconds. Maybe because 98% of Internet traffic is
now localized and the RTO is average now 0.2 seconds. Anyway, this will
penalize your network always and this will be the situation for one of my
customer. I can live with that, I see the benefits for the rest of the
world. But I am happy to see a knob where I can restore the old behavior.
Maybe some other environments will benefit from a even lower or higher
initial RTO.
Yep, that's why we've had a knob for this for years.
Jerry
On Fri, May 20, 2011 at 5:06 PM, H.K. Jerry Chu [off-list ref] wrote:
Yep, that's why we've had a knob for this for years.
I was traveling last week so sorry for not replying earlier to various
comments people made.
I talked to Jerry and he's agreed to share some patches that Google
has been using internally for years. I started this work because
after leaving Google and taking these changes for granted, I was
surprised to find that they weren't actually part of the mainline
Linux kernel.
It seems that David is willing to accept a change that will lower the
initRTO to 1s (compile-time constant), with a fallback to 3s
(compile-time constant), as per the draft rfc2988bis. Others are
legitimately worried about the impact this would cause in environments
where RTT is typically (or always) in the 1-3s range. Some would like
to see this as a per-destination thing.
Personally what I think would be ideal would be:
1. A sysctl knob for initRTO, to allow people to adjust this
appropriately for their environment.
2. Apply the srtt / rttvar seen on previous connections to new connections.
Does that sound reasonable?
For 2), I'm not sure how the details would work yet, I believe the
kernel already has what's necessary to remember these things on a per
peer basis, but it would be nice if I could specify things like "for
10.x.0.0/16 (local datacenter) use this aggressive setting, for
10.0.0.0/8 (my internal backend network) use that, for everything else
(Internets etc.) use the default".
--
Benoit "tsuna" Sigoure
Software Engineer @ www.StumbleUpon.com
From: Hagen Paul Pfeifer <hidden> Date: 2011-05-31 15:25:26
On Tue, 31 May 2011 07:48:09 -0700, tsuna [off-list ref] wrote:
I talked to Jerry and he's agreed to share some patches that Google
has been using internally for years.
Great!
Personally what I think would be ideal would be:
1. A sysctl knob for initRTO, to allow people to adjust this
appropriately for their environment.
2. Apply the srtt / rttvar seen on previous connections to new
connections.
Does that sound reasonable?
For 2), I'm not sure how the details would work yet, I believe the
kernel already has what's necessary to remember these things on a per
peer basis, but it would be nice if I could specify things like "for
10.x.0.0/16 (local datacenter) use this aggressive setting, for
10.0.0.0/8 (my internal backend network) use that, for everything else
(Internets etc.) use the default".
Skip sysctl, it is deprecated. The initRTO is the ideal candidate for a
per route knob. And happily you will solve 2) with the per route thing too!
;-)
Search the web, you will find some patches where you can see how to extend
the per route system - including iproute2.
Hagen
From: Hagen Paul Pfeifer <hidden> Date: 2011-05-31 15:43:27
On Tue, 31 May 2011 08:28:18 -0700, tsuna [off-list ref] wrote:
Sorry I meant a knob such as /proc/sys/net/ipv4/tcp_initrto.
That's the same! ;-)
quoted
The initRTO is the ideal candidate for a
quoted
per route knob. And happily you will solve 2) with the per route thing
quoted
too!
You still need a knob for the default system-wide value, don't you?
Yes, try to re-read the emails. Sysctl is a no-go, with a per route
interface you have the ability to tune the values. Talk with Jerry once
again - he wrote that at Google they already have a patch for this. And
with a per route knob you can select a even smaller value for your local
network (e.g. datacenter) and a larger value for all other routes. It makes
sense to provide a knob for this on a route basis, not on a global sysctl
basis.
But once again: talk with Jerry - he has the expert knowledge!
Hagen