Hi,
Recently I tackled round trip time estimation of a TCP connection.
After implementing a straight-forward approach (time stamping sending
and receiving of data using clock_gettime) I found this article:
http://linuxgazette.net/136/pfeiffer.html (using getsockopt() to get
struct tcp_info). The tcp_info structure conveniently has a rtt field.
Using the first method I get 1-3 ms RTT, and by using the second I get
=10 ms RTT.
By looking at the code it's clear that the time stamping is done with
jiffies, and my kernel has CONFIG_HZ=100.
I understand that this is for performance reasons (and the RTT
smoothing filter is implemented with bit shifting operations), but
would using a more precise time stamp have significant impact on
performance? Since RTT is used to compute RTO, wouldn't there be any
benefits of having more accurate estimate of this value?
Best regards,
Srećko Jurić-Kavelj, dipl.ing. (Ms.E.E)
Research and Teaching Assistant at University of Zagreb
(Faculty of Electrical Engineering and Computing, Department of
Control and Computer Engineering)
E-mail: srecko.juric-kavelj@fer.hr
URL: http://www.fer.hr/srecko.juric-kavelj
Sanctus Hieronymus: "Parce mihi, Domine, quia dalmata sum!"
From: Chris Friesen <hidden> Date: 2012-05-25 15:58:51
On 05/22/2012 11:21 AM, Srećko Jurić-Kavelj wrote:
By looking at the code it's clear that the time stamping is done with
jiffies, and my kernel has CONFIG_HZ=100.
I understand that this is for performance reasons (and the RTT
smoothing filter is implemented with bit shifting operations), but
would using a more precise time stamp have significant impact on
performance? Since RTT is used to compute RTO, wouldn't there be any
benefits of having more accurate estimate of this value?
I don't know if it would make any difference to the tcp algorithms, but
certainly on some architectures you can get a fast and accurate hardware
timestamp.
Chris
From: Dave Taht <hidden> Date: 2012-05-25 16:17:45
On Fri, May 25, 2012 at 4:58 PM, Chris Friesen
[off-list ref] wrote:
On 05/22/2012 11:21 AM, Srećko Jurić-Kavelj wrote:
quoted
By looking at the code it's clear that the time stamping is done with
jiffies, and my kernel has CONFIG_HZ=100.
I understand that this is for performance reasons (and the RTT
smoothing filter is implemented with bit shifting operations), but
would using a more precise time stamp have significant impact on
performance? Since RTT is used to compute RTO, wouldn't there be any
benefits of having more accurate estimate of this value?
I don't know if it would make any difference to the tcp algorithms, but
certainly on some architectures you can get a fast and accurate hardware
timestamp.
I would be interested in someone doing that experiment in light of the
codel work.
Chris
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, May 25, 2012 at 6:17 PM, Dave Taht [off-list ref] wrote:
On Fri, May 25, 2012 at 4:58 PM, Chris Friesen
[off-list ref] wrote:
quoted
I don't know if it would make any difference to the tcp algorithms, but
certainly on some architectures you can get a fast and accurate hardware
timestamp.
I would be interested in someone doing that experiment in light of the
codel work.
I've looked this up in other implementations, e.g. FreeBSD uses 1ms
granularity no matter what HZ says, NetBSD has 500ms ticks, ...
I guess that granularity also depends on the retransmit timers used. I
didn't make out what's the precision of the timers that Linux uses in
TCP, but I guess it uses high resolution timers? At least on x86?
I've done a simple experiment by repeatedly calling clock_gettime
(from userspace, but I guess it ends up as a vsyscall). I get >17
million calls per second on a Q6600.
--
JKS
From: Eric Dumazet <hidden> Date: 2012-05-25 16:54:43
On Fri, 2012-05-25 at 18:23 +0200, Srećko Jurić-Kavelj wrote:
On Fri, May 25, 2012 at 6:17 PM, Dave Taht [off-list ref] wrote:
quoted
On Fri, May 25, 2012 at 4:58 PM, Chris Friesen
[off-list ref] wrote:
quoted
I don't know if it would make any difference to the tcp algorithms, but
certainly on some architectures you can get a fast and accurate hardware
timestamp.
I would be interested in someone doing that experiment in light of the
codel work.
I've looked this up in other implementations, e.g. FreeBSD uses 1ms
granularity no matter what HZ says, NetBSD has 500ms ticks, ...
I guess that granularity also depends on the retransmit timers used. I
didn't make out what's the precision of the timers that Linux uses in
TCP, but I guess it uses high resolution timers? At least on x86?
I've done a simple experiment by repeatedly calling clock_gettime
(from userspace, but I guess it ends up as a vsyscall). I get >17
million calls per second on a Q6600.
linux TCP uses high precision timestamps (ktime_get_real()) where
needed.
# find net|xargs grep -n TCP_CONG_RTT_STAMP
net/ipv4/tcp_veno.c:205: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_vegas.c:308: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_cubic.c:478: cubictcp.flags |= TCP_CONG_RTT_STAMP;
net/ipv4/tcp_output.c:815: if (icsk->icsk_ca_ops->flags & TCP_CONG_RTT_STAMP)
net/ipv4/tcp_lp.c:317: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_yeah.c:229: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_illinois.c:326: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_input.c:3496: if (ca_ops->flags & TCP_CONG_RTT_STAMP &&
Other than that HZ=1000 seems fine.
HZ=100 seems a poor choice, we have NO_HZ since a long time.
On Fri, May 25, 2012 at 6:54 PM, Eric Dumazet [off-list ref] wrote:
linux TCP uses high precision timestamps (ktime_get_real()) where
needed.
# find net|xargs grep -n TCP_CONG_RTT_STAMP
net/ipv4/tcp_veno.c:205: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_vegas.c:308: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_cubic.c:478: cubictcp.flags |= TCP_CONG_RTT_STAMP;
net/ipv4/tcp_output.c:815: if (icsk->icsk_ca_ops->flags & TCP_CONG_RTT_STAMP)
net/ipv4/tcp_lp.c:317: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_yeah.c:229: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_illinois.c:326: .flags = TCP_CONG_RTT_STAMP,
net/ipv4/tcp_input.c:3496: if (ca_ops->flags & TCP_CONG_RTT_STAMP &&
Didn't know about TCP_CONG_RTT_STAMP.
Thing is, the device I'm connecting to doesn't even support TCP time
stamp option. The returning SYN ACK packet only has maximum segment
size 1460 bytes in options.
From the net/ipv4/tcp_input.c code, RTT is estimated using
#define tcp_time_stamp ((__u32)(jiffies))
from include/net/tcp.h.
Could ktime_get_real() be used for tcp_time_stamp instead of jiffies?
Other than that HZ=1000 seems fine.
HZ=100 seems a poor choice, we have NO_HZ since a long time.
I have:
$ grep HZ /boot/config-2.6.32-41-generic
CONFIG_NO_HZ=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_MACHZ_WDT=m
From what I've seen in the code, NO_HZ doesn't make jiffies go away,
it simply doesn't use regular CONFIG_HZ interrupt to update, but
updates them when has an opportunity?
--
JKS
From: Eric Dumazet <hidden> Date: 2012-05-25 19:00:37
On Fri, 2012-05-25 at 20:35 +0200, Srećko Jurić-Kavelj wrote:
From what I've seen in the code, NO_HZ doesn't make jiffies go away,
it simply doesn't use regular CONFIG_HZ interrupt to update, but
updates them when has an opportunity?
HZ=1000 makes jiffies 10 times more precise, and with NO_HZ, generates
no extra timer interrupts.
This also makes timers workload smoothed, instead of spikes.