Bad TCP timestamps on non-PC platforms

Greetings.
I'm dealing with 2.6.37-rc7 kernel on MPC8548 platform.

It so appears, that recent kernels have sysctl_tcp_timestamps set to "1"
by default.

On embedded platforms, where real time clock is initialized lately or
absent outright, this causes TSval field of outgoing TCP packets to be
set to some garbage value, in my case in the vicinity of 0xffffffff. As a
result, other Linux machines silently drop such packets, preventing normal
completion of network boot or any other TCP dependent operation.

Therefore, I feel, two changes are necessary (I can send in a patch):
1. Make sysctl_tcp_timestamps value config-time selectable (it must be
disabled by default on machines without RTC).
2. When re-enabling tcp_timestamps through sysctl, reset the timestamp
counter to the current system clock value.

And an optional, tricky one:
3. Postpone TCP timestamp counter initialization until RTC is actually
available (if RTC is connected to i2c bus, TCP is initialized well ahead
of it).



      

Le mercredi 05 janvier 2011 à 22:55 -0800, Alex Dubov a écrit :

Greetings.
I'm dealing with 2.6.37-rc7 kernel on MPC8548 platform.

It so appears, that recent kernels have sysctl_tcp_timestamps set to "1"
by default.

On embedded platforms, where real time clock is initialized lately or
absent outright, this causes TSval field of outgoing TCP packets to be
set to some garbage value, in my case in the vicinity of 0xffffffff. As a
result, other Linux machines silently drop such packets, preventing normal
completion of network boot or any other TCP dependent operation.

Therefore, I feel, two changes are necessary (I can send in a patch):
1. Make sysctl_tcp_timestamps value config-time selectable (it must be
disabled by default on machines without RTC).
2. When re-enabling tcp_timestamps through sysctl, reset the timestamp
counter to the current system clock value.

And an optional, tricky one:
3. Postpone TCP timestamp counter initialization until RTC is actually
available (if RTC is connected to i2c bus, TCP is initialized well ahead
of it).

I am trying to understand the problem, but I guess you hit something not
related to RTC at all.

#define tcp_time_stamp		((__u32)(jiffies))
#define INITIAL_JIFFIES ((unsigned long)(unsigned int) (-300*HZ))

jiffies can be ZERO 300 seconds after boot.

It should not matter with RFC1323. Some TCP session could in theory be
done without TCP timestamps being activated (because SYN paquet would
carry a ZERO  tsval/tsecr), but this should not prevents normal network
operation.

Are you sure the "other linux machines" dont have
strange /etc/sysctl.conf settings ?

From: Alex Dubov <oakad@yahoo.com>
Date: Wed, 5 Jan 2011 22:55:43 -0800 (PST)

On embedded platforms, where real time clock is initialized lately or
absent outright, this causes TSval field of outgoing TCP packets to be
set to some garbage value, in my case in the vicinity of 0xffffffff.

The TCP timestamp is set from "jiffies", not from the RTC clock.

Sorry for the awful synopsis of my problem. I never cease to amaze myself
at how bad those usually turn up. :-)

What I really meant to write is:

I have a dev board running 2.6.37-rc7. Normal kernel config, nothing fancy.
Remote machines are just usual linux boxes in constant operation (I tried
several of those).

UDP/DHCP works correctly all the time, so ethernet side is probably ok.

When tcp_timestamps are enabled, SYN packets from dev board just get
ignored by the remote side. I see them arrive in wireshark, but nothing
else happens.

When I disable tcp_timestamps on the dev board everything works.

The problem is reproducible every single time.

The only difference is the "Options" block of the SYN packets.
If timestamps are not really to blame, then it probably window scale
parameters. That's what I see on a usual dropped packet:

Options: (20 bytes)
        Maximum segment size: 1460 bytes
        SACK permitted
        Timestamps: TSval 4294893842, TSecr 0
        NOP
        Window scale: 5 (multiply by 32)



      

Le jeudi 06 janvier 2011 à 17:55 -0800, Alex Dubov a écrit :

Sorry for the awful synopsis of my problem. I never cease to amaze myself
at how bad those usually turn up. :-)

What I really meant to write is:

I have a dev board running 2.6.37-rc7. Normal kernel config, nothing fancy.
Remote machines are just usual linux boxes in constant operation (I tried
several of those).

UDP/DHCP works correctly all the time, so ethernet side is probably ok.

When tcp_timestamps are enabled, SYN packets from dev board just get
ignored by the remote side. I see them arrive in wireshark, but nothing
else happens.

When I disable tcp_timestamps on the dev board everything works.

The problem is reproducible every single time.

The only difference is the "Options" block of the SYN packets.
If timestamps are not really to blame, then it probably window scale
parameters. That's what I see on a usual dropped packet:

Options: (20 bytes)
        Maximum segment size: 1460 bytes
        SACK permitted
        Timestamps: TSval 4294893842, TSecr 0
        NOP
        Window scale: 5 (multiply by 32)



      

You dont give new informations ;)

I asked if you could give information on the other side : The bug is to
drop this legal packet.

uname -a
sysctl -a | grep tcp

You dont give new informations ;)

I asked if you could give information on the other side :
The bug is to
drop this legal packet.

uname -a
sysctl -a | grep tcp

One of the other machines:

Linux sunapp 2.6.31.3 #2 SMP Mon Oct 12 21:32:20 EST 2009 x86_64 Dual-Core AMD Opteron(tm) Processor 8220 AuthenticAMD GNU/Linux

error: permission denied on key 'net.ipv4.route.flush'                          
fs.nfs.nlm_tcpport = 0
fs.nfs.nfs_callback_tcpport = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_fack = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_mem = 3095040      4126720 6190080
net.ipv4.tcp_wmem = 4096        16384   4194304
net.ipv4.tcp_rmem = 4096        87380   4194304
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_frto = 2
net.ipv4.tcp_frto_response = 0
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_congestion_control = reno
net.ipv4.tcp_abc = 0
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.tcp_dma_copybreak = 4096
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_available_congestion_control = reno
net.ipv4.tcp_allowed_congestion_control = reno
net.ipv4.tcp_max_ssthresh = 0
error: permission denied on key 'net.ipv6.route.flush'
sunrpc.transports = tcp 1048576
sunrpc.tcp_slot_table_entries = 16
sunrpc.tcp_fin_timeout = 15