Hi,
I've got a problem with network on one of my embedded boards.
I'm testing download speed of 256MB file from my PC to embedded board
through 1Gbit ethernet link using ftp.
The problem is that sometimes I achieve 25MB/s and sometimes it is only
14MB/s. There are also situations where the transfer speed starts at
14MB/s and after a few seconds achieves 25MB/s.
I've caught the second case with tcpdump and I noticed that when the speed
is 14MB/s - the tcp window size is 534368 bytes and when the speed
achieved 25MB/s the tcp window size is 933888.
My question is: what causes such dynamic change in the window size (while
transferring data)? Is it some kernel parameter wrong set or something
like this?
Do I have any influence on such dynamic change in tcp window size?
sdrb
On Tue, Feb 23, 2016 at 6:19 AM, [off-list ref] wrote:
My question is: what causes such dynamic change in
the window size (while transferring data)?
Is it some kernel parameter wrong set or something like this?
That sounds like TCP receive buffer auto-tuning (also called "Dynamic
right-sizing (DRS)" (Fisk and Feng, 2001):
http://permalink.lanl.gov/object/tr?what=info:lanl-repo/lareport/LA-UR-01-5460
The Linux TCP receiver will, by default, dynamically adjust the
receive window to a value that supports the rate at which the
application successfully reads data out of the socket.
Do I have any influence on such dynamic change in tcp window size?
You might check on the receiver host with top/mpstat/strace/etc to see
whether the receiving application is limiting performance in some way.
This kind of behavior can show up if the receiver is sometimes
CPU-saturated, or limited by the throughput of the medium on to which
it is writing the data.
If you control the receiver software, you can use
setsockopt(SO_RCVBUF) to explicitly set the receive buffer size, and
see if that helps. If it doesn't, that would suggest that it is indeed
the receiving application that is limiting performance.
If you could provide a tcpdump trace (headers only, e.g., -s 96)
taken on the sender, we could check to see if we can see any problems
in the TCP sender or receiver behavior.
cheers,
neal
On Tue, Feb 23, 2016 at 6:19 AM, [off-list ref] wrote:
quoted
My question is: what causes such dynamic change in
the window size (while transferring data)?
Is it some kernel parameter wrong set or something like this?
That sounds like TCP receive buffer auto-tuning (also called "Dynamic
right-sizing (DRS)" (Fisk and Feng, 2001):
http://permalink.lanl.gov/object/tr?what=info:lanl-repo/lareport/LA-UR-01-5460
The Linux TCP receiver will, by default, dynamically adjust the
receive window to a value that supports the rate at which the
application successfully reads data out of the socket.
quoted
Do I have any influence on such dynamic change in tcp window size?
You might check on the receiver host with top/mpstat/strace/etc to see
whether the receiving application is limiting performance in some way.
This kind of behavior can show up if the receiver is sometimes
CPU-saturated, or limited by the throughput of the medium on to which
it is writing the data.
If you control the receiver software, you can use
setsockopt(SO_RCVBUF) to explicitly set the receive buffer size, and
see if that helps. If it doesn't, that would suggest that it is indeed
the receiving application that is limiting performance.
If you could provide a tcpdump trace (headers only, e.g., -s 96)
taken on the sender, we could check to see if we can see any problems
in the TCP sender or receiver behavior.
Thanks for the trace. It looks like for the first 0.8 seconds of the
trace some non-TCP component on the receiving machine (app, CPU, CPU
power-saving mechanisms, NIC, or NIC driver) is limiting throughput to
78Mbit/sec (e.g. the very first window of 10 packets is ACKed at that
rate). Then the throughput increases to over 200 Mbit/sec. AFAICT it
doesn't look like the TCP layer is doing anything wrong. I'd look for
issues with those other components. It might help to report the
receiver application, receiver CPU type, kernel version, NIC, and NIC
driver.
neal
Thanks for the trace. It looks like for the first 0.8 seconds of the
trace some non-TCP component on the receiving machine (app, CPU, CPU
power-saving mechanisms, NIC, or NIC driver) is limiting throughput to
78Mbit/sec (e.g. the very first window of 10 packets is ACKed at that
rate). Then the throughput increases to over 200 Mbit/sec. AFAICT it
doesn't look like the TCP layer is doing anything wrong. I'd look for
issues with those other components. It might help to report the
receiver application, receiver CPU type, kernel version, NIC, and NIC
driver.
Hi,
The receiver application was "wget" using ftp protocol.
Linux kernel is in version 3.4.11-rt19.
Concerning hardware I have not much information about it.
Everything I've got is:
- CPU: ARMv7 Processor rev 1 (v7l)
- NIC: Broadcom PCI: 14e4:aa52
- RAM: 256MB
Unfortunately I've no much info about kernel. Seems like it is some kind
of realtime version. Can this realtime extension cause such effect of
increasing throughput?
I've made a graph with window size and throughput from published
test7.pcap file:
https://www.dropbox.com/s/cwars3oaoax73fh/wget_test_test7pcap_20160224_1.png?dl=0
sdrb