Why ping latency is smaller with shorter send interval?

Hi, net experts,

Hope this is the right place to ask the question :)

Recently I've tried to measure the network latency between two
machines by using ping, one interesting observation I found is that
ping latency will be smaller if I use a shorter interval with -i
option. For example,

when I use default ping (interval is 1s), then the ping result is as
below with avg latency 0.062ms

# ping 9.9.9.2 -c 10
PING 9.9.9.2 (9.9.9.2) 56(84) bytes of data.
64 bytes from 9.9.9.2: icmp_seq=1 ttl=64 time=0.059 ms
64 bytes from 9.9.9.2: icmp_seq=2 ttl=64 time=0.079 ms
64 bytes from 9.9.9.2: icmp_seq=3 ttl=64 time=0.060 ms
64 bytes from 9.9.9.2: icmp_seq=4 ttl=64 time=0.072 ms
64 bytes from 9.9.9.2: icmp_seq=5 ttl=64 time=0.048 ms
64 bytes from 9.9.9.2: icmp_seq=6 ttl=64 time=0.069 ms
64 bytes from 9.9.9.2: icmp_seq=7 ttl=64 time=0.067 ms
64 bytes from 9.9.9.2: icmp_seq=8 ttl=64 time=0.055 ms
64 bytes from 9.9.9.2: icmp_seq=9 ttl=64 time=0.058 ms
64 bytes from 9.9.9.2: icmp_seq=10 ttl=64 time=0.055 ms

--- 9.9.9.2 ping statistics ---

10 packets transmitted, 10 received, 0% packet loss, time 9001ms
rtt min/avg/max/mdev = 0.048/0.062/0.079/0.010 ms

Then I use "-i 0.001", the lateny (0.038) is way better than defaut ping

# ping 9.9.9.2 -i 0.001 -c 10
PING 9.9.9.2 (9.9.9.2) 56(84) bytes of data.
64 bytes from 9.9.9.2: icmp_seq=1 ttl=64 time=0.069 ms
64 bytes from 9.9.9.2: icmp_seq=2 ttl=64 time=0.039 ms
64 bytes from 9.9.9.2: icmp_seq=3 ttl=64 time=0.034 ms
64 bytes from 9.9.9.2: icmp_seq=4 ttl=64 time=0.033 ms
64 bytes from 9.9.9.2: icmp_seq=5 ttl=64 time=0.033 ms
64 bytes from 9.9.9.2: icmp_seq=6 ttl=64 time=0.033 ms
64 bytes from 9.9.9.2: icmp_seq=7 ttl=64 time=0.034 ms
64 bytes from 9.9.9.2: icmp_seq=8 ttl=64 time=0.036 ms
64 bytes from 9.9.9.2: icmp_seq=9 ttl=64 time=0.037 ms
64 bytes from 9.9.9.2: icmp_seq=10 ttl=64 time=0.038 ms

--- 9.9.9.2 ping statistics ---

10 packets transmitted, 10 received, 0% packet loss, time 9ms
rtt min/avg/max/mdev = 0.033/0.038/0.069/0.012 ms


ping loopback shows the similar result.

Default ping avg latency is 0.049ms

# ping 127.0.0.1 -c 10
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.032 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.049 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.054 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.058 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.049 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.042 ms
64 bytes from 127.0.0.1: icmp_seq=7 ttl=64 time=0.052 ms
64 bytes from 127.0.0.1: icmp_seq=8 ttl=64 time=0.052 ms
64 bytes from 127.0.0.1: icmp_seq=9 ttl=64 time=0.053 ms
64 bytes from 127.0.0.1: icmp_seq=10 ttl=64 time=0.055 ms

--- 127.0.0.1 ping statistics ---

10 packets transmitted, 10 received, 0% packet loss, time 9001ms
rtt min/avg/max/mdev = 0.032/0.049/0.058/0.010 ms

ping with "-i 0.001" shows 0.014ms avg latency.

# ping 127.0.0.1 -i 0.001 -c 10
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.040 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.014 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.012 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.011 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.011 ms
64 bytes from 127.0.0.1: icmp_seq=6 ttl=64 time=0.011 ms
64 bytes from 127.0.0.1: icmp_seq=7 ttl=64 time=0.011 ms
64 bytes from 127.0.0.1: icmp_seq=8 ttl=64 time=0.010 ms
64 bytes from 127.0.0.1: icmp_seq=9 ttl=64 time=0.010 ms
64 bytes from 127.0.0.1: icmp_seq=10 ttl=64 time=0.011 ms

--- 127.0.0.1 ping statistics ---

10 packets transmitted, 10 received, 0% packet loss, time 9ms
rtt min/avg/max/mdev = 0.010/0.014/0.040/0.008 ms

I'm using centos 7.2 with kernel 3.10.

I am very confused about the result. As I understand it, it doesn't
matter how frequently I send packets, each packet's latency should be
the same. So How can I understand it from network stack point of view?

Any thoughts or suggestions would be highly appreciated.


Thanks,
Xiaolong

On 10/2/20 10:51 AM, 叶小龙 wrote:

61 more quoted lines

Many factors in play here.

1) if you keep cpus busy enough, they tend to keep in their caches
the data needed to serve your requests. In your case, time taken to
process an ICMP packet can be very different depending on how hot
cpu caches are.

2) Idle cpus can be put in a power conserving state.
  It takes time to exit from these states, as you noticed.
  These delays can typically be around 50 usec, or more.

Search for cpu C-states , and powertop program.

On Fri, Oct 02, 2020 at 11:26:00AM +0200, Eric Dumazet wrote:

2) Idle cpus can be put in a power conserving state.
  It takes time to exit from these states, as you noticed.
  These delays can typically be around 50 usec, or more.

This is often the case in my experience. I'm even used to starting a
busy loop in SCHED_IDLE prio on certain machines or in certain VMs
just to keep them busy, and to see the ping latency cut in half.

Willy

Hi,

Eric Dumazet [off-list ref] writes:

Many factors in play here.

1) if you keep cpus busy enough, they tend to keep in their caches
the data needed to serve your requests. In your case, time taken to
process an ICMP packet can be very different depending on how hot
cpu caches are.

2) Idle cpus can be put in a power conserving state.
  It takes time to exit from these states, as you noticed.
  These delays can typically be around 50 usec, or more.

Still on the power management theme, in my experience, in addition to
the CPU power states, the PCIe and NIC power management settings also
effect latency on the order of 10-100s usecs when the system is allowed
to go idle, some things that have helped:

 - setting CONFIG_PCIEASPM to performance;
 - disabling EEE (energy efficient ethernet) in your NIC;

Search for cpu C-states , and powertop program.

Cheers,
-- 
Vinicius