[Patch net] inet: fix sleeping inside inet_wait_for_connect()

Subsystems: networking [general], the rest

STALE3563d

5 messages, 4 authors, 2016-11-04 · open the first message on its own page

[Patch net] inet: fix sleeping inside inet_wait_for_connect()

From: Cong Wang <hidden>
Date: 2016-11-01 23:04:47

Andrey reported this kernel warning:

  WARNING: CPU: 0 PID: 4608 at kernel/sched/core.c:7724
  __might_sleep+0x14c/0x1a0 kernel/sched/core.c:7719
  do not call blocking ops when !TASK_RUNNING; state=1 set at
  [<ffffffff811f5a5c>] prepare_to_wait+0xbc/0x210
  kernel/sched/wait.c:178
  Modules linked in:
  CPU: 0 PID: 4608 Comm: syz-executor Not tainted 4.9.0-rc2+ #320
  Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
   ffff88006625f7a0 ffffffff81b46914 ffff88006625f818 0000000000000000
   ffffffff84052960 0000000000000000 ffff88006625f7e8 ffffffff81111237
   ffff88006aceac00 ffffffff00001e2c ffffed000cc4beff ffffffff84052960
  Call Trace:
   [<     inline     >] __dump_stack lib/dump_stack.c:15
   [<ffffffff81b46914>] dump_stack+0xb3/0x10f lib/dump_stack.c:51
   [<ffffffff81111237>] __warn+0x1a7/0x1f0 kernel/panic.c:550
   [<ffffffff8111132c>] warn_slowpath_fmt+0xac/0xd0 kernel/panic.c:565
   [<ffffffff811922fc>] __might_sleep+0x14c/0x1a0 kernel/sched/core.c:7719
   [<     inline     >] slab_pre_alloc_hook mm/slab.h:393
   [<     inline     >] slab_alloc_node mm/slub.c:2634
   [<     inline     >] slab_alloc mm/slub.c:2716
   [<ffffffff81508da0>] __kmalloc_track_caller+0x150/0x2a0 mm/slub.c:4240
   [<ffffffff8146be14>] kmemdup+0x24/0x50 mm/util.c:113
   [<ffffffff8388b2cf>] dccp_feat_clone_sp_val.part.5+0x4f/0xe0 net/dccp/feat.c:374
   [<     inline     >] dccp_feat_clone_sp_val net/dccp/feat.c:1141
   [<     inline     >] dccp_feat_change_recv net/dccp/feat.c:1141
   [<ffffffff8388d491>] dccp_feat_parse_options+0xaa1/0x13d0 net/dccp/feat.c:1411
   [<ffffffff83894f01>] dccp_parse_options+0x721/0x1010 net/dccp/options.c:128
   [<ffffffff83891280>] dccp_rcv_state_process+0x200/0x15b0 net/dccp/input.c:644
   [<ffffffff838b8a94>] dccp_v4_do_rcv+0xf4/0x1a0 net/dccp/ipv4.c:681
   [<     inline     >] sk_backlog_rcv ./include/net/sock.h:872
   [<ffffffff82b7ceb6>] __release_sock+0x126/0x3a0 net/core/sock.c:2044
   [<ffffffff82b7d189>] release_sock+0x59/0x1c0 net/core/sock.c:2502
   [<     inline     >] inet_wait_for_connect net/ipv4/af_inet.c:547
   [<ffffffff8316b2a2>] __inet_stream_connect+0x5d2/0xbb0 net/ipv4/af_inet.c:617
   [<ffffffff8316b8d5>] inet_stream_connect+0x55/0xa0 net/ipv4/af_inet.c:656
   [<ffffffff82b705e4>] SYSC_connect+0x244/0x2f0 net/socket.c:1533
   [<ffffffff82b72dd4>] SyS_connect+0x24/0x30 net/socket.c:1514
   [<ffffffff83fbf701>] entry_SYSCALL_64_fastpath+0x1f/0xc2
  arch/x86/entry/entry_64.S:209

Unlike commit 26cabd31259ba43f68026ce3f62b78094124333f
("sched, net: Clean up sk_wait_event() vs. might_sleep()"), the
sleeping function is called before schedule_timeout(), this is indeed
a bug. Fix this by moving the wait logic to the new API, it is similar
to commit ff960a731788a7408b6f66ec4fd772ff18833211
("netdev, sched/wait: Fix sleeping inside wait event").

Reported-by: Andrey Konovalov <redacted>
Cc: Andrey Konovalov <redacted>
Cc: Eric Dumazet <redacted>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Cong Wang <redacted>
---
 net/ipv4/af_inet.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 9648c97..5ddf5cd 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -533,9 +533,9 @@ EXPORT_SYMBOL(inet_dgram_connect);
 
 static long inet_wait_for_connect(struct sock *sk, long timeo, int writebias)
 {
-	DEFINE_WAIT(wait);
+	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+	add_wait_queue(sk_sleep(sk), &wait);
 	sk->sk_write_pending += writebias;
 
 	/* Basic assumption: if someone sets sk->sk_err, he _must_
@@ -545,13 +545,12 @@ static long inet_wait_for_connect(struct sock *sk, long timeo, int writebias)
 	 */
 	while ((1 << sk->sk_state) & (TCPF_SYN_SENT | TCPF_SYN_RECV)) {
 		release_sock(sk);
-		timeo = schedule_timeout(timeo);
+		timeo = wait_woken(&wait, TASK_INTERRUPTIBLE, timeo);
 		lock_sock(sk);
 		if (signal_pending(current) || !timeo)
 			break;
-		prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
 	}
-	finish_wait(sk_sleep(sk), &wait);
+	remove_wait_queue(sk_sleep(sk), &wait);
 	sk->sk_write_pending -= writebias;
 	return timeo;
 }
-- 
2.1.0

Re: [Patch net] inet: fix sleeping inside inet_wait_for_connect()

From: Eric Dumazet <hidden>
Date: 2016-11-02 01:55:00

On Tue, 2016-11-01 at 16:04 -0700, Cong Wang wrote:
Andrey reported this kernel warning:
Unlike commit 26cabd31259ba43f68026ce3f62b78094124333f
("sched, net: Clean up sk_wait_event() vs. might_sleep()"), the
sleeping function is called before schedule_timeout(), this is indeed
a bug. Fix this by moving the wait logic to the new API, it is similar
to commit ff960a731788a7408b6f66ec4fd772ff18833211
("netdev, sched/wait: Fix sleeping inside wait event").

Reported-by: Andrey Konovalov <redacted>
Cc: Andrey Konovalov <redacted>
Cc: Eric Dumazet <redacted>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Cong Wang <redacted>
---

Excellent.

I guess we could also define sk_wait_event_woken()
and use it instead of sk_wait_event(), and also in
inet_wait_for_connect()

+#define sk_wait_event_woken(__sk, __timeo, __condition, __wait)                \
+       ({      int __rc;                                               \
+               release_sock(__sk);                                     \
+               __rc = __condition;                                     \
+               if (!__rc) {                                            \
+                       *(__timeo) = wait_woken(__wait, TASK_INTERRUPTIBLE, \
+                                               *(__timeo));            \
+               }                                                       \
+               lock_sock(__sk);                                        \
+               __rc = __condition;                                     \
+               __rc;                                                   \
+       })

sk_wait_data() would need :
@@ -2078,14 +2080,14 @@ void __sk_flush_backlog(struct sock *sk)
  */
 int sk_wait_data(struct sock *sk, long *timeo, const struct sk_buff *skb)
 {
+       DEFINE_WAIT_FUNC(wait, woken_wake_function);
        int rc;
-       DEFINE_WAIT(wait);
 
-       prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+       add_wait_queue(sk_sleep(sk), &wait);
        sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
-       rc = sk_wait_event(sk, timeo, skb_peek_tail(&sk->sk_receive_queue) != skb);
+       rc = sk_wait_event_woken(sk, timeo, skb_peek_tail(&sk->sk_receive_queue) != skb, &wait);
        sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
-       finish_wait(sk_sleep(sk), &wait);
+       remove_wait_queue(sk_sleep(sk), &wait);
        return rc;
 }
 EXPORT_SYMBOL(sk_wait_data);

Re: [Patch net] inet: fix sleeping inside inet_wait_for_connect()

From: Cong Wang <hidden>
Date: 2016-11-02 20:01:33

On Tue, Nov 1, 2016 at 6:54 PM, Eric Dumazet [off-list ref] wrote:
On Tue, 2016-11-01 at 16:04 -0700, Cong Wang wrote:
quoted
Andrey reported this kernel warning:
quoted
Unlike commit 26cabd31259ba43f68026ce3f62b78094124333f
("sched, net: Clean up sk_wait_event() vs. might_sleep()"), the
sleeping function is called before schedule_timeout(), this is indeed
a bug. Fix this by moving the wait logic to the new API, it is similar
to commit ff960a731788a7408b6f66ec4fd772ff18833211
("netdev, sched/wait: Fix sleeping inside wait event").

Reported-by: Andrey Konovalov <redacted>
Cc: Andrey Konovalov <redacted>
Cc: Eric Dumazet <redacted>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Cong Wang <redacted>
---

Excellent.

I guess we could also define sk_wait_event_woken()
and use it instead of sk_wait_event(), and also in
inet_wait_for_connect()
Agreed, I will send some followup patches to address this,
probably all release_sock() before a schedule_*() need
to fix.

Thanks!

Re: [Patch net] inet: fix sleeping inside inet_wait_for_connect()

From: David Miller <davem@davemloft.net>
Date: 2016-11-03 19:19:09

From: Cong Wang <redacted>
Date: Tue,  1 Nov 2016 16:04:36 -0700
Andrey reported this kernel warning:
 ...
Unlike commit 26cabd31259ba43f68026ce3f62b78094124333f
("sched, net: Clean up sk_wait_event() vs. might_sleep()"), the
sleeping function is called before schedule_timeout(), this is indeed
a bug. Fix this by moving the wait logic to the new API, it is similar
to commit ff960a731788a7408b6f66ec4fd772ff18833211
("netdev, sched/wait: Fix sleeping inside wait event").

Reported-by: Andrey Konovalov <redacted>
Cc: Andrey Konovalov <redacted>
Cc: Eric Dumazet <redacted>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Cong Wang <redacted>
Applied, thanks.

[lkp] [inet] 34630bc073: netperf.Throughput_tps 4.2% improvement

From: kernel test robot <hidden>
Date: 2016-11-04 01:57:41


Greeting,

FYI, we noticed a 4.2% improvement of netperf.Throughput_tps due to commit:


commit 34630bc07341e30e42a974840a0d622b31f1845c ("inet: fix sleeping inside inet_wait_for_connect()")
https://github.com/0day-ci/linux Cong-Wang/inet-fix-sleeping-inside-inet_wait_for_connect/20161102-070925

in testcase: netperf
on test machine: 8 threads Intel(R) Atom(TM) CPU C2750 @ 2.40GHz with 16G memory
with following parameters:

	ip: ipv4
	runtime: 300s
	nr_threads: 200%
	cluster: cs-localhost
	test: TCP_CRR
	cpufreq_governor: performance

Netperf is a benchmark that can be use to measure various aspect of networking performance.


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
  cs-localhost/gcc-6/performance/ipv4/x86_64-rhel-7.2/200%/debian-x86_64-2016-08-31.cgz/300s/lkp-avoton3/TCP_CRR/netperf

commit: 
  b9b84fc07d ("net: mv643xx_eth: ensure coalesce settings survive read-modify-write")
  34630bc073 ("inet: fix sleeping inside inet_wait_for_connect()")

b9b84fc07dbc6ac7 34630bc07341e30e42a974840a 
---------------- -------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      2915 ±  0%      +4.2%       3039 ±  0%  netperf.Throughput_tps
  20900055 ±  0%     -97.4%     540516 ±  8%  netperf.time.involuntary_context_switches
    517.25 ±  0%     +13.8%     588.50 ±  0%  netperf.time.percent_of_cpu_this_job_got
      1536 ±  0%     +13.9%       1750 ±  0%  netperf.time.system_time
   7199745 ±  1%    +100.6%   14442880 ±  0%  netperf.time.voluntary_context_switches
     18454 ±  1%     +28.3%      23680 ±  4%  softirqs.SCHED
      0.08 ±  0%     +34.4%       0.11 ±  7%  turbostat.CPU%c1
    187728 ±  0%     -44.8%     103607 ±  0%  vmstat.system.cs
      9065 ±  1%     +24.6%      11296 ±  4%  vmstat.system.in
    118058 ± 13%    +582.9%     806220 ± 17%  cpuidle.C1-AVN.time
      2948 ± 28%    +822.4%      27193 ± 17%  cpuidle.C1-AVN.usage
     24.75 ± 16%    +238.4%      83.75 ± 20%  cpuidle.POLL.usage
      2121 ±  6%     -16.2%       1778 ±  6%  slabinfo.cred_jar.num_objs
      5215 ± 23%     +42.3%       7423 ±  3%  slabinfo.kmalloc-32.active_objs
      5215 ± 23%     +42.3%       7423 ±  3%  slabinfo.kmalloc-32.num_objs
 2.561e+10 ±  0%      -1.4%  2.526e+10 ±  0%  perf-stat.branch-misses
      1.27 ±  3%     +14.3%       1.45 ±  2%  perf-stat.cache-miss-rate%
 2.882e+09 ±  3%     +13.4%  3.269e+09 ±  2%  perf-stat.cache-misses
  57816048 ±  0%     -45.0%   31803360 ±  0%  perf-stat.context-switches
      8221 ±  2%     +24.9%      10268 ±  5%  perf-stat.cpu-migrations
 2.315e+10 ±  0%      -1.1%  2.289e+10 ±  0%  perf-stat.iTLB-load-misses
    285020 ±  0%      +4.2%     296904 ±  0%  perf-stat.minor-faults
    285022 ±  0%      +4.2%     296904 ±  0%  perf-stat.page-faults
    231706 ± 10%    +143.8%     564885 ± 13%  sched_debug.cfs_rq:/.MIN_vruntime.avg
    645748 ± 13%     +49.9%     967850 ±  0%  sched_debug.cfs_rq:/.MIN_vruntime.max
    297027 ± 10%     +52.4%     452531 ±  3%  sched_debug.cfs_rq:/.MIN_vruntime.stddev
    231706 ± 10%    +143.8%     564885 ± 13%  sched_debug.cfs_rq:/.max_vruntime.avg
    645748 ± 13%     +49.9%     967851 ±  0%  sched_debug.cfs_rq:/.max_vruntime.max
    297027 ± 10%     +52.4%     452532 ±  3%  sched_debug.cfs_rq:/.max_vruntime.stddev
    769174 ±  0%     +24.3%     955802 ±  0%  sched_debug.cfs_rq:/.min_vruntime.avg
    786747 ±  1%     +23.4%     970602 ±  0%  sched_debug.cfs_rq:/.min_vruntime.max
    758458 ±  0%     +24.1%     941243 ±  0%  sched_debug.cfs_rq:/.min_vruntime.min
      1.23 ±  3%     +24.9%       1.54 ±  2%  sched_debug.cfs_rq:/.nr_running.avg
      1.79 ±  4%     +18.6%       2.12 ±  6%  sched_debug.cfs_rq:/.nr_running.max
      0.41 ±  8%     +21.7%       0.50 ±  5%  sched_debug.cfs_rq:/.nr_running.stddev
    107.00 ±  2%     -20.2%      85.38 ± 10%  sched_debug.cfs_rq:/.runnable_load_avg.min
     11.17 ± 14%     +66.3%      18.58 ± 23%  sched_debug.cfs_rq:/.util_avg.stddev
    390011 ± 29%     -55.3%     174458 ± 12%  sched_debug.cpu.avg_idle.avg
    813413 ± 18%     -65.5%     280413 ± 22%  sched_debug.cpu.avg_idle.max
    257758 ± 22%     -77.5%      58027 ± 35%  sched_debug.cpu.avg_idle.stddev
      3.38 ± 25%     -45.6%       1.84 ± 20%  sched_debug.cpu.clock.stddev
      3.38 ± 25%     -45.6%       1.84 ± 20%  sched_debug.cpu.clock_task.stddev
    115535 ±  4%     +16.2%     134278 ±  7%  sched_debug.cpu.load.min
      0.58 ±  3%     +22.7%       0.71 ±  8%  sched_debug.cpu.nr_running.stddev
   3579054 ±  0%     -45.0%    1966781 ±  0%  sched_debug.cpu.nr_switches.avg
   3645312 ±  0%     -43.6%    2056851 ±  0%  sched_debug.cpu.nr_switches.max
   3525172 ±  0%     -45.9%    1908138 ±  0%  sched_debug.cpu.nr_switches.min
      0.00 ± 18%     -88.7%       0.00 ± 58%  sched_debug.rt_rq:/.rt_time.min
     15.67 ±  0%     -17.4%      12.94 ±  1%  perf-profile.calltrace.cycles-pp.SYSC_recvfrom.sys_recvfrom.entry_SYSCALL_64_fastpath
     16.23 ±  0%     +19.3%      19.36 ±  0%  perf-profile.calltrace.cycles-pp.SYSC_sendto.sys_sendto.entry_SYSCALL_64_fastpath
      2.41 ±  3%     +10.7%       2.67 ±  1%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_close.inet_release.sock_release
      0.69 ±  2%    +137.3%       1.64 ±  3%  perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.inet_sendmsg.sock_sendmsg
      1.17 ±  4%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__inet_stream_connect.inet_stream_connect
      2.06 ±  1%     -13.5%       1.78 ±  2%  perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg
      9.07 ±  0%     -28.3%       6.51 ±  2%  perf-profile.calltrace.cycles-pp.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
      0.82 ±  1%    +109.5%       1.71 ±  4%  perf-profile.calltrace.cycles-pp.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
     12.40 ±  0%     +16.7%      14.46 ±  1%  perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg
      1.94 ±  3%     -25.4%       1.45 ±  1%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish
      2.04 ±  3%     -25.8%       1.51 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      1.78 ±  3%     -14.8%       1.52 ±  1%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function
      1.87 ±  3%     -26.0%       1.39 ±  1%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv
      1.82 ±  3%     -25.9%       1.35 ±  1%  perf-profile.calltrace.cycles-pp.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue
      2.00 ±  3%     -24.8%       1.51 ±  3%  perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.dev_queue_xmit.ip_finish_output2.ip_finish_output
      1.15 ±  5%     -65.9%       0.39 ± 57%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up
      1.57 ±  3%     -14.8%       1.34 ±  1%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function
     14.95 ±  0%     -18.8%      12.14 ±  1%  perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.entry_SYSCALL_64_fastpath
     15.71 ±  0%     +20.0%      18.84 ±  0%  perf-profile.calltrace.cycles-pp.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto.entry_SYSCALL_64_fastpath
      7.37 ±  0%     -17.4%       6.08 ±  1%  perf-profile.calltrace.cycles-pp.ip_local_out.ip_queue_xmit.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check
      7.59 ±  0%     -16.9%       6.31 ±  1%  perf-profile.calltrace.cycles-pp.ip_queue_xmit.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established
      9.50 ±  0%     +13.6%      10.79 ±  1%  perf-profile.calltrace.cycles-pp.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push
      2.53 ±  2%     +10.4%       2.79 ±  1%  perf-profile.calltrace.cycles-pp.release_sock.tcp_close.inet_release.sock_release.sock_close
      0.82 ±  2%    +117.7%       1.79 ±  2%  perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
      1.24 ±  3%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__inet_stream_connect.inet_stream_connect.SYSC_connect
      2.17 ±  1%     -12.8%       1.89 ±  2%  perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg
      1.28 ±  3%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.schedule_timeout.__inet_stream_connect.inet_stream_connect.SYSC_connect.sys_connect
      2.21 ±  1%     -13.0%       1.93 ±  2%  perf-profile.calltrace.cycles-pp.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      0.81 ± 25%     +54.9%       1.25 ±  2%  perf-profile.calltrace.cycles-pp.security_sock_rcv_skb.sk_filter_trim_cap.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      1.19 ± 23%     +53.6%       1.82 ± 14%  perf-profile.calltrace.cycles-pp.sk_filter_trim_cap.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
      2.67 ±  0%     -15.5%       2.26 ±  2%  perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
     15.30 ±  0%     -18.1%      12.53 ±  1%  perf-profile.calltrace.cycles-pp.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.entry_SYSCALL_64_fastpath
     15.96 ±  0%     +19.6%      19.08 ±  0%  perf-profile.calltrace.cycles-pp.sock_sendmsg.SYSC_sendto.sys_sendto.entry_SYSCALL_64_fastpath
     15.77 ±  0%     -17.4%      13.02 ±  1%  perf-profile.calltrace.cycles-pp.sys_recvfrom.entry_SYSCALL_64_fastpath
     16.29 ±  0%     +19.2%      19.41 ±  0%  perf-profile.calltrace.cycles-pp.sys_sendto.entry_SYSCALL_64_fastpath
      1.11 ±  2%    -100.0%       0.00 ± -1%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
      1.81 ±  2%      +6.9%       1.93 ±  4%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      1.86 ±  1%     +34.3%       2.50 ±  1%  perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish
      0.26 ±173%    +211.4%       0.82 ± 37%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock
      0.55 ±  3%    +106.3%       1.14 ±  1%  perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv
      1.12 ±  2%     -19.6%       0.90 ±  6%  perf-profile.calltrace.cycles-pp.tcp_finish_connect.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock
      2.58 ±  2%     -26.3%       1.90 ±  4%  perf-profile.calltrace.cycles-pp.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
     10.67 ±  0%     -24.3%       8.08 ±  2%  perf-profile.calltrace.cycles-pp.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
     12.42 ±  0%     +16.7%      14.49 ±  1%  perf-profile.calltrace.cycles-pp.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
      0.57 ±  1%    +141.7%       1.39 ±  3%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg
      8.66 ±  0%     -11.5%       7.67 ±  1%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg
      3.09 ±  1%     -23.5%       2.36 ±  4%  perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      1.05 ±  2%      +9.1%       1.14 ±  3%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_child_process.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
      2.23 ±  3%      +9.5%       2.44 ±  1%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.tcp_close
     10.91 ±  1%      +9.6%      11.96 ±  1%  perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
     14.75 ±  0%     -19.2%      11.91 ±  1%  perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom
      8.79 ±  0%     -30.0%       6.16 ±  2%  perf-profile.calltrace.cycles-pp.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
      0.78 ±  1%    +110.2%       1.65 ±  4%  perf-profile.calltrace.cycles-pp.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv
     15.33 ±  0%     +20.3%      18.45 ±  0%  perf-profile.calltrace.cycles-pp.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto
      8.73 ±  0%     -20.4%       6.95 ±  1%  perf-profile.calltrace.cycles-pp.tcp_transmit_skb.tcp_send_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv
     10.37 ±  0%     +11.4%      11.56 ±  1%  perf-profile.calltrace.cycles-pp.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg
      2.29 ±  3%     +10.4%       2.53 ±  1%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_close.inet_release
      0.61 ±  2%    +140.6%       1.47 ±  3%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_sendmsg.inet_sendmsg
      7.73 ±  0%     -23.5%       5.91 ±  2%  perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg
      2.06 ±  1%      +7.3%       2.21 ±  3%  perf-profile.calltrace.cycles-pp.tcp_v4_syn_recv_sock.tcp_check_req.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
     12.26 ±  0%     +16.7%      14.31 ±  1%  perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg
      3.04 ±  1%     -12.7%       2.65 ±  0%  perf-profile.calltrace.cycles-pp.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
      2.36 ±  3%     -14.1%       2.02 ±  2%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
     15.69 ±  0%     -17.4%      12.96 ±  1%  perf-profile.children.cycles-pp.SYSC_recvfrom
     16.25 ±  0%     +19.3%      19.38 ±  1%  perf-profile.children.cycles-pp.SYSC_sendto
      1.66 ±  4%     +16.7%       1.93 ±  1%  perf-profile.children.cycles-pp.__kfree_skb
     15.12 ±  1%     +10.6%      16.72 ±  0%  perf-profile.children.cycles-pp.__release_sock
      5.51 ±  1%     -33.1%       3.69 ±  1%  perf-profile.children.cycles-pp.__schedule
     10.11 ±  0%     -18.0%       8.29 ±  2%  perf-profile.children.cycles-pp.__tcp_ack_snd_check
      3.83 ±  1%     -18.0%       3.14 ±  1%  perf-profile.children.cycles-pp.__wake_up_common
      3.63 ±  2%     -12.2%       3.18 ±  1%  perf-profile.children.cycles-pp.__wake_up_sync_key
      1.86 ±  3%     -11.7%       1.64 ±  1%  perf-profile.children.cycles-pp.activate_task
      3.66 ±  1%     -20.5%       2.91 ±  1%  perf-profile.children.cycles-pp.autoremove_wake_function
      3.56 ±  1%     -18.0%       2.92 ±  0%  perf-profile.children.cycles-pp.default_wake_function
      1.21 ±  5%     -10.4%       1.08 ±  1%  perf-profile.children.cycles-pp.enqueue_entity
      1.64 ±  2%     -10.8%       1.46 ±  2%  perf-profile.children.cycles-pp.enqueue_task_fair
     14.98 ±  0%     -18.8%      12.17 ±  1%  perf-profile.children.cycles-pp.inet_recvmsg
     15.72 ±  0%     +20.0%      18.87 ±  0%  perf-profile.children.cycles-pp.inet_sendmsg
      2.03 ±  2%     -50.4%       1.01 ±  5%  perf-profile.children.cycles-pp.pick_next_task_fair
     15.94 ±  1%     +10.2%      17.57 ±  1%  perf-profile.children.cycles-pp.release_sock
      5.86 ±  1%     -33.6%       3.89 ±  0%  perf-profile.children.cycles-pp.schedule
      5.25 ±  1%     -27.3%       3.81 ±  0%  perf-profile.children.cycles-pp.schedule_timeout
      2.68 ±  0%     -15.5%       2.27 ±  2%  perf-profile.children.cycles-pp.sk_wait_data
      1.35 ±  0%     +14.6%       1.55 ±  2%  perf-profile.children.cycles-pp.skb_release_all
     15.32 ±  0%     -18.1%      12.54 ±  1%  perf-profile.children.cycles-pp.sock_recvmsg
     15.97 ±  0%     +19.6%      19.10 ±  0%  perf-profile.children.cycles-pp.sock_sendmsg
     15.79 ±  0%     -17.3%      13.05 ±  1%  perf-profile.children.cycles-pp.sys_recvfrom
     16.32 ±  0%     +19.2%      19.46 ±  0%  perf-profile.children.cycles-pp.sys_sendto
      1.14 ±  2%     -19.3%       0.92 ±  6%  perf-profile.children.cycles-pp.tcp_finish_connect
      2.75 ±  1%     -24.1%       2.08 ±  4%  perf-profile.children.cycles-pp.tcp_prequeue
     10.69 ±  0%     -24.3%       8.09 ±  2%  perf-profile.children.cycles-pp.tcp_prequeue_process
     12.50 ±  0%     +16.6%      14.58 ±  1%  perf-profile.children.cycles-pp.tcp_push
      1.82 ±  3%     +18.1%       2.15 ±  3%  perf-profile.children.cycles-pp.tcp_rearm_rto
     14.78 ±  0%     -19.2%      11.95 ±  1%  perf-profile.children.cycles-pp.tcp_recvmsg
     15.36 ±  0%     +20.3%      18.48 ±  0%  perf-profile.children.cycles-pp.tcp_sendmsg
      2.07 ±  1%      +7.2%       2.22 ±  3%  perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock
      3.43 ±  2%     -17.5%       2.83 ±  0%  perf-profile.children.cycles-pp.try_to_wake_up
      2.46 ±  3%     -11.6%       2.17 ±  1%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.34 ±  3%      +8.4%       1.45 ±  4%  perf-profile.self.cycles-pp.__local_bh_enable_ip
      0.81 ±  6%     +18.8%       0.96 ±  3%  perf-profile.self.cycles-pp.mod_timer



                              perf-stat.context-switches

    6e+07 ++----------------------------------------------------------------+
          |   **.*.*.**.*.*.*.**.*     **.*.*    *.*.*.**.*.*.*.**.*.*.**.*.*
  5.5e+07 ++ +                    +   +      +  +                           |
          *.*                      *.*        **                            |
          |                                                                 |
    5e+07 ++                                                                |
          |                                                                 |
  4.5e+07 ++                                                                |
          |                                                                 |
    4e+07 ++                                                                |
          |                                                                 |
          |                                                                 |
  3.5e+07 ++                                                                |
          O O OO O O OO O O O OO O O O OO O O OO O O O                      |
    3e+07 ++----------------------------------------------------------------+


                              perf-stat.cpu-migrations

  11500 ++----O-------------------------------------------------------------+
        |   O                       OO                                      |
  11000 O+               O               O      O                           |
  10500 ++O      O O      O     O            O      O                       |
        |      O     O        O   O                                         |
  10000 ++                  O          O       O                            |
   9500 ++             O                   O      O                         |
        |                                                                   |
   9000 ++                                                                  |
   8500 ++                                                               .* |
        |       .*.          .*.          .*         .*.      .*. .*. *.*  +|
   8000 ++  *.**   *.*   **.*   *    *.*.*  :   *.*.*   *.**.*   *   *      *
   7500 ++ +          + +        +   :      :   :                           |
        *.*            *          *.*        *.*                            |
   7000 ++------------------------------------------------------------------+


                              netperf.Throughput_tps

  3050 ++------------------------------O-OO---O-O-O-------------------------+
       |        O O O O O O OO O O O O      O       O                       |
  3000 O+O O OO                                                             |
  2950 ++                                                                   |
       |                                                .*.   .*.        .*.|
  2900 ++  *.**.*. .*.*.*.*.**.*     *.*.**     *.*.*.**   *.*   *.*.**.*   *
       |   :      *            :     :    :     :                           |
  2850 ++  :                    :    :     :    :                           |
       |  :                     :   :      :   :                            |
  2800 ++ :                     :   :      :   :                            |
  2750 ++ :                     :   :      :   :                            |
       |  :                      :  :       :  :                            |
  2700 ++:                       : :        : :                             |
       *.*                       *.*        *.*                             |
  2650 ++-------------------------------------------------------------------+


                             netperf.time.system_time

  1800 ++-------------------------------------------------------------------+
       |                                                                    |
  1750 O+O O OO O O O O O O OO O O O O O OO O O O O O                       |
       |                                                                    |
       |                                                                    |
  1700 ++                                                                   |
       |                                                                    |
  1650 ++                                                                   |
       |                                                                    |
  1600 ++                                                                   |
       |                                                                    |
       |                                                                    |
  1550 *+*.*.**.*.*.*.*.*.*.**.*.*.*.*.*.**.*.*.*.*.*.**.*.*.*.*.*.*.**.*.*.*
       |                                                                    |
  1500 ++-------------------------------------------------------------------+


                     netperf.time.percent_of_cpu_this_job_got

  590 O+O-O-O-OO-O-O-O-O-O-O-O-OO-O-O-O-O-O-O-O-O-OO------------------------+
      |                                                                     |
  580 ++                                                                    |
  570 ++                                                                    |
      |                                                                     |
  560 ++                                                                    |
      |                                                                     |
  550 ++                                                                    |
      |                                                                     |
  540 ++                                                                    |
  530 ++                                                                    |
      |                                                                     |
  520 ++        .*.*. .*.*.                                                 |
      *.*.*.*.**     *     *.*.**.*.*.*.*.*.*.*.*.**.*.*.*.*.*.*.*.**.*.*.*.*
  510 ++--------------------------------------------------------------------+


                        netperf.time.voluntary_context_switches

  1.5e+07 ++----------------------------------------------------------------+
          O O OO O O OO O O O OO O O O OO O O OO O O O                      |
  1.4e+07 ++                                                                |
  1.3e+07 ++                                                                |
          |                                                                 |
  1.2e+07 ++                                                                |
  1.1e+07 ++                                                                |
          |                                                                 |
    1e+07 ++                                                                |
    9e+06 ++                                                                |
          |                                                                 |
    8e+06 ++                                                                |
    7e+06 ++ .**.*.*.**.*.*.*.**.*.   .**.*.*.  .*.*.*.**.*.*.*.**.*.*.**.*.*
          *.*                      *.*        **                            |
    6e+06 ++----------------------------------------------------------------+


                       netperf.time.involuntary_context_switches

  2.5e+07 ++----------------------------------------------------------------+
          |                                                                 |
          |  .**.*.*.**.*.*.*.**.*.   .**.*.*.  .*.*.*.**.*.*.*.**.*.*.**.*.*
    2e+07 *+*                      *.*        **                            |
          |                                                                 |
          |                                                                 |
  1.5e+07 ++                                                                |
          |                                                                 |
    1e+07 ++                                                                |
          |                                                                 |
          |                                                                 |
    5e+06 ++                                                                |
          |                                                                 |
          |                                                                 |
        0 O+O-OO-O-O-OO-O-O-O-OO-O-O-O-OO-O-O-OO-O-O-O----------------------+


	[*] bisect-good sample
	[O] bisect-bad  sample



Thanks,
Xiaolong
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help