Re: [linus:master] [net] 5628f3fe3b: sockperf.throughput.UDP.msg_per_sec 20.1% regression
From: Oliver Sang <hidden>
Date: 2025-12-16 02:16:21
Also in:
lkml, oe-lkp
On Mon, Dec 15, 2025 at 10:47:04AM +0800, Jason Xing wrote:
Hi Oliver, On Thu, Dec 11, 2025 at 10:31 PM kernel test robot [off-list ref] wrote:quoted
Hello, kernel test robot noticed a 20.1% regression of sockperf.throughput.UDP.msg_per_sec on:What kernel version/commit did you use to test as the baseline?
our report is based on bisect results. so the 20.1% regression is the data from
5628f3fe3b comparing to its parent commit:
844c9db7f7 ("net: use llist for sd->defer_list")
quoted
commit: 5628f3fe3b16114e8424bbfcf0594caef8958a06 ("net: add NUMA awareness to skb_attempt_defer_free()") https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master [still regression on linus/master cb015814f8b6eebcbb8e46e111d108892c5e6821] [still regression on linux-next/master c75caf76ed86bbc15a72808f48f8df1608a0886c] testcase: sockperf config: x86_64-rhel-9.4 compiler: gcc-14 test machine: 256 threads 2 sockets GENUINE INTEL(R) XEON(R) (Sierra Forest) with 128G memory parameters: runtime: 600s cluster: cs-localhost msg_size: 1472b cpufreq_governor: performance In addition to that, the commit also has significant impact on the following tests: +------------------+---------------------------------------------------------------------------------+ | testcase: change | netperf: netperf.Throughput_Mbps 6.0% regression | | test machine | 256 threads 2 sockets GENUINE INTEL(R) XEON(R) (Sierra Forest) with 128G memory | | test parameters | cluster=cs-localhost | | | cpufreq_governor=performance | | | ip=ipv4 | | | nr_threads=1 | | | runtime=300s | | | test=UDP_STREAM | +------------------+---------------------------------------------------------------------------------+ If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags | Reported-by: kernel test robot [off-list ref] | Closes: https://lore.kernel.org/oe-lkp/202512112119.5b9829a-lkp@intel.com (local) Details are as below: --------------------------------------------------------------------------------------------------> The kernel config and materials to reproduce are available at: https://download.01.org/0day-ci/archive/20251211/202512112119.5b9829a-lkp@intel.com ========================================================================================= cluster/compiler/cpufreq_governor/kconfig/msg_size/rootfs/runtime/tbox_group/testcase: cs-localhost/gcc-14/performance/x86_64-rhel-9.4/1472b/debian-13-x86_64-20250902.cgz/600s/lkp-srf-2sp1/sockperf commit: 844c9db7f7 ("net: use llist for sd->defer_list") 5628f3fe3b ("net: add NUMA awareness to skb_attempt_defer_free()")At the first glance, I cannot easily see why those two patches cause the regression.
844c9db7f7 is good base. 5628f3fe3b introduces regression in our tests.
But from what I've know, the commit e20dfbad8aab2
("net: fix napi_consume_skb() with alien skbs") and commit
21664814b89e ("net: use napi_skb_cache even in process context")
altogether can lead to the similar regression with only one sender in
xsk scenario.
My understanding is the lower numbers that have a strong relationship
with commits that I provide? I'm not so sure that they are the same
thing, even though UDP is quite like xsk copy mode. So, Oliver, could
you also launch some experiments just around those two commits?when we finish the bisect, we also tested the tip of mainline at that time [still regression on linus/master cb015814f8b6eebcbb8e46e111d108892c5e6821] this commit has the similar data to 5628f3fe3b, so it seems neither e20dfbad8aab2 or 21664814b89e introduces further regression.
Thanks, Jasonquoted
844c9db7f7f5fe1b 5628f3fe3b16114e8424bbfcf05 ---------------- --------------------------- %stddev %change %stddev \ | \ 0.10 +0.0 0.12 mpstat.cpu.all.soft% 16915992 ą 6% -16.0% 14216230 ą 6% turbostat.IRQ 207555 +4.0% 215887 vmstat.system.cs 28663 ą 6% -15.2% 24304 ą 6% vmstat.system.in 11048577 -1.6% 10873254 proc-vmstat.numa_hit 10782437 -1.6% 10607299 proc-vmstat.numa_local 75337098 -1.9% 73920567 proc-vmstat.pgalloc_normal 75257799 -1.9% 73843141 proc-vmstat.pgfree 2564 -2.0% 2514 sockperf.throughput.TCP.BandWidth_MBps 1826630 -2.0% 1790940 sockperf.throughput.TCP.msg_per_sec 941.44 -20.1% 751.96 sockperf.throughput.UDP.BandWidth_MBps 670636 -20.1% 535658 sockperf.throughput.UDP.msg_per_sec 6259 ą 3% +44.0% 9013 ą 12% sockperf.time.involuntary_context_switches 66.00 -3.0% 64.00 sockperf.time.percent_of_cpu_this_job_got 7.698e+08 +3.3% 7.955e+08 perf-stat.i.branch-instructions 208310 +4.0% 216675 perf-stat.i.context-switches 298.79 +6.0% 316.62 perf-stat.i.cpu-migrations 3.92e+09 +4.1% 4.081e+09 perf-stat.i.instructions 0.79 +3.9% 0.83 perf-stat.i.ipc 0.68 ą 3% +7.5% 0.73 perf-stat.i.metric.K/sec 0.11 -4.4% 0.10 ą 2% perf-stat.overall.MPKI 1.19 -3.7% 1.14 perf-stat.overall.cpi 0.84 +3.8% 0.87 perf-stat.overall.ipc 7.681e+08 +3.3% 7.937e+08 perf-stat.ps.branch-instructions 207867 +4.0% 216199 perf-stat.ps.context-switches 298.39 +6.0% 316.29 perf-stat.ps.cpu-migrations 3.911e+09 +4.1% 4.072e+09 perf-stat.ps.instructions 2.419e+12 +4.1% 2.519e+12 perf-stat.total.instructions 1.51 ą 5% -0.5 1.06 ą 26% perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb 1.46 ą 5% -0.4 1.03 ą 26% perf-profile.calltrace.cycles-pp.xmit_one.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_output 1.26 ą 7% -0.4 0.88 ą 28% perf-profile.calltrace.cycles-pp.loopback_xmit.xmit_one.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 6.40 ą 4% -0.3 6.06 ą 2% perf-profile.calltrace.cycles-pp.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 6.42 ą 4% -0.3 6.09 ą 2% perf-profile.calltrace.cycles-pp.inet6_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 1.06 ą 17% -0.3 0.75 ą 31% perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg 2.37 ą 6% -0.3 2.06 ą 3% perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.udpv6_sendmsg.__sys_sendto.__x64_sys_sendto 2.54 ą 9% -0.3 2.25 ą 4% perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg.inet6_recvmsg 0.69 ą 2% -0.1 0.63 ą 5% perf-profile.calltrace.cycles-pp.__wrgsbase_inactive 6.90 ą 4% +0.8 7.66 ą 3% perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 9.88 ą 2% +0.8 10.64 ą 4% perf-profile.calltrace.cycles-pp.udp_sendmsg.udpv6_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 9.94 ą 2% +0.8 10.72 ą 4% perf-profile.calltrace.cycles-pp.udpv6_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.77 ą 4% +0.8 7.58 ą 3% perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.__sys_sendto.__x64_sys_sendto 6.65 ą 4% +0.8 7.49 ą 3% perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.__sys_sendto 6.77 ą 2% +1.0 7.77 ą 5% perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.udpv6_sendmsg.__sys_sendto.__x64_sys_sendto 6.68 ą 2% +1.0 7.69 ą 5% perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.udpv6_sendmsg.__sys_sendto 6.58 ą 2% +1.0 7.59 ą 5% perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.udpv6_sendmsg 20.94 +1.3 22.20 ą 3% perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 20.92 +1.3 22.19 ą 3% perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.07 ą 2% +1.9 14.99 ą 4% perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg 12.72 ą 2% +2.0 14.69 ą 4% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb 10.50 ą 2% +2.3 12.85 ą 4% perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb 10.39 ą 2% +2.4 12.76 ą 4% perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_output 10.29 ą 2% +2.4 12.70 ą 4% perf-profile.calltrace.cycles-pp.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2 9.77 ą 2% +2.5 12.22 ą 4% perf-profile.calltrace.cycles-pp.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit 0.00 +4.1 4.08 ą 5% perf-profile.calltrace.cycles-pp.skb_defer_free_flush.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip 4.79 ą 3% -0.5 4.24 ą 5% perf-profile.children.cycles-pp.ip_make_skb 6.40 ą 4% -0.3 6.06 ą 2% perf-profile.children.cycles-pp.udpv6_recvmsg 6.43 ą 4% -0.3 6.09 ą 2% perf-profile.children.cycles-pp.inet6_recvmsg 1.51 ą 5% -0.3 1.22 ą 7% perf-profile.children.cycles-pp.dev_hard_start_xmit 1.46 ą 5% -0.3 1.18 ą 7% perf-profile.children.cycles-pp.xmit_one 0.95 ą 6% -0.2 0.77 ą 11% perf-profile.children.cycles-pp.sock_wfree 3.02 ą 2% -0.2 2.85 ą 2% perf-profile.children.cycles-pp.__ip_append_data 1.26 ą 7% -0.2 1.11 ą 7% perf-profile.children.cycles-pp.loopback_xmit 0.20 ą 11% -0.1 0.10 ą 11% perf-profile.children.cycles-pp.__enqueue_entity 0.52 ą 3% -0.1 0.43 ą 11% perf-profile.children.cycles-pp.arch_exit_to_user_mode_prepare 0.22 ą 7% -0.1 0.12 ą 21% perf-profile.children.cycles-pp.switch_fpu_return 0.22 ą 12% -0.1 0.13 ą 12% perf-profile.children.cycles-pp.ip_setup_cork 0.09 ą 11% -0.1 0.02 ą 99% perf-profile.children.cycles-pp.tick_nohz_tick_stopped 0.75 ą 2% -0.1 0.69 ą 4% perf-profile.children.cycles-pp.__wrgsbase_inactive 0.26 ą 13% -0.1 0.20 ą 17% perf-profile.children.cycles-pp.udp4_lib_lookup2 0.08 ą 20% -0.1 0.04 ą 71% perf-profile.children.cycles-pp.__ip_finish_output 0.25 ą 11% -0.0 0.20 ą 15% perf-profile.children.cycles-pp.__netif_receive_skb_core 0.17 ą 13% -0.0 0.13 ą 9% perf-profile.children.cycles-pp.ipv4_mtu 0.21 ą 11% -0.0 0.16 ą 10% perf-profile.children.cycles-pp.__mkroute_output 0.09 ą 15% -0.0 0.05 ą 52% perf-profile.children.cycles-pp.vruntime_eligible 0.10 ą 9% -0.0 0.07 ą 12% perf-profile.children.cycles-pp.update_min_vruntime 0.16 ą 12% +0.0 0.20 ą 8% perf-profile.children.cycles-pp.wakeup_preempt 0.01 ą223% +0.1 0.10 ą 15% perf-profile.children.cycles-pp.tick_nohz_get_next_hrtimer 9.94 ą 2% +0.8 10.72 ą 4% perf-profile.children.cycles-pp.udpv6_sendmsg 0.10 ą 11% +0.9 1.01 ą 6% perf-profile.children.cycles-pp._find_next_bit 20.93 +1.3 22.20 ą 3% perf-profile.children.cycles-pp.__sys_sendto 20.94 +1.3 22.21 ą 3% perf-profile.children.cycles-pp.__x64_sys_sendto 19.98 +1.3 21.26 ą 4% perf-profile.children.cycles-pp.udp_sendmsg 13.68 +1.7 15.42 ą 4% perf-profile.children.cycles-pp.udp_send_skb 13.46 ą 2% +1.8 15.28 ą 4% perf-profile.children.cycles-pp.ip_send_skb 13.28 ą 2% +1.8 15.13 ą 4% perf-profile.children.cycles-pp.ip_output 13.13 ą 2% +1.9 15.04 ą 4% perf-profile.children.cycles-pp.ip_finish_output2 12.78 ą 2% +2.0 14.74 ą 4% perf-profile.children.cycles-pp.__dev_queue_xmit 10.71 ą 3% +2.3 13.04 ą 4% perf-profile.children.cycles-pp.__local_bh_enable_ip 9.84 ą 3% +2.4 12.28 ą 4% perf-profile.children.cycles-pp.net_rx_action 13.63 +2.5 16.11 ą 3% perf-profile.children.cycles-pp.do_softirq 14.05 +2.5 16.57 ą 3% perf-profile.children.cycles-pp.handle_softirqs 0.00 +4.1 4.09 ą 5% perf-profile.children.cycles-pp.skb_defer_free_flush 0.98 ą 13% -0.9 0.08 ą 14% perf-profile.self.cycles-pp.net_rx_action 0.94 ą 7% -0.2 0.76 ą 12% perf-profile.self.cycles-pp.sock_wfree 0.20 ą 16% -0.1 0.06 ą 46% perf-profile.self.cycles-pp.xmit_one 0.70 ą 10% -0.1 0.58 ą 5% perf-profile.self.cycles-pp.menu_select 0.20 ą 6% -0.1 0.11 ą 20% perf-profile.self.cycles-pp.switch_fpu_return 0.18 ą 9% -0.1 0.10 ą 12% perf-profile.self.cycles-pp.__enqueue_entity 0.70 ą 9% -0.1 0.62 ą 5% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.43 ą 10% -0.1 0.36 ą 7% perf-profile.self.cycles-pp.__ip_append_data 0.30 ą 11% -0.1 0.22 ą 8% perf-profile.self.cycles-pp.dequeue_entity 0.35 ą 10% -0.1 0.28 ą 11% perf-profile.self.cycles-pp.sched_balance_newidle 0.13 ą 5% -0.1 0.06 ą 48% perf-profile.self.cycles-pp.__udp4_lib_rcv 0.41 ą 8% -0.1 0.35 ą 9% perf-profile.self.cycles-pp.cpuidle_idle_call 0.29 ą 15% -0.1 0.22 ą 4% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.71 -0.1 0.65 ą 4% perf-profile.self.cycles-pp.__wrgsbase_inactive 0.21 ą 13% -0.1 0.15 ą 15% perf-profile.self.cycles-pp.handle_softirqs 0.84 ą 2% -0.0 0.79 ą 2% perf-profile.self.cycles-pp.__switch_to 0.25 ą 11% -0.0 0.20 ą 15% perf-profile.self.cycles-pp.__netif_receive_skb_core 0.18 ą 21% -0.0 0.13 ą 10% perf-profile.self.cycles-pp.udp_send_skb 0.09 ą 12% -0.0 0.04 ą 75% perf-profile.self.cycles-pp.vruntime_eligible 0.16 ą 14% -0.0 0.12 ą 10% perf-profile.self.cycles-pp.ipv4_mtu 0.07 ą 24% -0.0 0.04 ą 45% perf-profile.self.cycles-pp.sock_recvmsg 0.24 ą 11% +0.1 0.29 ą 3% perf-profile.self.cycles-pp.udp_sendmsg 0.08 ą 16% +0.1 0.15 ą 9% perf-profile.self.cycles-pp.ttwu_do_activate 0.01 ą223% +0.1 0.10 ą 15% perf-profile.self.cycles-pp.tick_nohz_get_next_hrtimer 0.10 ą 15% +0.9 0.98 ą 6% perf-profile.self.cycles-pp._find_next_bit 0.00 +2.9 2.88 ą 7% perf-profile.self.cycles-pp.skb_defer_free_flush *************************************************************************************************** lkp-srf-2sp1: 256 threads 2 sockets GENUINE INTEL(R) XEON(R) (Sierra Forest) with 128G memory ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: cs-localhost/gcc-14/performance/ipv4/x86_64-rhel-9.4/1/debian-13-x86_64-20250902.cgz/300s/lkp-srf-2sp1/UDP_STREAM/netperf commit: 844c9db7f7 ("net: use llist for sd->defer_list") 5628f3fe3b ("net: add NUMA awareness to skb_attempt_defer_free()") 844c9db7f7f5fe1b 5628f3fe3b16114e8424bbfcf05 ---------------- --------------------------- %stddev %change %stddev \ | \ 20600541 ą 3% +14.1% 23500650 ą 2% cpuidle..usage 214491 ą 5% +11.5% 239155 ą 2% meminfo.Shmem 0.11 +0.0 0.13 ą 2% mpstat.cpu.all.soft% 107642 ą 4% +17.1% 126046 ą 3% vmstat.system.cs 0.04 ą 8% +0.0 0.07 turbostat.C1% 0.46 +9.8% 0.50 turbostat.IPC 4284 ą 13% +28.8% 5519 ą 10% numa-meminfo.node0.PageTables 5684 ą 10% -21.7% 4451 ą 11% numa-meminfo.node1.PageTables 208697 ą 5% +11.0% 231719 ą 2% numa-meminfo.node1.Shmem 1073 ą 13% +28.9% 1383 ą 9% numa-vmstat.node0.nr_page_table_pages 1423 ą 10% -21.7% 1114 ą 11% numa-vmstat.node1.nr_page_table_pages 52145 ą 5% +11.0% 57900 ą 2% numa-vmstat.node1.nr_shmem 12387 ą 4% -12.9% 10792 ą 4% sched_debug.cfs_rq:/.avg_vruntime.avg 12387 ą 4% -12.9% 10792 ą 4% sched_debug.cfs_rq:/.min_vruntime.avg 64470 ą 3% +16.0% 74816 ą 3% sched_debug.cpu.nr_switches.avg 8.08 ą 5% -12.6% 7.07 ą 3% perf-sched.total_wait_and_delay.average.ms 197564 ą 5% +13.5% 224145 ą 2% perf-sched.total_wait_and_delay.count.ms 8.08 ą 5% -12.6% 7.06 ą 3% perf-sched.total_wait_time.average.ms 8.08 ą 5% -12.6% 7.07 ą 3% perf-sched.wait_and_delay.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown] 197564 ą 5% +13.5% 224145 ą 2% perf-sched.wait_and_delay.count.[unknown].[unknown].[unknown].[unknown].[unknown] 8.08 ą 5% -12.6% 7.06 ą 3% perf-sched.wait_time.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown] 224037 +3.1% 230892 proc-vmstat.nr_active_anon 53676 ą 4% +11.5% 59861 ą 2% proc-vmstat.nr_shmem 224037 +3.1% 230892 proc-vmstat.nr_zone_active_anon 1.129e+08 -5.9% 1.063e+08 proc-vmstat.numa_hit 1.127e+08 -5.9% 1.06e+08 proc-vmstat.numa_local 8.928e+08 -6.0% 8.394e+08 proc-vmstat.pgalloc_normal 8.927e+08 -6.0% 8.393e+08 proc-vmstat.pgfree 194605 -6.0% 182934 netperf.ThroughputBoth_Mbps 194605 -6.0% 182934 netperf.ThroughputBoth_total_Mbps 97220 -6.0% 91387 netperf.ThroughputRecv_Mbps 97220 -6.0% 91387 netperf.ThroughputRecv_total_Mbps 97384 -6.0% 91547 netperf.Throughput_Mbps 97384 -6.0% 91547 netperf.Throughput_total_Mbps 78.67 -6.8% 73.33 netperf.time.percent_of_cpu_this_job_got 232.75 -6.6% 217.32 netperf.time.system_time 1.114e+08 -6.0% 1.047e+08 netperf.workload 5.854e+08 +7.9% 6.313e+08 perf-stat.i.branch-instructions 0.60 -0.0 0.57 perf-stat.i.branch-miss-rate% 3570607 +2.7% 3668021 perf-stat.i.branch-misses 108376 ą 4% +17.1% 126925 ą 3% perf-stat.i.context-switches 2.29 -9.3% 2.08 perf-stat.i.cpi 3.086e+09 +8.7% 3.353e+09 perf-stat.i.instructions 0.44 +10.2% 0.49 perf-stat.i.ipc 0.61 -0.0 0.58 perf-stat.overall.branch-miss-rate% 2.24 -9.1% 2.04 perf-stat.overall.cpi 0.45 +10.0% 0.49 perf-stat.overall.ipc 8344 +15.6% 9643 perf-stat.overall.path-length 5.833e+08 +7.9% 6.292e+08 perf-stat.ps.branch-instructions 3558414 +2.8% 3656824 perf-stat.ps.branch-misses 108014 ą 4% +17.1% 126500 ą 3% perf-stat.ps.context-switches 3.075e+09 +8.7% 3.341e+09 perf-stat.ps.instructions 9.295e+11 +8.6% 1.01e+12 perf-stat.total.instructions 29.40 -1.8 27.59 perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 28.64 -1.8 26.86 perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.__sys_sendto.__x64_sys_sendto 25.28 -1.7 23.60 perf-profile.calltrace.cycles-pp.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg.__sys_sendto 25.91 -1.6 24.27 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.udp_recvmsg.inet_recvmsg 33.24 -1.6 31.66 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv_omni.process_requests 33.22 -1.6 31.64 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv_omni 31.95 -1.5 30.42 perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 33.73 -1.5 32.21 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv_omni.process_requests.spawn_child.accept_connection 31.86 -1.5 30.35 perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 33.68 -1.5 32.17 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv_omni.process_requests.spawn_child 31.84 -1.5 30.33 perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 24.73 -1.5 23.26 perf-profile.calltrace.cycles-pp._copy_from_iter.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg 26.98 -1.5 25.52 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom 26.96 -1.5 25.50 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.udp_recvmsg.inet_recvmsg.sock_recvmsg 35.15 -1.4 33.74 perf-profile.calltrace.cycles-pp.recv_omni.process_requests.spawn_child.accept_connection.accept_connections 0.66 ą 5% -0.3 0.35 ą 71% perf-profile.calltrace.cycles-pp.skb_attempt_defer_free.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom 0.71 ą 5% +0.2 0.91 ą 4% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry 1.06 ą 9% +0.2 1.26 ą 6% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp 1.09 ą 9% +0.2 1.29 ą 6% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 1.14 ą 8% +0.2 1.35 ą 6% perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg 1.28 ą 7% +0.3 1.56 ą 7% perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg.inet_recvmsg.sock_recvmsg 0.98 ą 13% +0.4 1.36 ą 8% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary 1.36 ą 5% +0.4 1.80 ą 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.sock_def_readable 1.36 ą 5% +0.4 1.80 ą 4% perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.sock_def_readable.__udp_enqueue_schedule_skb 0.17 ą141% +0.5 0.64 ą 7% perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key 4.74 ą 3% +0.5 5.22 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip 4.72 ą 3% +0.5 5.21 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.handle_softirqs.do_softirq 1.38 ą 8% +0.5 1.88 ą 6% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.common_startup_64 0.00 +0.5 0.52 ą 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.schedule_timeout 4.49 ą 3% +0.5 5.02 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.handle_softirqs 3.89 ą 3% +0.5 4.42 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.ip_local_deliver.__netif_receive_skb_one_core.process_backlog.__napi_poll 1.73 ą 6% +0.5 2.27 ą 4% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_sync_key.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb 3.88 ą 3% +0.5 4.41 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.__netif_receive_skb_one_core.process_backlog 3.90 ą 3% +0.5 4.44 perf-profile.calltrace.cycles-pp.ip_local_deliver.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 0.00 +0.5 0.54 perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.schedule_timeout.__skb_wait_for_more_packets 3.82 ą 3% +0.5 4.36 perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver.__netif_receive_skb_one_core 3.39 ą 4% +0.5 3.94 ą 2% perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 3.40 ą 4% +0.6 3.95 ą 2% perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.ip_local_deliver 1.77 ą 6% +0.6 2.33 ą 4% perf-profile.calltrace.cycles-pp.__wake_up_sync_key.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb 0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key 3.15 ą 4% +0.6 3.74 perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu 0.00 +0.6 0.62 ą 3% perf-profile.calltrace.cycles-pp._find_next_bit.skb_defer_free_flush.net_rx_action.handle_softirqs.do_softirq 2.16 ą 5% +0.6 2.80 ą 3% perf-profile.calltrace.cycles-pp.sock_def_readable.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv 0.00 +0.8 0.76 ą 3% perf-profile.calltrace.cycles-pp.__free_frozen_pages.skb_release_data.napi_consume_skb.skb_defer_free_flush.net_rx_action 6.82 ą 3% +0.8 7.65 ą 2% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.00 +1.2 1.24 ą 4% perf-profile.calltrace.cycles-pp.skb_release_data.napi_consume_skb.skb_defer_free_flush.net_rx_action.handle_softirqs 0.00 +1.3 1.27 ą 4% perf-profile.calltrace.cycles-pp.napi_consume_skb.skb_defer_free_flush.net_rx_action.handle_softirqs.do_softirq 15.03 ą 2% +1.4 16.40 ą 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64 15.04 ą 2% +1.4 16.42 ą 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64 15.05 ą 2% +1.4 16.42 ą 2% perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64 10.13 +2.3 12.44 perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 10.02 +2.3 12.34 perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.__sys_sendto.__x64_sys_sendto 9.73 +2.3 12.07 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg 9.79 +2.3 12.13 perf-profile.calltrace.cycles-pp.ip_output.ip_send_skb.udp_send_skb.udp_sendmsg.__sys_sendto 9.39 ą 2% +2.4 11.76 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb.udp_send_skb 8.41 ą 2% +2.5 10.88 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_output 8.36 ą 2% +2.5 10.82 perf-profile.calltrace.cycles-pp.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2 8.51 ą 2% +2.5 10.98 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_output.ip_send_skb 7.91 ą 2% +2.5 10.45 perf-profile.calltrace.cycles-pp.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit 0.00 +5.1 5.13 ą 2% perf-profile.calltrace.cycles-pp.skb_defer_free_flush.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip 29.41 -1.8 27.59 perf-profile.children.cycles-pp.ip_make_skb 28.64 -1.8 26.87 perf-profile.children.cycles-pp.__ip_append_data 25.28 -1.7 23.61 perf-profile.children.cycles-pp.ip_generic_getfrag 25.95 -1.6 24.31 perf-profile.children.cycles-pp._copy_to_iter 33.23 -1.6 31.65 perf-profile.children.cycles-pp.__sys_recvfrom 33.24 -1.6 31.66 perf-profile.children.cycles-pp.__x64_sys_recvfrom 31.95 -1.5 30.42 perf-profile.children.cycles-pp.sock_recvmsg 31.86 -1.5 30.35 perf-profile.children.cycles-pp.inet_recvmsg 31.84 -1.5 30.34 perf-profile.children.cycles-pp.udp_recvmsg 24.78 -1.5 23.29 perf-profile.children.cycles-pp._copy_from_iter 26.98 -1.5 25.52 perf-profile.children.cycles-pp.skb_copy_datagram_iter 26.96 -1.5 25.51 perf-profile.children.cycles-pp.__skb_datagram_iter 35.15 -1.4 33.74 perf-profile.children.cycles-pp.accept_connection 35.15 -1.4 33.74 perf-profile.children.cycles-pp.accept_connections 35.15 -1.4 33.74 perf-profile.children.cycles-pp.process_requests 35.15 -1.4 33.74 perf-profile.children.cycles-pp.spawn_child 35.15 -1.4 33.74 perf-profile.children.cycles-pp.recv_omni 76.98 -1.1 75.86 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 76.90 -1.1 75.78 perf-profile.children.cycles-pp.do_syscall_64 0.65 ą 7% -0.2 0.47 ą 6% perf-profile.children.cycles-pp.__virt_addr_valid 0.66 ą 5% -0.1 0.51 ą 8% perf-profile.children.cycles-pp.skb_attempt_defer_free 0.94 ą 6% -0.1 0.81 ą 3% perf-profile.children.cycles-pp.__check_object_size 0.78 ą 6% -0.1 0.66 ą 4% perf-profile.children.cycles-pp.check_heap_object 0.44 ą 3% -0.1 0.38 ą 6% perf-profile.children.cycles-pp.sched_clock 0.47 ą 3% -0.1 0.41 ą 6% perf-profile.children.cycles-pp.sched_clock_cpu 0.32 ą 8% -0.0 0.27 ą 3% perf-profile.children.cycles-pp.tick_nohz_next_event 0.20 ą 13% -0.0 0.16 ą 7% perf-profile.children.cycles-pp.__get_next_timer_interrupt 0.06 ą 7% +0.0 0.09 ą 10% perf-profile.children.cycles-pp.udp4_csum_init 0.07 ą 14% +0.0 0.10 ą 11% perf-profile.children.cycles-pp.prepare_to_wait_exclusive 0.08 ą 17% +0.0 0.11 ą 7% perf-profile.children.cycles-pp.update_cfs_group 0.23 ą 7% +0.0 0.27 ą 8% perf-profile.children.cycles-pp.__rseq_handle_notify_resume 0.09 ą 19% +0.0 0.13 ą 14% perf-profile.children.cycles-pp.available_idle_cpu 0.37 ą 6% +0.0 0.41 ą 6% perf-profile.children.cycles-pp.enqueue_task_fair 0.30 ą 8% +0.0 0.34 ą 7% perf-profile.children.cycles-pp.enqueue_entity 0.39 ą 7% +0.0 0.44 ą 5% perf-profile.children.cycles-pp.enqueue_task 0.26 ą 8% +0.1 0.31 ą 5% perf-profile.children.cycles-pp.update_curr 0.32 ą 6% +0.1 0.38 ą 8% perf-profile.children.cycles-pp.exit_to_user_mode_loop 0.37 ą 4% +0.1 0.42 ą 6% perf-profile.children.cycles-pp.ttwu_do_activate 0.34 ą 6% +0.1 0.41 ą 4% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.27 ą 9% +0.1 0.35 ą 4% perf-profile.children.cycles-pp.cpus_share_cache 0.47 ą 6% +0.1 0.56 ą 3% perf-profile.children.cycles-pp.dequeue_entities 0.42 ą 5% +0.1 0.51 ą 2% perf-profile.children.cycles-pp.dequeue_entity 0.48 ą 6% +0.1 0.57 ą 3% perf-profile.children.cycles-pp.dequeue_task_fair 0.48 ą 5% +0.1 0.57 ą 2% perf-profile.children.cycles-pp.try_to_block_task 0.43 ą 4% +0.1 0.53 ą 6% perf-profile.children.cycles-pp.call_function_single_prep_ipi 0.39 ą 3% +0.1 0.49 ą 3% perf-profile.children.cycles-pp.os_xsave 0.46 ą 4% +0.1 0.58 ą 6% perf-profile.children.cycles-pp.__smp_call_single_queue 0.48 ą 8% +0.1 0.60 ą 7% perf-profile.children.cycles-pp.select_task_rq_fair 0.37 ą 9% +0.1 0.48 ą 5% perf-profile.children.cycles-pp.select_idle_sibling 0.43 ą 8% +0.1 0.58 ą 2% perf-profile.children.cycles-pp.select_task_rq 0.50 ą 3% +0.2 0.66 ą 6% perf-profile.children.cycles-pp.ttwu_queue_wakelist 1.58 ą 6% +0.2 1.77 ą 4% perf-profile.children.cycles-pp.schedule 0.73 ą 5% +0.2 0.92 ą 4% perf-profile.children.cycles-pp.sched_ttwu_pending 1.18 ą 8% +0.2 1.39 ą 6% perf-profile.children.cycles-pp.schedule_timeout 1.28 ą 7% +0.3 1.56 ą 7% perf-profile.children.cycles-pp.__skb_wait_for_more_packets 2.25 ą 7% +0.4 2.60 ą 4% perf-profile.children.cycles-pp.__schedule 1.04 ą 12% +0.4 1.43 ą 8% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 1.38 ą 5% +0.4 1.81 ą 4% perf-profile.children.cycles-pp.autoremove_wake_function 1.42 ą 5% +0.4 1.86 ą 4% perf-profile.children.cycles-pp.try_to_wake_up 4.72 ą 3% +0.5 5.21 perf-profile.children.cycles-pp.process_backlog 4.74 ą 3% +0.5 5.22 perf-profile.children.cycles-pp.__napi_poll 1.76 ą 6% +0.5 2.28 ą 4% perf-profile.children.cycles-pp.__wake_up_common 4.49 ą 3% +0.5 5.02 perf-profile.children.cycles-pp.__netif_receive_skb_one_core 3.89 ą 3% +0.5 4.42 perf-profile.children.cycles-pp.ip_local_deliver_finish 3.90 ą 3% +0.5 4.44 perf-profile.children.cycles-pp.ip_local_deliver 3.88 ą 3% +0.5 4.41 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 3.82 ą 3% +0.5 4.37 perf-profile.children.cycles-pp.__udp4_lib_rcv 3.40 ą 4% +0.5 3.95 ą 2% perf-profile.children.cycles-pp.udp_unicast_rcv_skb 1.79 ą 6% +0.6 2.34 ą 4% perf-profile.children.cycles-pp.__wake_up_sync_key 3.39 ą 4% +0.6 3.94 ą 2% perf-profile.children.cycles-pp.udp_queue_rcv_one_skb 3.16 ą 4% +0.6 3.75 perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb 0.14 ą 17% +0.6 0.77 ą 6% perf-profile.children.cycles-pp._find_next_bit 2.17 ą 5% +0.6 2.82 ą 3% perf-profile.children.cycles-pp.sock_def_readable 7.10 ą 3% +0.8 7.90 ą 2% perf-profile.children.cycles-pp.intel_idle 15.05 ą 2% +1.4 16.42 ą 2% perf-profile.children.cycles-pp.start_secondary 12.18 +2.2 14.38 perf-profile.children.cycles-pp.handle_softirqs 11.50 +2.3 13.75 perf-profile.children.cycles-pp.do_softirq 10.13 +2.3 12.44 perf-profile.children.cycles-pp.udp_send_skb 10.02 +2.3 12.34 perf-profile.children.cycles-pp.ip_send_skb 9.73 +2.3 12.07 perf-profile.children.cycles-pp.ip_finish_output2 9.79 +2.3 12.13 perf-profile.children.cycles-pp.ip_output 9.40 ą 2% +2.4 11.76 perf-profile.children.cycles-pp.__dev_queue_xmit 8.60 ą 2% +2.5 11.07 perf-profile.children.cycles-pp.__local_bh_enable_ip 7.91 ą 2% +2.6 10.47 perf-profile.children.cycles-pp.net_rx_action 0.00 +5.1 5.14 ą 2% perf-profile.children.cycles-pp.skb_defer_free_flush 25.81 -1.7 24.15 perf-profile.self.cycles-pp._copy_to_iter 1.73 ą 4% -1.6 0.10 ą 12% perf-profile.self.cycles-pp.net_rx_action 24.62 -1.5 23.14 perf-profile.self.cycles-pp._copy_from_iter 0.64 ą 7% -0.2 0.47 ą 7% perf-profile.self.cycles-pp.__virt_addr_valid 0.66 ą 5% -0.1 0.51 ą 7% perf-profile.self.cycles-pp.skb_attempt_defer_free 0.23 ą 8% -0.0 0.19 ą 11% perf-profile.self.cycles-pp.skb_release_data 0.16 ą 11% -0.0 0.13 ą 11% perf-profile.self.cycles-pp.handle_softirqs 0.15 ą 7% -0.0 0.13 ą 8% perf-profile.self.cycles-pp.__sys_sendto 0.05 ą 8% +0.0 0.07 ą 6% perf-profile.self.cycles-pp.__flush_smp_call_function_queue 0.06 ą 7% +0.0 0.08 ą 14% perf-profile.self.cycles-pp.udp4_csum_init 0.08 ą 17% +0.0 0.11 ą 10% perf-profile.self.cycles-pp.update_cfs_group 0.09 ą 17% +0.0 0.13 ą 14% perf-profile.self.cycles-pp.available_idle_cpu 0.31 ą 6% +0.0 0.36 ą 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.12 ą 14% +0.1 0.17 ą 11% perf-profile.self.cycles-pp.irqtime_account_irq 0.00 +0.1 0.06 ą 19% perf-profile.self.cycles-pp.__skb_wait_for_more_packets 0.02 ą 99% +0.1 0.09 ą 26% perf-profile.self.cycles-pp.ttwu_queue_wakelist 0.27 ą 9% +0.1 0.35 ą 4% perf-profile.self.cycles-pp.cpus_share_cache 0.38 ą 6% +0.1 0.47 ą 10% perf-profile.self.cycles-pp.sock_def_readable 0.37 ą 11% +0.1 0.46 ą 5% perf-profile.self.cycles-pp.flush_smp_call_function_queue 0.37 ą 9% +0.1 0.47 ą 3% perf-profile.self.cycles-pp.__wake_up_common 0.43 ą 4% +0.1 0.53 ą 6% perf-profile.self.cycles-pp.call_function_single_prep_ipi 0.39 ą 3% +0.1 0.49 ą 3% perf-profile.self.cycles-pp.os_xsave 0.36 ą 5% +0.1 0.46 ą 9% perf-profile.self.cycles-pp.try_to_wake_up 0.58 ą 6% +0.1 0.71 ą 3% perf-profile.self.cycles-pp.__skb_datagram_iter 0.12 ą 11% +0.6 0.74 ą 5% perf-profile.self.cycles-pp._find_next_bit 7.10 ą 3% +0.8 7.90 ą 2% perf-profile.self.cycles-pp.intel_idle 0.00 +3.1 3.08 ą 4% perf-profile.self.cycles-pp.skb_defer_free_flush Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki