On Tue, Dec 31, 2024 at 6:52 AM Yafang Shao [off-list ref] wrote:

The LINUX_MIB_LISTENDROPS counter can be incremented for several reasons,
such as:
- A full SYN backlog queue
- SYN flood attacks
- Memory exhaustion
- Other resource constraints

Recently, one of our services experienced frequent ListenDrops. While
attempting to trace the root cause, we discovered that tracing the function
tcp_listendrop() was not straightforward because it is currently inlined.
To overcome this, we had to create a livepatch that defined a non-inlined
version of the function, which we then traced using BPF programs.

  $ grep tcp_listendrop /proc/kallsyms
  ffffffffc093fba0 t tcp_listendrop_x     [livepatch_tmp]

Through this process, we eventually determined that the ListenDrops were
caused by SYN attacks.

Since tcp_listendrop() is not part of the critical path, defining it as
noinline would make it significantly easier to trace and diagnose issues
without requiring workarounds such as livepatching. This function can be
used by kernel modules like smc, so export it with EXPORT_SYMBOL_GPL().

After that change, the result is as follows,

  $ grep tcp_listendrop /proc/kallsyms
  ffffffffb718eaa0 T __pfx_tcp_listendrop
  ffffffffb718eab0 T tcp_listendrop
  ffffffffb7e636b8 r __ksymtab_tcp_listendrop

Signed-off-by: Yafang Shao <redacted>
--

Are we going to accept one patch at a time to un-inline all TCP
related functions in the kernel ?

My understanding is that current tools work fine. You may need to upgrade yours.

# perf probe -a tcp_listendrop
Added new events:
  probe:tcp_listendrop (on tcp_listendrop)
  probe:tcp_listendrop (on tcp_listendrop)
  probe:tcp_listendrop (on tcp_listendrop)
  probe:tcp_listendrop (on tcp_listendrop)
  probe:tcp_listendrop (on tcp_listendrop)
  probe:tcp_listendrop (on tcp_listendrop)
  probe:tcp_listendrop (on tcp_listendrop)
  probe:tcp_listendrop (on tcp_listendrop)

You can now use it in all perf tools, such as:

perf record -e probe:tcp_listendrop -aR sleep 1