Re: [PATCH v4 net-next 2/2] net: introduce budget_squeeze to help us tune rx behavior

On Fri, Mar 17, 2023 at 11:26 AM Jakub Kicinski [off-list ref] wrote:

On Fri, 17 Mar 2023 10:27:11 +0800 Jason Xing wrote:

quoted

quoted

That is the common case, and can be understood from the napi trace

Thanks for your reply. It is commonly happening every day on many servers.

Right but the common issue is the time squeeze, not budget squeeze,
and either way the budget squeeze doesn't really matter because
the softirq loop will call us again soon, if softirq itself is
not scheduled out.

So if you want to monitor a meaningful event in your fleet, I think
a better event to monitor is the number of times ksoftirqd was woken
up and latency of it getting onto the CPU.

Did you try to measure that?

[...]

(Please do *not* send patches to touch softirq code right now, just
measure first. We are trying to improve the situation but the core
kernel maintainers are weary of changes:
https://lwn.net/Articles/925540/
so if both of us start sending code they will probably take neither
patches :()

Hello Jakub,

I'm wondering for now if I can update and resend this patch to have a
better monitor (actually we do need one) on this part since we have
touched the net_rx_action() in the rps optimization patch series?
Also, just like Jesper mentioned before, it can be considered as one
'fix' to a old problem but targetting to net-next is just fine. What
do you think about it ?

Thanks,
Jason

quoted

quoted

point and probing the kernel with bpftrace. We should only add

We probably can deduce (or guess) which one causes the latency because
trace_napi_poll() only counts the budget consumed per poll.

Besides, tracing napi poll is totally ok with the testbed but not ok
with those servers with heavy load which bpftrace related tools
capturing the data from the hot path may cause some bad impact,
especially with special cards equipped, say, 100G nic card. Resorting
to legacy file softnet_stat is relatively feasible based on my limited
knowledge.

Right, but we're still measuring something relatively irrelevant.
As I said the softirq loop will call us again. In my experience
network queues get long when ksoftirqd is woken up but not scheduled
for a long time. That is the source of latency. You may have the same
problem (high latency) without consuming the entire budget.

I think if we wanna make new stats we should try to come up with a way
of capturing the problem rather than one of the symptoms.

quoted

Paolo also added backlog queues into this file in 2020 (see commit:
7d58e6555870d). I believe that after this patch, there are few or no
more new data that is needed to print for the next few years.

quoted

uAPI for statistics which must be maintained contiguously. For

In this patch, I didn't touch the old data as suggested in the
previous emails and only separated the old way of counting
@time_squeeze into two parts (time_squeeze and budget_squeeze). Using
budget_squeeze can help us profile the server and tune it more
usefully.

quoted

investigations tracing will always be orders of magnitude more
powerful :(

quoted

On the time squeeze BTW, have you found out what the problem was?
In workloads I've seen the time problems are often because of noise
in how jiffies are accounted (cgroup code disables interrupts
for long periods of time, for example, making jiffies increment
by 2, 3 or 4 rather than by 1).

Yes ! The issue of jiffies increment troubles those servers more often
than not. For a small group of servers, budget limit is also a
problem. Sometimes we might treat guest OS differently.