On 10/14/25 3:43 AM, Roman Gushchin wrote:

Daniel Sedlak [off-list ref] writes:

quoted

On 10/9/25 9:02 PM, Roman Gushchin wrote:

quoted

Shakeel Butt [off-list ref] writes:

quoted

On Thu, Oct 09, 2025 at 10:58:51AM -0700, Roman Gushchin wrote:

quoted

Shakeel Butt [off-list ref] writes:

quoted

On Thu, Oct 09, 2025 at 08:32:27AM -0700, Roman Gushchin wrote:

quoted

Daniel Sedlak [off-list ref] writes:

quoted

Hi Roman,

On 10/8/25 8:58 PM, Roman Gushchin wrote:

quoted

quoted

This patch exposes a new file for each cgroup in sysfs which is a
read-only single value file showing how many microseconds this cgroup
contributed to throttling the throughput of network sockets. The file is
accessible in the following path.

     /sys/fs/cgroup/**/<cgroup name>/memory.net.throttled_usec

Hi Daniel!
How this value is going to be used? In other words, do you need an
exact number or something like memory.events::net_throttled would be
enough for your case?

Just incrementing a counter each time the vmpressure() happens IMO
provides bad semantics of what is actually happening, because it can
hide important details, mainly the _time_ for how long the network
traffic was slowed down.

For example, when memory.events::net_throttled=1000, it can mean that
the network was slowed down for 1 second or 1000 seconds or something
between, and the memory.net.throttled_usec proposed by this patch
disambiguates it.

In addition, v1/v2 of this series started that way, then from v3 we
rewrote it to calculate the duration instead, which proved to be
better information for debugging, as it is easier to understand
implications.

But how are you planning to use this information? Is this just
"networking is under pressure for non-trivial amount of time ->
raise the memcg limit" or something more complicated?

We plan to use it mostly for observability purposes and to better
understand which traffic patterns affect the socket pressure the most
(so we can try to fix/delay/improve it). We do not know how commonly
this issue appears in other deployments, but in our deployment, many
of servers were affected by this slowdown, which varied in terms of
hardware and software configuration. Currently, it is very hard to
detect if the socket is under pressure without using tools like
bpftrace, so we would like to expose this metric in a more accessible
way. So in the end, we do not really care in which file this "socket
pressure happened" notification will be stored.

quoted

quoted

quoted

quoted

quoted

I totally get it from the debugging perspective, but not sure about
usefulness of it as a permanent metric. This is why I'm asking if there
are lighter alternatives, e.g. memory.events or maybe even tracepoints.

If the combination of memory.events(.local) and tracepoint hook(s) is
okay with you(?), we can use that and export the same information as
in the current patch version. We can incorporate that into the next
version.

In my opinion
tracepoint > memory.events entry > memory.stat entry > new cgroupfs file.

Thanks, noted, we will incorporate it to the next version.

quoted

Also, would it be possible to make the socket pressure signal
configurable, e.g., allowing it to be configured via sysctl or per
cgroup not to trigger the socket pressure signal? I cannot find the
reasoning why this throttling cannot (maybe it can) be opt-out.

It's a good point.

First, I think that vmpressure implementation is not the best
and we might want to switch to PSI (or something else) there.
This is why I'm resistant to exposing implementation-specific
metrics.

That said, I believe that some level of customization here is justified.
Maybe opting out completely is too much, but in the end it's hard for
the kernel to balance the importance of e.g. page cache vs networking
buffers as it might be really workload-dependent. Or some workloads
would prefer to risk being oom-killed rather than to tolerate a sub-par
networking performance.

As of now, socket pressure throttling can be disabled by moving 
processes, causing the pressure, into the root cgroup. So we would 
definitely benefit from disabling it more idiomatically.

This bpftrace output is captured from a production server using nginx 
proxy (the left-most column is a timestamp in nanoseconds) which we use 
as a HTTP cache. As you can see, it fluctuates a lot.

26920285712831843, unified:/system.slice/nginx.service, scanned: 556, 
reclaimed: 146, pressure: 73
26920285731493743, unified:/system.slice/nginx.service, scanned: 22886, 
reclaimed: 13606, pressure: 40
26920285779559500, unified:/system.slice/nginx.service, scanned: 21775, 
reclaimed: 11525, pressure: 47
26920285784845147, unified:/system.slice/nginx.service, scanned: 698, 
reclaimed: 522, pressure: 25
26920285833808666, unified:/system.slice/nginx.service, scanned: 740, 
reclaimed: 232, pressure: 68
26920285835668081, unified:/system.slice/nginx.service, scanned: 1475, 
reclaimed: 1224, pressure: 17
26920285838877445, unified:/system.slice/nginx.service, scanned: 2919, 
reclaimed: 2334, pressure: 20
26920285854811898, unified:/system.slice/nginx.service, scanned: 11586, 
reclaimed: 7666, pressure: 33
26920285873634643, unified:/system.slice/nginx.service, scanned: 22898, 
reclaimed: 13387, pressure: 41
26920285899176135, unified:/system.slice/nginx.service, scanned: 10957, 
reclaimed: 7077, pressure: 35
26920285901529378, unified:/system.slice/nginx.service, scanned: 587, 
reclaimed: 156, pressure: 73
26920286020702357, unified:/system.slice/nginx.service, scanned: 563, 
reclaimed: 87, pressure: 84
26920286037434038, unified:/system.slice/nginx.service, scanned: 22072, 
reclaimed: 14161, pressure: 35
26920285789562313, unified:/system.slice/nginx.service, scanned: 2810, 
reclaimed: 1696, pressure: 39
26920285879597883, unified:/system.slice/nginx.service, scanned: 693, 
reclaimed: 625, pressure: 9
26920285884686863, unified:/system.slice/nginx.service, scanned: 2768, 
reclaimed: 2284, pressure: 17

We believe that the issue originates from suboptimally chosen constants, 
as seen in [1]. Currently, the vmpressure triggers when it cannot 
reclaim a few MiB of memory on a server that has over 500 GiB of memory.

Link: 
https://elixir.bootlin.com/linux/v6.17.1/source/mm/vmpressure.c#L38 [1]

We would like to work on that more after this patch to try to find a 
better constant or at least make it _more configurable_ if that makes 
sense for you.

Thanks!
Daniel