Hi Sebastian,

-----Original Message-----
From: Frederic Weisbecker <redacted>
Sent: Friday, November 12, 2021 5:37 PM
To: Sebastian Andrzej Siewior <redacted>
Cc: Moessbauer, Felix (T RDA IOT SES-DE) <redacted>;
cgroups-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; linux-rt-users-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Schild, Henning (T
RDA IOT SES-DE) [off-list ref]; Kiszka, Jan (T RDA IOT)
[off-list ref]; Schmidt, Adriaan (T RDA IOT SES-DE)
[off-list ref]; Zefan Li [off-list ref]; Tejun
Heo [off-list ref]; Johannes Weiner [off-list ref]; Waiman Long
[off-list ref]
Subject: Re: Questions about replacing isolcpus by cgroup-v2

On Fri, Nov 12, 2021 at 04:36:56PM +0100, Sebastian Andrzej Siewior wrote:

quoted

On 2021-11-04 17:29:08 [+0000], Moessbauer, Felix wrote:

quoted

Dear subscribers,

Hi,

I Cced cgroups@vger since thus question fits there better.
I Cced Frederic in case he has come clues regarding isolcpus and
cgroups.

quoted

we are currently evaluating how to rework realtime tuning to use cgroup-v2

cpusets instead of the isolcpus kernel parameter.

quoted

quoted

Our use-case are realtime applications with rt and non-rt threads. Hereby,

the non-rt thread might create additional non-rt threads:

quoted

quoted

Example (RT CPU=1, 4 CPUs):
- Non-RT Thread (A) with default affinity 0xD (1101b)
- RT Thread (B) with Affinity 0x2 (0010b, via set_affinity)

When using pure isolcpus and cgroup-v1, just setting isolcpus=1 perfectly

works:

quoted

quoted

Thread A gets affinity 0xD, Thread B gets 0x2 and additional threads get a

default affinity of 0xD.

quoted

quoted

By that, independent of the threads' priorities, we can ensure that nothing is

scheduled on our RT cpu (except from kernel threads, etc...).

quoted

quoted

During this journey, we discovered the following:

Using cgroup-v2 cpusets and isolcpus together seems to be incompatible:
When activating the cpuset controller on a cgroup (for the first time), all

default CPU affinities are reset.

quoted

quoted

By that, also the default affinity is set to 0xFFFF..., while with isolcpus we

expect it to be (0xFFFF - isolcpus).

quoted

quoted

This breaks the example from above, as now the non-RT thread can
also be scheduled on the RT CPU.

That sounds buggy from the cpuset-v2 side (adding the maintainers in Cc).

Also please have a look into "[PATCH v8 0/6] cgroup/cpuset: Add new cpuset
partition type & empty effecitve cpus":


https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.kern
el.org%2Flkml%2F20211018143619.205065-1-
longman%40redhat.com%2F&data=04%7C01%7Cfelix.moessbauer%40sie
mens.com%7C1a74cbf4e3d140a9031808d9a5faad3c%7C38ae3bcd95794fd4add
ab42e1495d55a%7C1%7C0%7C637723318334809165%7CUnknown%7CTWFpb
GZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6M
n0%3D%7C1000&sdata=Jm3j2vDvOOtikU5ZpusupQ6d6koPII9oYZZhpDUkvY
c%3D&reserved=0

This stuff adds support for a new "isolated" partition type on cpuset/cgroup-v2
which should behave just like isolcpus.

I already tested the patch and reported back on the ML.
However, it only covers load-balancing aspects, not isolcpu like default affinities.
When setting cpusets.cpus.partition=isolated, you get similar behavior as with =root
(cpus are removed from all other groups), but the schedulers load-balancing is disabled
for this domain.
For details, please have a look in the other thread. 

quoted

quoted

When only using cgroup-v2, we can isolate our RT process by placing it in a

cgroup with CPUs=0,1 and remove CPU=1 from all other cgroups.

quoted

quoted

However, we do not know of a strategy to set a default affinity:
Given the example above, we have no way to ensure that newly created

threads are born with an affinity of just 0x2 (without changing the application).

quoted

quoted

Finally, isolcpus itself is deprecated since kernel 5.4.

Where is this the deprecation of isolcpus announced/ written?

We tried to deprecate it but too many people are still using it. Better pick an
interface that allows you to change the isolated set at runtime like
cpuset.sched_load_balance on cpuset/cgroup-v1 or the above patchset on v2.

Currently, the only workaround we know of to get isolcpu semantics on systems where
other tools like container runtimes or libvirt fiddle around with the cpuset controller, is to simply enforce the cgroups-v1.
But maybe we are just running into the bug from above.

Felix

Thanks.