On Wed, Nov 10, 2021 at 03:21:54PM +0000, "Moessbauer, Felix" [off-list ref] wrote:

2. Threads can be started on non-rt CPUs and then bound to a specific rt CPU.
This binding can be specified before thread creation via pthread_create.
By that, you can make sure that at no point in time a thread has a
"forbidden" CPU in its affinities.

It should boil down to some clone$version(2) and sched_setaffinity(2)
calls, so strictly speaking even with pthread_create(3) the thread is
shortly running with the parent's affinity.

With cgroup2, you cannot guarantee the second aspect, as thread
creation and moving to a cgroup is not an atomic operation.

As suggested by others, CLONE_INTO_CGROUP (into cpuset cgroup) can
actually "hide" the migration into the clone3() call.

At creation time, you cannot set the final affinity mask (as you
create it in the non-rt group and there the CPU is not in the
cpuset.cpus).
Once you move the thread to the rt cgroup, it has a default mask and
by that can be executed on other rt cores.

Good point. Perhaps you could work this around by having another level
of (non-root partition) cpuset cgroups for individual CPUs? (Maybe
there's more clever approach, this is just first to come into my mind.)

Michal