On 2020-12-30 14:09:19 [+0100], Jonathan Schwender wrote:

Hi everyone,

I've been trying to test the real-time `performance` possible with
containers, by running cyclictest in a container on an RT-Kernel.
The issue I've been having does not require containers or an
RT kernel though.

Issue: cyclictest freezes after running for a few seconds
to minutes. After that only the loadavg section is updated,
while the count line does not change anymore.
cyclictest can't be killed after that point
other than by restarting the machine, and
this also takes a few minutes until the kernel kills
cyclictest.

This behaviour only occurs when the following conditions are
met:

- RT_GROUP_SCHED is used
- cyclictest is bound to an isolated cpu core with
  nohz_full=<core>, and isolcpus=nohz,domain,<core>

So if you remove RT_GROUP_SCHED and use cyclictest on the nohz_full
cores then everything is fine?

I've tested this on a machine with Fedora 33 and vanilla
stable 5.10.3 kernel with RT_GROUP_SCHED.
The same behaviour also exists on 5.10.1-rt20 with
PREEMPT_RT and RT_GROUP_SCHED configured.

After booting I configure the rt_runtime_us like this:
`echo "700000" > /sys/fs/cgroup/cpu,cpuacct/user.slice/cpu.rt_runtime_us`
`echo "100000" > /sys/fs/cgroup/cpu,cpuacct/system.slice/cpu.rt_runtime_us`

Then I start cyclictest via:
`taskset -c 14 cgexec -g cpu,cpuacct:user.slice cyclictest --mlockall \
  --priority=96 --interval=200 --affinity=14 --duration=15m`

These are the cmdline options I tried out to narrow the problem down:
working: `isolcpus=14 irqaffinity=0-3 maxcpus=15
systemd.unified_cgroup_hierarchy=0`
working: `isolcpus=nohz,14 nohz_full=14 irqaffinity=0-3 maxcpus=15
systemd.unified_cgroup_hierarchy=0`
working: `isolcpus=nohz,domain,14 irqaffinity=0-3 maxcpus=15
systemd.unified_cgroup_hierarchy=0`
broken:  `isolcpus=nohz,domain,14 nohz_full=14 irqaffinity=0-3 maxcpus=15
systemd.unified_cgroup_hierarchy=0`

unified_cgroup_hierarchy is needed to get cgroups v1, which
seems to be needed for RT_GROUP_SCHED (at least I couldn't
find any options similar to cpu.rt_runtime_us with the default
cgroup v2).
Basically it boils down to that the combination of the
domain parameter to isolcpus and nohz_full together with
RT_GROUP_SCHED cause the problem I'm observing.

Does anyone have any idea what could be causing this?
Am I doing something wrong, or is there an issue with cyclictest or
even the kernel that's causing this?

My motivation is running (testing) a real-time container on isolated
cores, so I think I do need all the kernel parameters I used above to
get good latencies.

You might want to try without nohz_full. My understanding is that this
used if your application remains mostly in userland (and uses no
syscalls, etc.).

Let me this on my list of things to try out.

Regards,

Jonathan Schwender

Sebastian