Hello,
On Mon, Sep 12, 2016 at 10:39:04AM -0700, Andy Lutomirski wrote:
quoted
quoted
Your idea of "trivially" doesn't match mine. You gave a use case in
I suppose I wasn't clear enough. It is trivial in the sense that if
the userland implements something which works for namespace-root, it
would work the same in system-root without further modifications.
So I guess userspace can trivially get it right and can just as trivially
get it wrong.
I wasn't trying to play a word game. What I was trying to say is that
a configuration which works for namespace-roots works for the
system-root too, in terms of cgroup hierarchy, without any
modifications.
quoted
Great, now we agree that what's currently implemented is valid. I
think you're still failing to recognize the inherent specialness of
the system-root and how much unnecessary pain the removal of the
exemption would cause at virtually no practical gain. I won't repeat
the same backing points here.
I'm starting to think that you could extend the exemption with considerably
less difficulty.
Can you please elaborate? It feels like you're repeating the same
opinions without really describing them in detail or backing them up
in the last couple replies. Having differing opinions is fine but to
actually hash them out, the opinions and their rationles need to be
laid out in detail.
quoted
There isn't much which is getting in the way of doing that. Again,
something which follows no-internal-task rule would behave the same no
matter where it is. The system-root is different in that it is exempt
from the rule and thus is more flexible but that difference is serving
the purpose of handling the inherent specialness of the system-root.
From *userspace's* POV, I still don't think there's any specialness except
from an accounting POV. After all, userspace has no control over the
special stuff anyway. And accounting doesn't matter: a namespace could
just see zeros in any special root accounting slots.
The disagreement here isn't really consequential. The only reason
this part became imporant is because you felt that something must be
broken, which you now don't think is the case.
I agree that there can be other ways to handle this but what's your
proposal here? And how would that be practically and substantically
better than what is implemented now?
quoted
You've been pushing for enforcing the restriction on the system-root
too and now are jumping to the opposite end. It's really frustrating
that this is such a whack-a-mole game where you throw ideas without
really thinking through them and only concede the bare minimum when
all other logical avenues are closed off. Here, again, you seem to be
stating a strong opinion when you haven't fully thought about it or
tried to understand the reasons behind it.
I think you should make it work the same way in namespace roots as it does
in the system root. I acknowledge that there are pros and cons of each. I
think the current middle ground is worse than either of the consistent
options.
Again, the only thing you're doing is restating the same opinion. I
understand that you have an impression that this can be done better
but how exactly?
quoted
But, whatever, let's go there: Given the arguments that I laid out for
the no-internal-tasks rule, how does the problem seem fixable through
relaxing the constraint?
By deciding that, despite the arguments you laid out, it's still worth
relaxing the constraint. Or by deciding to add the constraint to the root.
You're not really saying anything of substance in the above paragraph.
quoted
quoted
Isn't this the same thing? IIUC the constraint in question is that,
if a non-root cgroup has subtree control on, then it can't have
processes in it. This is the no-internal-tasks constraint, right?
Yes, that is what no-internal-tasks rule is but I don't understand how
that is the same thing as process granularity. Am I completely
misunderstanding what you are trying to say here?
Yes. I'm saying that no-internal-tasks could be relaxed per controller.
I was asking whether you were wondering whether no-internal-tasks rule
and process-granularity are the same thing. And, if that's not the
case, what the previous sentence meant. I can't make out what you're
responding to.
quoted
If you confine it to the cpu controller, ignore anonymous
consumptions, the rather ugly mapping between nice and weight values
and the fact that nobody could come up with a practical usefulness for
such setup, yes. My point was never that the cpu controller can't do
it but that we should find a better way of coordinating it with other
controllers and exposing it to individual applications.
I'm not sure what the nice-vs-weight thing has to do with internal
processes, but all of this is a question for Peter.
That part is from cgroup cpu controller weight being mapped to task
nice numbers because the priorities between the two have to be somehow
comparable. It's not a critical issue, just awkward.
quoted
After a migration, the cgroup and its interface knobs are a different
directory and files. Semantically, during migration, we aren't moving
the directory or files and it'd be bizarre to overlay the semantics
you're describing on top of the existing cgroupfs. We will have to
break away from the very basic vfs rules such as a fd, once opened,
always corresponding to the same file.
What kind of migration do you mean? Having fds follow rename(2) around is
the normal vfs behavior, so I don't really know what you mean.
Process or task migration by writing pid to cgroup.procs or tasks
file. cgroup never supported directory / cgroup level migrations.
quoted
If I'm not mistaken, namespaces don't allow this type of dynamic
migrations.
I don't see why they couldn't allow exactly this. If you rename(2) a
cgroup, any namespace with that cgroup as root should keep it as root,
completely atomically. If this doesn't work, I'd argue that it's a bug.
I hope this part is clear now.
quoted
A system agent has a large part of the system configuration under its
control (it's the system agent after all) and thus is way more
flexible in what assumptions it can dictate and depend on.
Can you give an example of any use case for which a system agent would
fork, exec a daemon that isn't written by the same developers as the system
agent, and then walk that daemon's process tree and move the processes
around in the cgroup hierarchy one by one? I think this is what you're
describing, and I don't see why doing so is sensible. Certainly if a
system agent gives the daemon write access to cgroupfs, it should not start
moving that daemon's children around individually.
That's the only way anything can be moved across cgroups. In terms of
resource control, I can't think of scenarios which would *require*
this behavior but it's still a behavior cgroup has to allow as there's
no "spawn this process in that cgroup" call and all migrations are
dynamic.
We can proclaim that once an application is started outer scope
shouldn't meddle with it. It would be another restriction where
violation would actually break applications tho. And it doesn't
address other downsides - making in-application controls less
approachable as it requires specific setup and cooperation from the
system agent, and the interface being awkward.
quoted
We can make that dynamic as long as the subtree is properly scoped;
however, there is an important design decision to make here. If we
open up full-on dynamic migrations to individual applications, we
commit ourselves to supporting arbitrarily high frequency migration
operations, which we've never supported before and will restrict what
we can do in terms of optimizing hot paths over migration.
I haven't (yet?) seen use cases where changing cgroups *quickly* is
important.
Android does something along this line - creating preset cgroups and
migrating processes according to their current states. The problem is
that once we generally open up the API to individual applications,
there is no good way of policing the usages and there certainly are
multiple ways to make use of frequent cgroup membership changes
especially for stateless controllers like CPU.
We can easily end up in situations where having several of these
usages on the same machine bogs down the whole system. One way to
avoid this is building the API so that changing cgroup membership is
naturally unattractive - e.g. membership can only be assigned only on
creation of a new thread or process, or migration can only be towards
deeper level in the tree, so that migrations can be used to organize
the threads and processes as necessary but not used as the primary
method of adjusting configurations dynamically.
quoted
It is really difficult to understand your position without
understanding where the requirements are coming from. Can you please
elaborate more on the workload? Why is the specific configuration
useful? What is it trying to achieve?
Multiple cooperating RT processes, most of which have non-RT helper
threads. For scheduling purposes, I lump the non-RT threads together.
I see. Can you please share how the cgroups are actually configured
(ie. how the weights are assigned and so on)?
Will you (Tejun), PeterZ, and maybe some of the other interested parties be
at KS? Maybe this is worth hashing out in person.
Yeap, it'd be nice to talk in person. However, I'm not sure talking
offline is the best way to hash out technical details. The discussion
has been painful but we're actually addressing technical
misunderstandings and where the actual disgreements lie. We really
need to agree on what we disagree on and why first.
Thanks.
--
tejun
From: Andy Lutomirski <luto@amacapital.net> Date: 2016-09-15 20:08:38
On Wed, Sep 14, 2016 at 1:00 PM, Tejun Heo [off-list ref] wrote:
Hello,
With regard to no-internal-tasks, I see (at least) three options:
1. Keep the cgroup2 status quo. Lots of distros and such are likely
to have their cgroup management fail if run in a container. I really,
really dislike this option.
2. Enforce no-internal-tasks for the root cgroup. Un-cgroupable
thinks will still get accounted to the root cgroup even if subtree
control is on, but no tasks can be in the root cgroup if the root
cgroup has subtree control on. (If some controllers removed the
no-internal-tasks restriction, this would apply to the root as well.)
I think this may annoy certain users. If so, and if those users are
doing something valid, then I think that either those users should be
strongly encouraged or even forced to changed so namespacing works for
them or that we should do (3) instead.
3. Remove the no-internal-tasks restriction entirely. I can see this
resulting in a lot of configuration awkwardness, but I think it will
*work*, especially since all of the controllers already need to do
something vaguely intelligent when subtree control is on in the root
and there are tasks in the root.
What I'm trying to say is that I think that option (1) is sufficiently
bad that cgroup2 should do (2) or (3) instead. If option (2) is
preferred and if it would break userspace, then I think we can work
around it by entirely deprecating cgroup2, renaming it to cgroup3, and
doing option (2) there. You've given reasons you don't like options
(2) and (3). I mostly agree with those reasons, but I don't think
they're strong enough to overcome the problems with (1).
BTW, Mike keeps mentioning exclusive cgroups as problematic with the
no-internal-tasks constraints. Do exclusive cgroups still exist in
cgroup2? Could we perhaps just remove that capability entirely? I've
never understood what problem exlusive cpusets and such solve that
can't be more comprehensibly solved by just assigning the cpusets the
normal inclusive way.
quoted
quoted
After a migration, the cgroup and its interface knobs are a different
directory and files. Semantically, during migration, we aren't moving
the directory or files and it'd be bizarre to overlay the semantics
you're describing on top of the existing cgroupfs. We will have to
break away from the very basic vfs rules such as a fd, once opened,
always corresponding to the same file.
What kind of migration do you mean? Having fds follow rename(2) around is
the normal vfs behavior, so I don't really know what you mean.
Process or task migration by writing pid to cgroup.procs or tasks
file. cgroup never supported directory / cgroup level migrations.
Ugh. Perhaps cgroup2 should start supporting this. I think that
making rename(2) work is simpler than adding a whole new API for
rgroups, and I think it could solve a lot of the same problems that
rgroups are trying to solve.
--Andy
From: Peter Zijlstra <peterz@infradead.org> Date: 2016-09-16 07:51:58
On Thu, Sep 15, 2016 at 01:08:07PM -0700, Andy Lutomirski wrote:
BTW, Mike keeps mentioning exclusive cgroups as problematic with the
no-internal-tasks constraints. Do exclusive cgroups still exist in
cgroup2? Could we perhaps just remove that capability entirely? I've
never understood what problem exlusive cpusets and such solve that
can't be more comprehensibly solved by just assigning the cpusets the
normal inclusive way.
Without exclusive sets we cannot split the sched_domain structure.
Which leads to not being able to actually partition things. That would
break DL for one.
On Thu, Sep 15, 2016 at 01:08:07PM -0700, Andy Lutomirski wrote:
quoted
BTW, Mike keeps mentioning exclusive cgroups as problematic with the
no-internal-tasks constraints. Do exclusive cgroups still exist in
cgroup2? Could we perhaps just remove that capability entirely? I've
never understood what problem exlusive cpusets and such solve that
can't be more comprehensibly solved by just assigning the cpusets the
normal inclusive way.
Without exclusive sets we cannot split the sched_domain structure.
Which leads to not being able to actually partition things. That would
break DL for one.
On Thu, Sep 15, 2016 at 01:08:07PM -0700, Andy Lutomirski wrote:
quoted
BTW, Mike keeps mentioning exclusive cgroups as problematic with the
no-internal-tasks constraints. Do exclusive cgroups still exist in
cgroup2? Could we perhaps just remove that capability entirely? I've
never understood what problem exlusive cpusets and such solve that
can't be more comprehensibly solved by just assigning the cpusets the
normal inclusive way.
Without exclusive sets we cannot split the sched_domain structure.
Which leads to not being able to actually partition things. That would
break DL for one.
Can you sketch out a toy example?
[ Also see Documentation/cgroup-v1/cpusets.txt section 1.7 ]
mkdir /cpuset
mount -t cgroup -o cpuset none /cpuset
mkdir /cpuset/A
mkdir /cpuset/B
cat /sys/devices/system/node/node0/cpulist > /cpuset/A/cpuset.cpus
echo 0 > /cpuset/A/cpuset.mems
cat /sys/devices/system/node/node1/cpulist > /cpuset/B/cpuset.cpus
echo 1 > /cpuset/B/cpuset.mems
# move all movable tasks into A
cat /cpuset/tasks | while read task; do echo $task > /cpuset/A/tasks ; done
# kill machine wide load-balancing
echo 0 > /cpuset/cpuset.sched_load_balance
# now place 'special' tasks in B
This partitions the scheduler into two, one for each node.
Hereafter no task will be moved from one node to another. The
load-balancer is split in two, one balances in A one balances in B
nothing crosses. (It is important that A.cpus and B.cpus do not
intersect.)
Ideally no task would remain in the root group, back in the day we could
actually do this (with exception of the cpu bound kernel threads), but
this has significantly regressed :-(
(still hate the workqueue affinity interface)
As is, tasks that are left in the root group get balanced within
whatever domain they ended up in.
And what's DL?
SCHED_DEADLINE, its a 'Global'-EDF like scheduler that doesn't support
CPU affinities (because that doesn't make sense). The only way to
restrict it is to partition.
'Global' because you can partition it. If you reduce your system to
single CPU partitions you'll reduce to P-EDF.
(The same is true of SCHED_FIFO, that's a 'Global'-FIFO on the same
partition scheme, it however does support sched_affinity, but using it
gives 'interesting' schedulability results -- call it a historic
accident).
Note that related, but differently, we have the isolcpus boot parameter
which creates single CPU partitions for all listed CPUs and gives the
rest to the root cpuset. Ideally we'd kill this option given its a boot
time setting (for something which is trivially to do at runtime).
But this cannot be done, because that would mean we'd have to start with
a !0 cpuset layout:
'/'
load_balance=0
/ \
'system' 'isolated'
cpus=~isolcpus cpus=isolcpus
load_balance=0
And start with _everything_ in the /system group (inclding default IRQ
affinities).
Of course, that will break everything cgroup :-(
On Thu, Sep 15, 2016 at 01:08:07PM -0700, Andy Lutomirski wrote:
quoted
BTW, Mike keeps mentioning exclusive cgroups as problematic with the
no-internal-tasks constraints. Do exclusive cgroups still exist in
cgroup2? Could we perhaps just remove that capability entirely? I've
never understood what problem exlusive cpusets and such solve that
can't be more comprehensibly solved by just assigning the cpusets the
normal inclusive way.
Without exclusive sets we cannot split the sched_domain structure.
Which leads to not being able to actually partition things. That would
break DL for one.
Can you sketch out a toy example?
[ Also see Documentation/cgroup-v1/cpusets.txt section 1.7 ]
mkdir /cpuset
mount -t cgroup -o cpuset none /cpuset
mkdir /cpuset/A
mkdir /cpuset/B
cat /sys/devices/system/node/node0/cpulist > /cpuset/A/cpuset.cpus
echo 0 > /cpuset/A/cpuset.mems
cat /sys/devices/system/node/node1/cpulist > /cpuset/B/cpuset.cpus
echo 1 > /cpuset/B/cpuset.mems
# move all movable tasks into A
cat /cpuset/tasks | while read task; do echo $task > /cpuset/A/tasks ; done
# kill machine wide load-balancing
echo 0 > /cpuset/cpuset.sched_load_balance
# now place 'special' tasks in B
This partitions the scheduler into two, one for each node.
Hereafter no task will be moved from one node to another. The
load-balancer is split in two, one balances in A one balances in B
nothing crosses. (It is important that A.cpus and B.cpus do not
intersect.)
Ideally no task would remain in the root group, back in the day we could
actually do this (with exception of the cpu bound kernel threads), but
this has significantly regressed :-(
(still hate the workqueue affinity interface)
I wonder if we could address this by creating (automatically at boot
or when the cpuset controller is enabled or whatever) a
/cpuset/random_kernel_shit cgroup and have all of the unmoveable tasks
land there?
As is, tasks that are left in the root group get balanced within
whatever domain they ended up in.
quoted
And what's DL?
SCHED_DEADLINE, its a 'Global'-EDF like scheduler that doesn't support
CPU affinities (because that doesn't make sense). The only way to
restrict it is to partition.
'Global' because you can partition it. If you reduce your system to
single CPU partitions you'll reduce to P-EDF.
(The same is true of SCHED_FIFO, that's a 'Global'-FIFO on the same
partition scheme, it however does support sched_affinity, but using it
gives 'interesting' schedulability results -- call it a historic
accident).
Hmm, I didn't realize that the deadline scheduler was global. But
ISTM requiring the use of "exclusive" to get this working is
unfortunate. What if a user wants two separate partitions, one using
CPUs 1 and 2 and the other using CPUs 3 and 4 (with 5 reserved for
non-RT stuff)? Shouldn't we be able to have a cgroup for each of the
DL partitions and do something to tell the deadline scheduler "here is
your domain"?
Note that related, but differently, we have the isolcpus boot parameter
which creates single CPU partitions for all listed CPUs and gives the
rest to the root cpuset. Ideally we'd kill this option given its a boot
time setting (for something which is trivially to do at runtime).
But this cannot be done, because that would mean we'd have to start with
a !0 cpuset layout:
'/'
load_balance=0
/ \
'system' 'isolated'
cpus=~isolcpus cpus=isolcpus
load_balance=0
And start with _everything_ in the /system group (inclding default IRQ
affinities).
Of course, that will break everything cgroup :-(
I would actually *much* prefer this over the status quo. I'm tired of
my crappy, partially-working script that sits there and creates
exactly this configuration (minus the isolcpus part because I actually
want migration to work) on boot. (Actually, it could have two
automatic cgroups: /kernel and /init -- init and UMH would go in init
and kernel threads and such would go in /kernel. Userspace would be
able to request that a different cgroup be used for newly-created
kernel threads.)
Heck, even systemd would probably prefer this. Then it could cleanly
expose a "slice" or whatever it's called for random kernel shit and at
least you could configure it meaningfully.
From: Peter Zijlstra <peterz@infradead.org> Date: 2016-09-16 16:51:02
On Fri, Sep 16, 2016 at 09:29:06AM -0700, Andy Lutomirski wrote:
quoted
SCHED_DEADLINE, its a 'Global'-EDF like scheduler that doesn't support
CPU affinities (because that doesn't make sense). The only way to
restrict it is to partition.
'Global' because you can partition it. If you reduce your system to
single CPU partitions you'll reduce to P-EDF.
(The same is true of SCHED_FIFO, that's a 'Global'-FIFO on the same
partition scheme, it however does support sched_affinity, but using it
gives 'interesting' schedulability results -- call it a historic
accident).
Hmm, I didn't realize that the deadline scheduler was global. But
ISTM requiring the use of "exclusive" to get this working is
unfortunate. What if a user wants two separate partitions, one using
CPUs 1 and 2 and the other using CPUs 3 and 4 (with 5 reserved for
non-RT stuff)?
{1,2} {3,4} {5} seem exclusive, did I miss something? (other than that 5
cpu parts are 'rare').
Shouldn't we be able to have a cgroup for each of the
DL partitions and do something to tell the deadline scheduler "here is
your domain"?
Somewhat confused, by doing the non-overlapping domains, you do exactly
that no?
You end up with 2 (or more) independent deadline schedulers, but if
you're not running deadline tasks (like in the /system partition) you
don't care its there.
quoted
Note that related, but differently, we have the isolcpus boot parameter
which creates single CPU partitions for all listed CPUs and gives the
rest to the root cpuset. Ideally we'd kill this option given its a boot
time setting (for something which is trivially to do at runtime).
But this cannot be done, because that would mean we'd have to start with
a !0 cpuset layout:
'/'
load_balance=0
/ \
'system' 'isolated'
cpus=~isolcpus cpus=isolcpus
load_balance=0
And start with _everything_ in the /system group (inclding default IRQ
affinities).
Of course, that will break everything cgroup :-(
I would actually *much* prefer this over the status quo. I'm tired of
my crappy, partially-working script that sits there and creates
exactly this configuration (minus the isolcpus part because I actually
want migration to work) on boot. (Actually, it could have two
automatic cgroups: /kernel and /init -- init and UMH would go in init
and kernel threads and such would go in /kernel. Userspace would be
able to request that a different cgroup be used for newly-created
kernel threads.)
So there's a problem with sticking kernel threads (and esp. kthreadd)
into !root groups. For example if you place it in a cpuset that doesn't
have all cpus, then binding your shiny new kthread to a cpu will fail.
You can fix that of course, and we used to do exactly that, but we kept
running into 'fun' cases like that.
The unbound workqueue stuff is totally arbitrary borkage though, that
can be made to work just fine, TJ didn't like it for some reason which I
really cannot remember.
Also, UMH?
Heck, even systemd would probably prefer this. Then it could cleanly
expose a "slice" or whatever it's called for random kernel shit and at
least you could configure it meaningfully.
No clue about systemd, I'm still on systems without that virus.
From: Andy Lutomirski <luto@amacapital.net> Date: 2016-09-16 18:20:12
On Fri, Sep 16, 2016 at 9:50 AM, Peter Zijlstra [off-list ref] wrote:
On Fri, Sep 16, 2016 at 09:29:06AM -0700, Andy Lutomirski wrote:
quoted
quoted
SCHED_DEADLINE, its a 'Global'-EDF like scheduler that doesn't support
CPU affinities (because that doesn't make sense). The only way to
restrict it is to partition.
'Global' because you can partition it. If you reduce your system to
single CPU partitions you'll reduce to P-EDF.
(The same is true of SCHED_FIFO, that's a 'Global'-FIFO on the same
partition scheme, it however does support sched_affinity, but using it
gives 'interesting' schedulability results -- call it a historic
accident).
Hmm, I didn't realize that the deadline scheduler was global. But
ISTM requiring the use of "exclusive" to get this working is
unfortunate. What if a user wants two separate partitions, one using
CPUs 1 and 2 and the other using CPUs 3 and 4 (with 5 reserved for
non-RT stuff)?
{1,2} {3,4} {5} seem exclusive, did I miss something? (other than that 5
cpu parts are 'rare').
There's no overlap, so they're logically exclusive, but it avoids
needing the "cpu_exclusive" parameter. It always seemed confusing to
me that a setting on a child cgroup would strictly remove a resource
from the parent. (To be clear: I don't have any particularly strong
objection to cpu_exclusive. It just always seemed like a bit of a
hack that mostly duplicated what you could get by just setting the
cpusets appropriately throughout the hierarchy.)
quoted
quoted
Note that related, but differently, we have the isolcpus boot parameter
which creates single CPU partitions for all listed CPUs and gives the
rest to the root cpuset. Ideally we'd kill this option given its a boot
time setting (for something which is trivially to do at runtime).
But this cannot be done, because that would mean we'd have to start with
a !0 cpuset layout:
'/'
load_balance=0
/ \
'system' 'isolated'
cpus=~isolcpus cpus=isolcpus
load_balance=0
And start with _everything_ in the /system group (inclding default IRQ
affinities).
Of course, that will break everything cgroup :-(
I would actually *much* prefer this over the status quo. I'm tired of
my crappy, partially-working script that sits there and creates
exactly this configuration (minus the isolcpus part because I actually
want migration to work) on boot. (Actually, it could have two
automatic cgroups: /kernel and /init -- init and UMH would go in init
and kernel threads and such would go in /kernel. Userspace would be
able to request that a different cgroup be used for newly-created
kernel threads.)
So there's a problem with sticking kernel threads (and esp. kthreadd)
into !root groups. For example if you place it in a cpuset that doesn't
have all cpus, then binding your shiny new kthread to a cpu will fail.
You can fix that of course, and we used to do exactly that, but we kept
running into 'fun' cases like that.
Blech. But may this *should* have that effect. I'm sick of random
kernel crap being scheduled on my RT CPUs and on the CPUs that I
intend to be kept forcibly idle.
The unbound workqueue stuff is totally arbitrary borkage though, that
can be made to work just fine, TJ didn't like it for some reason which I
really cannot remember.
Also, UMH?
User mode helper. Fortunately most users are gone now, but it still exists.
From: Peter Zijlstra <peterz@infradead.org> Date: 2016-09-17 01:47:51
On Fri, Sep 16, 2016 at 11:19:38AM -0700, Andy Lutomirski wrote:
On Fri, Sep 16, 2016 at 9:50 AM, Peter Zijlstra [off-list ref] wrote:
quoted
{1,2} {3,4} {5} seem exclusive, did I miss something? (other than that 5
cpu parts are 'rare').
There's no overlap, so they're logically exclusive, but it avoids
needing the "cpu_exclusive" parameter.
I'd need to double check, but I don't think you _need_ that. That's more
for enforcing nobody else steals your CPUs and 'accidentally' creates
overlaps. But if you configure it right, non-overlap should be enough.
That is, generate_sched_domains() only uses cpusets_overlap() which is
cpumask_intersects(). Then again, it is almost 4am, so who knows.
quoted
So there's a problem with sticking kernel threads (and esp. kthreadd)
into !root groups. For example if you place it in a cpuset that doesn't
have all cpus, then binding your shiny new kthread to a cpu will fail.
You can fix that of course, and we used to do exactly that, but we kept
running into 'fun' cases like that.
Blech. But may this *should* have that effect. I'm sick of random
kernel crap being scheduled on my RT CPUs and on the CPUs that I
intend to be kept forcibly idle.
Hehe, so ideally those threads don't do anything unless the tasks
running on those CPUs explicitly ask for it. If you find any of the
CPU-bound kernel tasks do work that is unrelated to the tasks running on
that CPU, we should certainly look into it.
Personally I'm not much bothered by idle threads sitting about.
Hello,
On Thu, Sep 15, 2016 at 01:08:07PM -0700, Andy Lutomirski wrote:
With regard to no-internal-tasks, I see (at least) three options:
1. Keep the cgroup2 status quo. Lots of distros and such are likely
to have their cgroup management fail if run in a container. I really,
I don't know where you're getting this. No-internal-tasks rule has
*NOTHING* to do with how or how not cgroup v1 hierarchies can be used
inside a namespace. I suppose this is coming from the same
misunderstanding that Austin has. Please see my reply there for more
details.
really dislike this option.
Up until this point, you haven't supplied any valid technical reasons
for your objection. Repeating "really" doesn't add to the discussion
at all. If you're indicating that you don't like it on an aeshtetic
ground, please just say so.
2. Enforce no-internal-tasks for the root cgroup. Un-cgroupable
thinks will still get accounted to the root cgroup even if subtree
control is on, but no tasks can be in the root cgroup if the root
cgroup has subtree control on. (If some controllers removed the
no-internal-tasks restriction, this would apply to the root as well.)
I think this may annoy certain users. If so, and if those users are
doing something valid, then I think that either those users should be
strongly encouraged or even forced to changed so namespacing works for
them or that we should do (3) instead.
Theoretically, we can do that but what are the upsides and are they
enough to justify the added inconveniences? Up until now, the only
argument you provided is that people may do certain things in
system-root which might not work in namespace-root but that isn't a
critical problem. No real functionalities are lost by implementing
the same behaviors both inside and outside namespaces.
3. Remove the no-internal-tasks restriction entirely. I can see this
resulting in a lot of configuration awkwardness, but I think it will
*work*, especially since all of the controllers already need to do
something vaguely intelligent when subtree control is on in the root
and there are tasks in the root.
The reasons for no-internal-tasks restriction have been explained
multiple times in the documentations and throughout this thread, and
we also discussed how and why system-root is special and allowing
system-root's special treatment doesn't break things.
What I'm trying to say is that I think that option (1) is sufficiently
bad that cgroup2 should do (2) or (3) instead. If option (2) is
preferred and if it would break userspace, then I think we can work
around it by entirely deprecating cgroup2, renaming it to cgroup3, and
doing option (2) there. You've given reasons you don't like options
(2) and (3). I mostly agree with those reasons, but I don't think
they're strong enough to overcome the problems with (1).
And you keep suggesting very drastic measures for an issue which isn't
critical without providing any substantial technical reasons why such
drastic measures would be necessary. This part of discussion started
with your misunderstanding of the implications of the system-root
being special, and the only reason you presented in the previous
message is still a, different, misunderstanding.
The only thing which isn't changing here is your opinions on how it
should be. It is a baffling situation because your opinions don't
seem to be affected at all by the validity of reasons for thinking so.
BTW, Mike keeps mentioning exclusive cgroups as problematic with the
no-internal-tasks constraints. Do exclusive cgroups still exist in
cgroup2? Could we perhaps just remove that capability entirely? I've
never understood what problem exlusive cpusets and such solve that
can't be more comprehensibly solved by just assigning the cpusets the
normal inclusive way.
This was explained before during the discussion. Maybe it wasn't
clear enough. The knob is a config protector which protects oneself
from changing its configs. It doesn't really belong in the kernel.
My guess is that it was added because delegation model wasn't properly
established and people tried to delegate resource control knobs along
with the cgroups and then wanted to prevent those knobs from changed
in certain ways.
quoted
quoted
What kind of migration do you mean? Having fds follow rename(2) around is
the normal vfs behavior, so I don't really know what you mean.
Process or task migration by writing pid to cgroup.procs or tasks
file. cgroup never supported directory / cgroup level migrations.
Ugh. Perhaps cgroup2 should start supporting this. I think that
making rename(2) work is simpler than adding a whole new API for
rgroups, and I think it could solve a lot of the same problems that
rgroups are trying to solve.
We haven't needed that yet and supporting rename(2) doesn't
necessarily make the API safe in terms of migration atomicity. Also,
as pointed out in my previous reply (and rgroup documentation),
atomicity is only one part of rationales for rgroup.
Thanks.
--
tejun