@@ -178,6 +178,8 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable,ret=0;}+update_resource_highwatermark(RLIMIT_MEMLOCK,locked<<PAGE_SHIFT);+out:if(ret<0)usnic_uiom_put_pages(chunk_list,0);
From: Peter Zijlstra <peterz@infradead.org> Date: 2016-07-15 12:43:30
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
And how is that useful? Setting things to the high watermark is
basically the same as not setting the limit at all.
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
Example run of program from Documentation/accounting/getdelauys.c:
./getdelays -R -p `pidof smartd`
printing resource accounting
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
./getdelays -R -C /sys/fs/cgroup/systemd/system.slice/smartd.service/
printing resource accounting
sleeping 1, blocked 0, running 0, stopped 0, uninterruptible 0
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
Does this mean that rlimit_data and rlimit_stack should be set to the
values as specified by the data above?
Do we expect a smart user space daemon to then tweak the RLIMIT values?
Balbir Singh.
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
quoted
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
And how is that useful? Setting things to the high watermark is
basically the same as not setting the limit at all.
From: Peter Zijlstra <peterz@infradead.org> Date: 2016-07-15 13:59:56
On Fri, Jul 15, 2016 at 01:52:48PM +0000, Topi Miettinen wrote:
On 07/15/16 12:43, Peter Zijlstra wrote:
quoted
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
quoted
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
And how is that useful? Setting things to the high watermark is
basically the same as not setting the limit at all.
What else would you use, too small limits?
That question doesn't make sense.
What's the point of setting a limit if it ends up being the same as
no-limit (aka unlimited).
If you cannot explain; and you have not so far; what use these values
are, why would we look at the patches.
From: Richard Weinberger <richard@nod.at> Date: 2016-07-15 14:19:13
Hi!
Am 15.07.2016 um 12:35 schrieb Topi Miettinen:
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
Example run of program from Documentation/accounting/getdelauys.c:
./getdelays -R -p `pidof smartd`
printing resource accounting
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
./getdelays -R -C /sys/fs/cgroup/systemd/system.slice/smartd.service/
printing resource accounting
sleeping 1, blocked 0, running 0, stopped 0, uninterruptible 0
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
In this example, smartd is running as a non-root user. The presented
values can be used as a starting point for giving new limits to the
service.
I don't think it is worth sprinkling the kernel with update_resource_highwatermark()
calls just to get these metrics.
Can't we teach the existing perf infrastructure to collect these highwatermarks for us?
Thanks,
//richard
Track maximum size of locked memory, to be able to configure
RLIMIT_MEMLOCK resource limits. The information is available
with taskstats and cgroupstats netlink socket.
So I personally still dislike the very idea of this series... but I won't
argue if you convince maintainers.
quoted hunk
@@ -2020,6 +2020,10 @@ static int acct_stack_growth(struct vm_area_struct *vma, unsigned long size, uns return -ENOMEM; update_resource_highwatermark(RLIMIT_STACK, actual_size);+ if (vma->vm_flags & VM_LOCKED)+ update_resource_highwatermark(RLIMIT_MEMLOCK,+ (mm->locked_vm + grow) <<+ PAGE_SHIFT);
Btw this is not right. The same for the previous patch which tracks
RLIMIT_STACK. The "current" task can debugger/etc.
Yes, yes, this just reminds that the whole rlimit logic in this path
is broken but still...
Oleg.
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
quoted
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
Example run of program from Documentation/accounting/getdelauys.c:
./getdelays -R -p `pidof smartd`
printing resource accounting
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
./getdelays -R -C /sys/fs/cgroup/systemd/system.slice/smartd.service/
printing resource accounting
sleeping 1, blocked 0, running 0, stopped 0, uninterruptible 0
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
Does this mean that rlimit_data and rlimit_stack should be set to the
values as specified by the data above?
My plan is that either system administrator, distro maintainer or even
upstream developer can get reasonable values for the limits. They may
still be wrong, but things would be better than without any help to
configure the system.
Do we expect a smart user space daemon to then tweak the RLIMIT values?
Someone could write an autotuning daemon that checks if the system has
changed (for example due to upgrade) and then run some tests to
reconfigure the system. But the limits are a bit too fragile, or rather,
applications can't handle failure, so I don't know if that would really
work.
-Topi
On Fri, Jul 15, 2016 at 01:52:48PM +0000, Topi Miettinen wrote:
quoted
On 07/15/16 12:43, Peter Zijlstra wrote:
quoted
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
quoted
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
And how is that useful? Setting things to the high watermark is
basically the same as not setting the limit at all.
What else would you use, too small limits?
That question doesn't make sense.
What's the point of setting a limit if it ends up being the same as
no-limit (aka unlimited).
Having a limit is not the same as not having any limits at all. You're
in a way right that good limits don't affect the program normally. But
they can make a difference if the flow is not normal. For example a
successful exploit or a memory leak bug could cause RLIMIT_AS to trigger.
If you cannot explain; and you have not so far; what use these values
are, why would we look at the patches.
The use case is to allow system administrators, distro maintainers and
developers to configure systems to use the resource limits. The limits
are not very useful right now, as there is no way to figure out what
values to use. There are a few /proc files to look, for example current
number of file descriptors (for RLIMIT_NOFILE) could be counted via
/proc/pid/fd. But now there is no way to know if there were more in use
at some point. Likewise, a program can use more address space when you
are not looking. The source code does not tell these things explicitly.
-Topi
Hi!
Am 15.07.2016 um 12:35 schrieb Topi Miettinen:
quoted
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
Example run of program from Documentation/accounting/getdelauys.c:
./getdelays -R -p `pidof smartd`
printing resource accounting
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
./getdelays -R -C /sys/fs/cgroup/systemd/system.slice/smartd.service/
printing resource accounting
sleeping 1, blocked 0, running 0, stopped 0, uninterruptible 0
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
In this example, smartd is running as a non-root user. The presented
values can be used as a starting point for giving new limits to the
service.
I don't think it is worth sprinkling the kernel with update_resource_highwatermark()
calls just to get these metrics.
Can't we teach the existing perf infrastructure to collect these highwatermarks for us?
I don't know. What kind of changes do you think would be needed?
-Topi
Track maximum size of locked memory, to be able to configure
RLIMIT_MEMLOCK resource limits. The information is available
with taskstats and cgroupstats netlink socket.
So I personally still dislike the very idea of this series... but I won't
argue if you convince maintainers.
quoted
@@ -2020,6 +2020,10 @@ static int acct_stack_growth(struct vm_area_struct *vma, unsigned long size, uns return -ENOMEM; update_resource_highwatermark(RLIMIT_STACK, actual_size);+ if (vma->vm_flags & VM_LOCKED)+ update_resource_highwatermark(RLIMIT_MEMLOCK,+ (mm->locked_vm + grow) <<+ PAGE_SHIFT);
Btw this is not right. The same for the previous patch which tracks
RLIMIT_STACK. The "current" task can debugger/etc.
acct_stack_growth() is called from expand_upwards() and
expand_downwards(). They call security_mmap_addr() and the various LSM
implementations also use current task in the checks. Are these also not
right?
Yes, yes, this just reminds that the whole rlimit logic in this path
is broken but still...
I'd be happy to fix the logic with a separate prerequisite patch and
then use the right logic for this patch, but I'm not sure I know how.
Could you elaborate a bit?
-Topi
On Fri, Jul 15, 2016 at 01:52:48PM +0000, Topi Miettinen wrote:
quoted
On 07/15/16 12:43, Peter Zijlstra wrote:
quoted
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
quoted
Hello,
There are many basic ways to control processes, including
capabilities,
quoted
quoted
quoted
cgroups and resource limits. However, there are far fewer ways to
find out
quoted
quoted
quoted
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice
starting
quoted
quoted
quoted
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update
the mark
quoted
quoted
quoted
nearby.
And how is that useful? Setting things to the high watermark is
basically the same as not setting the limit at all.
What else would you use, too small limits?
That question doesn't make sense.
What's the point of setting a limit if it ends up being the same as
no-limit (aka unlimited).
If you cannot explain; and you have not so far; what use these values
are, why would we look at the patches.
One reason is to catch a malfunctioning process rather than dragging the whole system down with it. It could also be useful for development.
--
Sent from my Android device with K-9 Mail. Please excuse brevity and formatting.
Btw this is not right. The same for the previous patch which tracks
RLIMIT_STACK. The "current" task can debugger/etc.
acct_stack_growth() is called from expand_upwards() and
expand_downwards(). They call security_mmap_addr() and the various LSM
implementations also use current task in the checks. Are these also not
right?
Just suppose that the stack grows because you read/write to /proc/pid/mem.
quoted
Yes, yes, this just reminds that the whole rlimit logic in this path
is broken but still...
I'd be happy to fix the logic with a separate prerequisite patch and
then use the right logic for this patch, but I'm not sure I know how.
Could you elaborate a bit?
If only I Knew how to fix this ;) I mean, if only I could suggest a
simple fix. Because IMHO we do not really care, rlimts are obsolete.
Oleg.
Hi!
Am 15.07.2016 um 12:35 schrieb Topi Miettinen:
quoted
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
Example run of program from Documentation/accounting/getdelauys.c:
./getdelays -R -p `pidof smartd`
printing resource accounting
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
./getdelays -R -C /sys/fs/cgroup/systemd/system.slice/smartd.service/
printing resource accounting
sleeping 1, blocked 0, running 0, stopped 0, uninterruptible 0
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
In this example, smartd is running as a non-root user. The presented
values can be used as a starting point for giving new limits to the
service.
I don't think it is worth sprinkling the kernel with update_resource_highwatermark()
calls just to get these metrics.
Can't we teach the existing perf infrastructure to collect these highwatermarks for us?
I'm not sure about perf (I don't know the internals of perf well enough
to comment), but I'm sure the systemtap infrastructure could do this,
and a preconfigured systemtap script could be shipped with the package
that would allow this.
--
Doug Ledford [off-list ref]
GPG Key ID: 0E572FDD
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
quoted
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
Example run of program from Documentation/accounting/getdelauys.c:
./getdelays -R -p `pidof smartd`
printing resource accounting
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
./getdelays -R -C /sys/fs/cgroup/systemd/system.slice/smartd.service/
printing resource accounting
sleeping 1, blocked 0, running 0, stopped 0, uninterruptible 0
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
Does this mean that rlimit_data and rlimit_stack should be set to the
values as specified by the data above?
My plan is that either system administrator, distro maintainer or even
upstream developer can get reasonable values for the limits. They may
still be wrong, but things would be better than without any help to
configure the system.
This is not necessarily true. It seems like there is a disconnect
between what these various values are for and what you are positioning
them as. Most of these limits are meant to protect the system from
resource starvation crashes. They aren't meant to be any sort of double
check on a specific application. The vast majority of applications can
have bugs, leak resources, and do all sorts of other bad things and
still not hit these limits. A program that leaks a file handle an hour
but only normally has 50 handles in use would take 950 hours of constant
leaking before these limits would kick in to bring the program under
control. That's over a month. What's more though, the kernel couldn't
really care less that a single application leaked files until it got to
1000 open. The real point of the limit on file handles (since they are
cheap) is just not to let the system get brought down. Someone could
maliciously fire up 1000 processes, and they could all attempt to open
up as many files as possible in order to drown the system in open
inodes. The combination of the limit on maximum user processes and
maximum files per process are intended to prevent this. They are not
intended to prevent a single, properly running application from
operating. In fact, there are very few applications that are likely to
break the 1000 file per process limit. It is outrageously high for most
applications. They will leak files and do all sorts of bad things
without this ever stopping them. But it does stop malicious programs.
And the process limit stops malicious users too. The max locked memory
is used by almost no processes, and for the very few that use it, the
default is more than enough. The major exception is the RDMA stack,
which uses it so much that we just disable it on large systems because
it's impossible to predict how much we'll need and we don't want a job
to get killed because it couldn't get the memory it needs for buffers.
The limit on POSIX message queues is another one where it's more than
enough for most applications which don't use this feature at all, and
the few systems that use this feature adjust the limit to something sane
on their system (we can't make the default sane for these special
systems or else it becomes an avenue for Denial of Service attack, so
the default must stay low and servers that make extensive use of this
feature must up their limit on a case by case basis).
quoted
Do we expect a smart user space daemon to then tweak the RLIMIT values?
Someone could write an autotuning daemon that checks if the system has
changed (for example due to upgrade) and then run some tests to
reconfigure the system. But the limits are a bit too fragile, or rather,
applications can't handle failure, so I don't know if that would really
work.
This misses the point of most of these limits. They aren't there to
keep normal processes and normal users in check. They are there to stop
runaway use. This runaway situation might be accidental, or it might be
a nefarious users. They are generally set exceedingly high for those
things every application uses, and fairly low for those things that
almost no application uses but which could be abused by the nefarious
user crowd.
Moreover, for a large percentage of applications, the highwatermark is a
source of great trickery. For instance, if you have a web server that
is hosting web pages written in python, and therefore are using
mod_python in the httpd server (assuming apache here), then your
highwatermark will never be a reliable, stable thing. If you get 1000
web requests in a minute, all utilizing the mod_python resource in the
web server, and you don't have your httpd configured to restart after
every few hundred requests handled, then mod_python in your httpd
process will grow seemingly without limit. It will consume tons of
memory. And the only limit on how much memory it will consume is
determined by how many web requests it handles in between its garbage
collection intervals * how much memory it allocates per request. If you
don't happen to catch the absolute highest amount while you are
gathering your watermarks, then when you actually switch the system to
enforcing the limits you learned from all your highwatermarks (you are
planning on doing that aren't you?....I didn't see a copy of the patch
1/14, so I don't know if this infrastructure ever goes back to enforcing
the limits or not, but I would assume so, what point is there in
learning what the limits should be if you then never turn around and
enforce them?), load spikes will cause random program failures.
Really, this looks like a solution in search of a problem. Right now,
the limits are set where they are because they do two things:
1) Stay out of the way of the vast majority of applications. Those
applications that get tripped up by the defaults (like RDMA applications
getting stopped by memlock settings) have setup guides that spell out
which limits need changed and hints on what to change them too.
2) Stop nefarious users or errant applications from a total runaway
situation on a machine.
If your applications run without fail unless they have already failed,
and the whole machine doesn't go down with your failed application, then
the limits are working as designed. If your typical machine
configuration includes 256GB of RAM, then you could probably stand to
increase some of the limits safely if you wanted to. But unless you
have applications getting killed because of these limits, why would you?
Right now, I'm inclined to NAK the patch set. I've only seen patch 9/14
since you didn't Cc: everyone on the patch 1/14 that added the
infrastructure. But, as I mentioned in another email, I think this can
be accomplished via a systemtap script instead so we keep the clutter
out of the kernel. And more importantly, these patches seem to be
thinking about these limits as though they are supposed to be some sort
of tight fitting container around applications that catch an errant
application as soon as it steps out of bounds. Nothing could be further
from the truth, and if we actually implemented something of that sort,
programs susceptible to high resource usage during load spikes would
suddenly start failing on a frequent basis. The proof that these limits
are working is given by the fact that we rarely hear from users about
their programs being killed for resource consumption, and yet we also
don't hear from users about their systems going down due to runaway
applications. From what I can tell from these patches, I would suspect
complaints from one of those two issues to increase once these patches
are in place and put in use, and that doesn't seem like a good thing.
--
Doug Ledford [off-list ref]
GPG Key ID: 0E572FDD
On Fri, Jul 15, 2016 at 01:35:47PM +0300, Topi Miettinen wrote:
quoted
Hello,
There are many basic ways to control processes, including capabilities,
cgroups and resource limits. However, there are far fewer ways to find out
useful values for the limits, except blind trial and error.
This patch series attempts to fix that by giving at least a nice starting
point from the highwater mark values of the resources in question.
I looked where each limit is checked and added a call to update the mark
nearby.
Example run of program from Documentation/accounting/getdelauys.c:
./getdelays -R -p `pidof smartd`
printing resource accounting
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
./getdelays -R -C /sys/fs/cgroup/systemd/system.slice/smartd.service/
printing resource accounting
sleeping 1, blocked 0, running 0, stopped 0, uninterruptible 0
RLIMIT_CPU=0
RLIMIT_FSIZE=0
RLIMIT_DATA=18198528
RLIMIT_STACK=135168
RLIMIT_CORE=0
RLIMIT_RSS=0
RLIMIT_NPROC=1
RLIMIT_NOFILE=55
RLIMIT_MEMLOCK=0
RLIMIT_AS=130879488
RLIMIT_LOCKS=0
RLIMIT_SIGPENDING=0
RLIMIT_MSGQUEUE=0
RLIMIT_NICE=0
RLIMIT_RTPRIO=0
RLIMIT_RTTIME=0
Does this mean that rlimit_data and rlimit_stack should be set to the
values as specified by the data above?
My plan is that either system administrator, distro maintainer or even
upstream developer can get reasonable values for the limits. They may
still be wrong, but things would be better than without any help to
configure the system.
This is not necessarily true. It seems like there is a disconnect
between what these various values are for and what you are positioning
them as. Most of these limits are meant to protect the system from
resource starvation crashes. They aren't meant to be any sort of double
check on a specific application. The vast majority of applications can
have bugs, leak resources, and do all sorts of other bad things and
still not hit these limits. A program that leaks a file handle an hour
but only normally has 50 handles in use would take 950 hours of constant
leaking before these limits would kick in to bring the program under
control. That's over a month. What's more though, the kernel couldn't
really care less that a single application leaked files until it got to
1000 open. The real point of the limit on file handles (since they are
cheap) is just not to let the system get brought down. Someone could
maliciously fire up 1000 processes, and they could all attempt to open
up as many files as possible in order to drown the system in open
inodes. The combination of the limit on maximum user processes and
maximum files per process are intended to prevent this. They are not
intended to prevent a single, properly running application from
operating. In fact, there are very few applications that are likely to
break the 1000 file per process limit. It is outrageously high for most
applications. They will leak files and do all sorts of bad things
without this ever stopping them. But it does stop malicious programs.
And the process limit stops malicious users too. The max locked memory
is used by almost no processes, and for the very few that use it, the
default is more than enough. The major exception is the RDMA stack,
which uses it so much that we just disable it on large systems because
it's impossible to predict how much we'll need and we don't want a job
to get killed because it couldn't get the memory it needs for buffers.
The limit on POSIX message queues is another one where it's more than
enough for most applications which don't use this feature at all, and
the few systems that use this feature adjust the limit to something sane
on their system (we can't make the default sane for these special
systems or else it becomes an avenue for Denial of Service attack, so
the default must stay low and servers that make extensive use of this
feature must up their limit on a case by case basis).
quoted
quoted
Do we expect a smart user space daemon to then tweak the RLIMIT values?
Someone could write an autotuning daemon that checks if the system has
changed (for example due to upgrade) and then run some tests to
reconfigure the system. But the limits are a bit too fragile, or rather,
applications can't handle failure, so I don't know if that would really
work.
This misses the point of most of these limits. They aren't there to
keep normal processes and normal users in check. They are there to stop
runaway use. This runaway situation might be accidental, or it might be
a nefarious users. They are generally set exceedingly high for those
things every application uses, and fairly low for those things that
almost no application uses but which could be abused by the nefarious
user crowd.
Moreover, for a large percentage of applications, the highwatermark is a
source of great trickery. For instance, if you have a web server that
is hosting web pages written in python, and therefore are using
mod_python in the httpd server (assuming apache here), then your
highwatermark will never be a reliable, stable thing. If you get 1000
web requests in a minute, all utilizing the mod_python resource in the
web server, and you don't have your httpd configured to restart after
every few hundred requests handled, then mod_python in your httpd
process will grow seemingly without limit. It will consume tons of
memory. And the only limit on how much memory it will consume is
determined by how many web requests it handles in between its garbage
collection intervals * how much memory it allocates per request. If you
don't happen to catch the absolute highest amount while you are
gathering your watermarks, then when you actually switch the system to
enforcing the limits you learned from all your highwatermarks (you are
planning on doing that aren't you?....I didn't see a copy of the patch
1/14, so I don't know if this infrastructure ever goes back to enforcing
the limits or not, but I would assume so, what point is there in
learning what the limits should be if you then never turn around and
enforce them?), load spikes will cause random program failures.
Really, this looks like a solution in search of a problem. Right now,
the limits are set where they are because they do two things:
1) Stay out of the way of the vast majority of applications. Those
applications that get tripped up by the defaults (like RDMA applications
getting stopped by memlock settings) have setup guides that spell out
which limits need changed and hints on what to change them too.
2) Stop nefarious users or errant applications from a total runaway
situation on a machine.
If your applications run without fail unless they have already failed,
and the whole machine doesn't go down with your failed application, then
the limits are working as designed. If your typical machine
configuration includes 256GB of RAM, then you could probably stand to
increase some of the limits safely if you wanted to. But unless you
have applications getting killed because of these limits, why would you?
Thanks for the long explanation. I'd suppose loose limits are also used
because it's hard to know good tighter values. I was thinking of using
tighter settings to make things less easy for exploit writers. With
tight limits for RLIMIT_AS, RLIMIT_DATA and RLIMIT_STACK (also
RLIMIT_FSIZE in case a daemon is not supposed to create new files) it
would not so easy for the initial exploit to mmap() a large next stage
payload.
But there could be more direct ways to prevent that. For example, if
there was a way for seccomp filters to access a share state, they could
implement a state machine that could switch to a stricter mode after the
application has entered the event loop. Most of the limits or the denial
of service case are not interesting to me anyway.
Right now, I'm inclined to NAK the patch set. I've only seen patch 9/14
since you didn't Cc: everyone on the patch 1/14 that added the
infrastructure. But, as I mentioned in another email, I think this can
be accomplished via a systemtap script instead so we keep the clutter
out of the kernel. And more importantly, these patches seem to be
thinking about these limits as though they are supposed to be some sort
of tight fitting container around applications that catch an errant
application as soon as it steps out of bounds. Nothing could be further
from the truth, and if we actually implemented something of that sort,
programs susceptible to high resource usage during load spikes would
suddenly start failing on a frequent basis. The proof that these limits
are working is given by the fact that we rarely hear from users about
their programs being killed for resource consumption, and yet we also
don't hear from users about their systems going down due to runaway
applications. From what I can tell from these patches, I would suspect
complaints from one of those two issues to increase once these patches
are in place and put in use, and that doesn't seem like a good thing.
Those complaints could increase either way if the users want to use
tight limits, with kernel assistance or with systemtap. Again, the cause
of lack of complaints could also be that users are unaware of how to get
tight limits, so the users have no option but to either ignore the
limits or to use loose settings.
-Topi
Hello,
I'm trying the systemtap approach and it looks promising. The script is
annotating strace-like output with capability, device access and RLIMIT
information. In the end there's a summary. Here's sample output from
wpa_supplicant run:
mprotect(0x7efebf140000, 16384, PROT_READ) = 0 [DATA 548864 -> 573440]
[AS 44986368 -> 45002752]
brk(0x55d9611f8000) = 94392125718528 missing
[Capabilities=CAP_SYS_ADMIN] [AS 45002752 -> 45010944]
open(0x55d960716462, O_RDWR) = 3 [DeviceAllow=/dev/char/1:3 rw ]
open("/dev/random", O_RDONLY|O_NONBLOCK) = 3 [DeviceAllow=/dev/char/1:8 r ]
socket(PF_LOCAL, SOCK_STREAM|SOCK_CLOEXEC, 0) = 4
[RestrictAddressFamilies=AF_UNIX] [NOFILE 3 -> 4]
open("/etc/wpa_supplicant.conf", O_RDONLY) = 5 [NOFILE 4 -> 5]
socket(PF_NETLINK, SOCK_RAW, 0) = 5 [RestrictAddressFamilies=AF_NETLINK]
socket(PF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, 16) = 6
[RestrictAddressFamilies=AF_NETLINK] [NOFILE 5 -> 6]
socket(PF_NETLINK, SOCK_RAW|SOCK_CLOEXEC, 16) = 7
[RestrictAddressFamilies=AF_NETLINK] [NOFILE 6 -> 7]
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 8
[RestrictAddressFamilies=AF_INET] [NOFILE 7 -> 8]
open("/dev/rfkill", O_RDONLY) = 9 [DeviceAllow=/dev/char/10:58 r ]
[NOFILE 8 -> 9]
socket(PF_LOCAL, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 10
[RestrictAddressFamilies=AF_UNIX] [NOFILE 9 -> 10]
sendmsg(6, 0x7ffc778f35b0, 0x0) = 36 [Capabilities=CAP_NET_ADMIN]
Summary:
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_RAW
Consider also missing CapabilityBoundingSet=CAP_SYS_ADMIN
DeviceAllow=/dev/char/1:3 rw
DeviceAllow=/dev/char/1:8 r
DeviceAllow=/dev/char/10:58 r
DeviceAllow=/dev/char/1:9 r
LimitFSIZE=0
LimitDATA=577536
LimitSTACK=139264
LimitCORE=0
LimitNOFILE=15
LimitAS=45146112
LimitNPROC=171
LimitMEMLOCK=0
LimitSIGPENDING=0
LimitMSGQUEUE=0
LimitNICE=0
LimitRTPRIO=0
RestrictAddressFamilies=AF_UNIX AF_INET AF_NETLINK AF_PACKET
MemoryDenyWriteExecute=true
Some values are not correct. NPROC is wrong because staprun needs to be
run as root instead of the separate privileged user for wpa_supplicant
and that messes user process count. DATA/AS/STACK seems to be a bit off.
I can easily use this as systemd service configuration drop-in otherwise.
Now, the relevant part for the kernel is that I'd like to analyze error
paths better, so the system calls would be also annotated when there's a
failure when a RLIMIT is too tight. It would be easier to insert probes
if there was only one path for RLIMIT checks. Would it be OK to make the
function task_rlimit() a full check against the limit and also make it a
non-inlined function, just for improved probing purposes?
There's already error analysis for the capabilities, but there are some
false positive hits (like brk() complaining about missing CAP_SYS_ADMIN
above).
-Topi