On Thu, 29 Jun 2006 09:44:08 -0700
Paul Jackson [off-list ref] wrote:
quoted
quoted
You're probably correct on that model. However, it all depends on the actual
workload. Are people who actually have large-CPU (>256) systems actually
running fork()-heavy things like webservers on them, or are they running things
like database servers and computations, which tend to have persistent
processes?
It may well be mostly as you say - the large-CPU systems not running
the fork() heavy jobs.
Sooner or later, someone will want to run a fork()-heavy job on a
large-CPU system. On a 1024 CPU system, it would apparently take
just 14 exits/sec/CPU to hit this bottleneck, if Jay's number of
14000 applied.
Chris Sturdivant's reply is reasonable -- we'll hit it sooner or later,
and deal with it then.
I agree, and I'm viewing this as blocking the taskstats merge. Because if
this _is_ a problem then it's a big one because fixing it will be
intrusive, and might well involve userspace-visible changes.
First off, just a reminder that this is inherently a netlink flow
control issue...which was being exacerbated
earlier by taskstats decision to send per-tgid data (no longer the case).
But I'd like to know whats our target here ? How many messages per
second do we want to be able to be sent
and received without risking any loss of data ? Netlink will lose
messages at a high enough rate so the design point
will need to be known a bit.
For statistics type usage of the genetlink/netlink, I would have thought
that userspace, provided it is reliably informed
about the loss of data through ENOBUFS, could take measures to just
account for the missing data and carry on ?
The only ways I can see of fixing the problem generally are to either
a) throw more CPU(s) at stats collection: allow userspace to register for
"stats generated by CPU N", then run a stats collection daemon on each
CPU or
b) make the kernel recognise when it's getting overloaded and switch to
some degraded mode where it stops trying to send all the data to
userspace - just send a summary, or a "we goofed" message or something.
One of the unused features of genetlink that's meant for high volume
data output from the kernel is
the "dump" callback of a genetlink connection. Essentially kernel space
keeps getting provided sk_buffs
to fill which the netlink layer then supplies to user space (over time I
guess ?)
But whatever we do, there's going to be some limit so its useful to
decide what the design point should be ?
Adding Jamal for his thoughts on netlink's flow control in the context
of genetlink.
--Shailabh
From: Paul Jackson <hidden> Date: 2006-06-29 19:24:17
Shailabh wrote:
First off, just a reminder that this is inherently a netlink flow
control issue...which was being exacerbated earlier by taskstats
decision to send per-tgid data (no longer the case).
But I'd like to know whats our target here ? How many messages
per second do we want to be able to be sent and received without
risking any loss of data ? Netlink will lose messages at a high
enough rate so the design point will need to be known a bit.
Perhaps its not so much an issue of the design rate, as an issue of
how we deal with hitting the limit. Sooner or later, perhaps due to
operator error, almost any implementable rate will be exceeded.
Ideally, we would both of the remedies that Andrew mentioned,
rephrasing:
1) a way for a customer who needs a higher rate to scale
the useful resources he can apply to the collection, and
2) a clear indicator when the supported rate was exceeded
anyway.
For statistics type usage of the genetlink/netlink, I would have
thought that userspace, provided it is reliably informed about the loss
of data through ENOBUFS, could take measures to just account for the
missing data and carry on ?
If that's so, then the ENOBUFS error may well meet my remedy (2) above,
leaving just the question of how a customer could scale to higher
rates, if they found it was worth doing so.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson [off-list ref] 1.925.600.0401
From: Andrew Morton <hidden> Date: 2006-06-29 19:34:19
On Thu, 29 Jun 2006 15:10:31 -0400
Shailabh Nagar [off-list ref] wrote:
quoted
I agree, and I'm viewing this as blocking the taskstats merge. Because if
this _is_ a problem then it's a big one because fixing it will be
intrusive, and might well involve userspace-visible changes.
First off, just a reminder that this is inherently a netlink flow
control issue...which was being exacerbated
earlier by taskstats decision to send per-tgid data (no longer the case).
But I'd like to know whats our target here ? How many messages per
second do we want to be able to be sent
and received without risking any loss of data ? Netlink will lose
messages at a high enough rate so the design point
will need to be known a bit.
For statistics type usage of the genetlink/netlink, I would have thought
that userspace, provided it is reliably informed
about the loss of data through ENOBUFS, could take measures to just
account for the missing data and carry on ?
Could be so. But we need to understand how significant the impact of this
will be in practice.
We could find, once this is deployed is real production environments on
large machines that the data loss is sufficiently common and sufficiently
serious that the feature needs a lot of rework.
Now there's always a risk of that sort of thing happening with all
features, but it's usually not this evident so early in the development
process. We need to get a better understanding of the risk before
proceeding too far.
And there's always a 100% reliable fix for this: throttling. Make the
sender of the messages block until the consumer can catch up. In some
situations, that is what people will want to be able to do. I suspect a
good implementation would be to run a collection daemon on each CPU and
make the delivery be cpu-local. That's sounding more like relayfs than
netlink.
On Thu, 29 Jun 2006 15:10:31 -0400
Shailabh Nagar [off-list ref] wrote:
quoted
quoted
I agree, and I'm viewing this as blocking the taskstats merge. Because if
this _is_ a problem then it's a big one because fixing it will be
intrusive, and might well involve userspace-visible changes.
First off, just a reminder that this is inherently a netlink flow
control issue...which was being exacerbated
earlier by taskstats decision to send per-tgid data (no longer the case).
But I'd like to know whats our target here ? How many messages per
second do we want to be able to be sent
and received without risking any loss of data ? Netlink will lose
messages at a high enough rate so the design point
will need to be known a bit.
For statistics type usage of the genetlink/netlink, I would have thought
that userspace, provided it is reliably informed
about the loss of data through ENOBUFS, could take measures to just
account for the missing data and carry on ?
Could be so. But we need to understand how significant the impact of this
will be in practice.
We could find, once this is deployed is real production environments on
large machines that the data loss is sufficiently common and sufficiently
serious that the feature needs a lot of rework.
Now there's always a risk of that sort of thing happening with all
features, but it's usually not this evident so early in the development
process. We need to get a better understanding of the risk before
proceeding too far.
And there's always a 100% reliable fix for this: throttling. Make the
sender of the messages block until the consumer can catch up.
Is blocking exits an option ?
In some
situations, that is what people will want to be able to do. I suspect a
good implementation would be to run a collection daemon on each CPU and
make the delivery be cpu-local. That's sounding more like relayfs than
netlink.
Yup...the per-cpu, high speed requirements are up relayfs' alley, unless
Jamal or netlink folks
are planning something (or can shed light on) how large flows can be
managed over netlink. I suspect
this discussion has happened before :-)
On Thu, 29 Jun 2006 15:10:31 -0400
Shailabh Nagar [off-list ref] wrote:
quoted
quoted
I agree, and I'm viewing this as blocking the taskstats merge. Because if
this _is_ a problem then it's a big one because fixing it will be
intrusive, and might well involve userspace-visible changes.
First off, just a reminder that this is inherently a netlink flow
control issue...which was being exacerbated
earlier by taskstats decision to send per-tgid data (no longer the case).
But I'd like to know whats our target here ? How many messages per
second do we want to be able to be sent
and received without risking any loss of data ? Netlink will lose
messages at a high enough rate so the design point
will need to be known a bit.
For statistics type usage of the genetlink/netlink, I would have thought
that userspace, provided it is reliably informed
about the loss of data through ENOBUFS, could take measures to just
account for the missing data and carry on ?
Could be so. But we need to understand how significant the impact of this
will be in practice.
We could find, once this is deployed is real production environments on
large machines that the data loss is sufficiently common and sufficiently
serious that the feature needs a lot of rework.
Now there's always a risk of that sort of thing happening with all
features, but it's usually not this evident so early in the development
process. We need to get a better understanding of the risk before
proceeding too far.
Ok.
I suppose we should first determine what number of tasks can be
forked/exited at a sustained rate
on these m/c's and that would be one upper bound.
Paul, Chris, Jay,
What total exit rate would be a good upper bound ? How much memory do
these 1024 CPU machines
have (in high end configurations, not just based on 64-bit
addressability) and how many tasks can actually be
forked/exited in such a machine ?
And there's always a 100% reliable fix for this: throttling. Make the
sender of the messages block until the consumer can catch up. In some
situations, that is what people will want to be able to do.
Is this really an option for taskstats ? Allowing exits to get throttled
? I suppose its one way
but seems like overkill for something like stats.
I suspect a
good implementation would be to run a collection daemon on each CPU and
make the delivery be cpu-local. That's sounding more like relayfs than
netlink.
Yup...per-cpu, high speed delivery is looking like relayfs alright.
One option that we've not explored in detail is the "dump" functionality
of genetlink which allows
kernel space to keep getting called with skb's to fill until its done.
How much buffering that affords us
in the face of a slow user is not known. But if we're discussing large
exit rates happening in a burst, not
a sustained way, that may be one way out.
Jamal,
any thoughts on the flow control capabilities of netlink that apply here
? Usage of the connection is to
supply statistics data to userspace.
--Shailabh
From: Andrew Morton <hidden> Date: 2006-06-29 20:01:29
On Thu, 29 Jun 2006 15:43:41 -0400
Shailabh Nagar [off-list ref] wrote:
quoted
Could be so. But we need to understand how significant the impact of this
will be in practice.
We could find, once this is deployed is real production environments on
large machines that the data loss is sufficiently common and sufficiently
serious that the feature needs a lot of rework.
Now there's always a risk of that sort of thing happening with all
features, but it's usually not this evident so early in the development
process. We need to get a better understanding of the risk before
proceeding too far.
quoted
And there's always a 100% reliable fix for this: throttling. Make the
sender of the messages block until the consumer can catch up.
Is blocking exits an option ?
I think it has to be an option. I'm sure that some peope under some
circumstances will just want to collect all the data, thank you very much.
And I doubt if it'll be a performance problem for them - the amount of CPU
time per exit will be small - if you're exitting at great frequency then the
stats collecion overhead rises proportionately. That is to be expected.
There will be buffering in the channel, so we'd expect to gather thousands
of records per context switch.
quoted
In some
situations, that is what people will want to be able to do. I suspect a
good implementation would be to run a collection daemon on each CPU and
make the delivery be cpu-local. That's sounding more like relayfs than
netlink.
Yup...the per-cpu, high speed requirements are up relayfs' alley, unless
Jamal or netlink folks
are planning something (or can shed light on) how large flows can be
managed over netlink. I suspect
this discussion has happened before :-)
From: Paul Jackson <hidden> Date: 2006-06-29 21:22:57
Shailabh wrote:
How much memory do these 1024 CPU machines have
From:
http://www.hpcwire.com/hpc/653963.html (May 12, 2006)
SGI has already shipped more than a dozen SGI systems with
over a terabyte of memory and about a hundred systems of half
a terabyte or larger. But the new Altix will have much larger
memory capacities. The systems SGI has in mind will scale to tens
of terabytes and beyond. In fact, a few SGI customers are already
testing with systems in the 10-terabyte range. "The largest we
have shipped is a 13-terabyte memory system for the Japan Atomic
Energy Agency," said [SGI CTO Dr. Eng Lim] Goh.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson [off-list ref] 1.925.600.0401
Yup...the per-cpu, high speed requirements are up relayfs' alley, unless
Jamal or netlink folks
are planning something (or can shed light on) how large flows can be
managed over netlink. I suspect
this discussion has happened before :-)
yeah.
And now I remember why I didn't go down that path earlier. Relayfs is one-way
kernel->user and lacks the ability to send query commands from user space
that we need. Either we would need to send commands up through a separate interface
(even a syscall) or try and ensure that the exiting genetlink interface can
scale better with message volume (including throttling).
--Shailabh
On Thu, 2006-29-06 at 16:01 -0400, Shailabh Nagar wrote:
Jamal,
any thoughts on the flow control capabilities of netlink that apply here
? Usage of the connection is to supply statistics data to userspace.
if you want reliable delivery, then you cant just depend on async events
from the kernel -> user - which i am assuming is the way stats get
delivered as processes exit? Sorry, i dont remember the details. You
need some synchronous scheme to ask the kernel to do a "get" or "dump".
Lets be clear about one thing:
The problem really has nothing to do with gen/netlink or any other
scheme you use;->
It has everything to do with reliability implications and the fact
that you need to assume memory is a finite resource - at one point
or another you will run out of memory ;-> And of course then messages
will be lost. So for gen/netlink, just make sure you have large socket
buffer and you would most likely be fine.
I havent seen how the numbers were reached: But if you say you receive
14K exits/sec each of which is a 50B message, I would think a 1M socket
buffer would be plenty.
You can find out about lack of memory in netlink when you get a ENOBUFS.
As an example, you should then do a kernel query. Clearly if you do a
query of that sort, you may not want to find obsolete info. Therefore,
as a suggestion, you may want to keep sequence numbers of sorts as
markers. Perhaps keep a 32-bit field which monotically increases per
process exit or use the pid as the sequence number etc..
As for throttling - Shailabh, I think we talked about this:
- You could maintain info using some thresholds and timer. Then
when a timer expires or threshold is exceeded send to user space.
BTW, where is the doc fixes ? ;->
cheers,
jamal
On Thu, 2006-29-06 at 18:13 -0400, Shailabh Nagar wrote:
And now I remember why I didn't go down that path earlier. Relayfs is one-way
kernel->user and lacks the ability to send query commands from user space
that we need. Either we would need to send commands up through a separate interface
(even a syscall) or try and ensure that the exiting genetlink interface can
scale better with message volume (including throttling).
Refer to my other email - whatever it takes to store "bulk" data in the
kernel is subject to the constraint of the fact memory is finite.
You can send messages from the kernel in sizes constrained by the memory
socket size. You can tune the socket size.
cheers,
jamal
On Thu, 2006-29-06 at 16:01 -0400, Shailabh Nagar wrote:
quoted
Jamal,
any thoughts on the flow control capabilities of netlink that apply here
? Usage of the connection is to supply statistics data to userspace.
if you want reliable delivery, then you cant just depend on async events
from the kernel -> user - which i am assuming is the way stats get
delivered as processes exit?
Yes.
Sorry, i dont remember the details. You
need some synchronous scheme to ask the kernel to do a "get" or "dump".
Oh, yes. Dump is synchronous. So it won't be useful unless we buffer
task exit records within
taskstats.
Lets be clear about one thing:
The problem really has nothing to do with gen/netlink or any other
scheme you use;->
It has everything to do with reliability implications and the fact
that you need to assume memory is a finite resource - at one point
or another you will run out of memory ;-> And of course then messages
will be lost. So for gen/netlink, just make sure you have large socket
buffer and you would most likely be fine.
I havent seen how the numbers were reached: But if you say you receive
14K exits/sec each of which is a 50B message, I would think a 1M socket
buffer would be plenty.
The rates (or upper bounds) that are being discussed here, as of now,
are 1000 exits/sec/CPU for
1024 CPU systems. That would be roughly 1M exits/system *
248Bytes/message = 248 MB/sec.
You can find out about lack of memory in netlink when you get a ENOBUFS.
As an example, you should then do a kernel query. Clearly if you do a
query of that sort, you may not want to find obsolete info. Therefore,
as a suggestion, you may want to keep sequence numbers of sorts as
markers. Perhaps keep a 32-bit field which monotically increases per
process exit or use the pid as the sequence number etc..
As for throttling - Shailabh, I think we talked about this:
- You could maintain info using some thresholds and timer. Then
when a timer expires or threshold is exceeded send to user space.
Hmm. So we could buffer the per-task exit data within taskstats (the mem
consumption would grow
but thats probably not a problem) and then send it out later.
Jay - would not getting exit data soon after exit be a problem for CSA ?
I'm guessing not, if the
timeout is kept small enough. Internally, taskstats could always pace
its sends so that "too much"
isn't sent out at one shot.
--Shailabh
From: Andrew Morton <hidden> Date: 2006-06-30 01:02:29
Shailabh Nagar [off-list ref] wrote:
The rates (or upper bounds) that are being discussed here, as of now,
are 1000 exits/sec/CPU for
1024 CPU systems. That would be roughly 1M exits/system *
248Bytes/message = 248 MB/sec.
I think it's worth differentiating between burst rates and sustained rates
here.
One could easily imagine 10,000 threads all exiting at once, and the user
being interested in reliably collecting the results.
But if the machine is _sustaining_ such a high rate then that means that
these exiting tasks all have a teeny runtime and the user isn't going to be
interested in the per-thread statistics.
So if we can detect the silly sustained-high-exit-rate scenario then it
seems to me quite legitimate to do some aggressive data reduction on that.
Like, a single message which says "20,000 sub-millisecond-runtime tasks
exited in the past second" or something.
The rates (or upper bounds) that are being discussed here, as of now,
are 1000 exits/sec/CPU for
1024 CPU systems. That would be roughly 1M exits/system *
248Bytes/message = 248 MB/sec.
I think it's worth differentiating between burst rates and sustained rates
here.
One could easily imagine 10,000 threads all exiting at once, and the user
being interested in reliably collecting the results.
But if the machine is _sustaining_ such a high rate then that means that
these exiting tasks all have a teeny runtime and the user isn't going to be
interested in the per-thread statistics.
So if we can detect the silly sustained-high-exit-rate scenario then it
seems to me quite legitimate to do some aggressive data reduction on that.
Like, a single message which says "20,000 sub-millisecond-runtime tasks
exited in the past second" or something.
The "buffering within taskstats" might be a way out then.
As long as the user is willing to pay the price in terms of memory, we
can collect the exiting task's
taskstats data but not send it immediately (taskstats_cache would grow)
unless a high water mark had
been crossed. Otherwise a timer event would do the sends of accumalated
taskstats (not all at once but
iteratively if necessary).
At task exit, despite doing a few rounds of sending of pending data, if
netlink were still reporting errors
then it would be a sign of unsustainable rate and the pending queue
could be dropped and a message
like you suggest could be sent.
Thoughts ?
--Shailabh
On Thu, 2006-29-06 at 21:11 -0400, Shailabh Nagar wrote:
Andrew Morton wrote:
quoted
Shailabh Nagar [off-list ref] wrote:
[..]
quoted
So if we can detect the silly sustained-high-exit-rate scenario then it
seems to me quite legitimate to do some aggressive data reduction on that.
Like, a single message which says "20,000 sub-millisecond-runtime tasks
exited in the past second" or something.
The "buffering within taskstats" might be a way out then.
Thats what it looks like.
As long as the user is willing to pay the price in terms of memory,
You may wanna draw a line to the upper limit - maybe even allocate slab
space.
we can collect the exiting task's taskstats data but not send it
immediately (taskstats_cache would grow)
unless a high water mark had been crossed. Otherwise a timer event would do the
sends of accumalated taskstats (not all at once but
iteratively if necessary).
Sounds reasonable. Thats what xfrm events do. Try to have those
parameters settable because different machines or users may have
different view as to what is proper - maybe even as simple as sysctl.
At task exit, despite doing a few rounds of sending of pending data, if
netlink were still reporting errors
then it would be a sign of unsustainable rate and the pending queue
could be dropped and a message like you suggest could be sent.
When you send inside the kernel - you will get an error if there's
problems sending to the socket queue. So you may wanna use that info
to release the kernel allocated entries or keep them for a little
longer.
Hopefully that helps.
cheers,
jamal
From: Paul Jackson <hidden> Date: 2006-06-30 02:26:00
Andrew wrote:
Like, a single message which says "20,000 sub-millisecond-runtime tasks
exited in the past second" or something.
System wide accumulation of such data in the exit() code path still
risks being a bottleneck, just a bit later on.
I'm more inclined now to look for ways to disable collection on some
CPUs, and/or to allow for multiple streams in the future, as need be,
along the lines of Shailabh's multiple TASKSTATS_LISTEN_GROUPs.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson [off-list ref] 1.925.600.0401
From: Paul Jackson <hidden> Date: 2006-06-30 02:44:15
Andrew wrote:
Nah. Stick it in the same cacheline as tasklist_lock (I'm amazed that
we've continued to get away with a global lock for that).
Yes - a bit amazing. But no sense compounding the problem now.
We shouldn't be adding global locks/modifiable data in the
fork/exit code path if we can help it, without at least
providing some simple way to ameliorate the problem when
folks do start hitting it.
--
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson [off-list ref] 1.925.600.0401
On Thu, 2006-29-06 at 21:11 -0400, Shailabh Nagar wrote:
quoted
Andrew Morton wrote:
quoted
Shailabh Nagar [off-list ref] wrote:
[..]
quoted
quoted
So if we can detect the silly sustained-high-exit-rate scenario then it
seems to me quite legitimate to do some aggressive data reduction on that.
Like, a single message which says "20,000 sub-millisecond-runtime tasks
exited in the past second" or something.
The "buffering within taskstats" might be a way out then.
Thats what it looks like.
quoted
As long as the user is willing to pay the price in terms of memory,
You may wanna draw a line to the upper limit - maybe even allocate slab
space.
Didn't quite understand...could you please elaborate ?
Today we have a slab cache from which the taskstats structure gets
allocated at the beginning
of the exit() path.
The upper limit to which you refer is the amount of slab memory the user
is willing to be used
to store the bursty traffic ?
quoted
we can collect the exiting task's taskstats data but not send it
immediately (taskstats_cache would grow)
unless a high water mark had been crossed. Otherwise a timer event would do the
sends of accumalated taskstats (not all at once but
iteratively if necessary).
Sounds reasonable. Thats what xfrm events do.
Try to have those
parameters settable because different machines or users may have
different view as to what is proper - maybe even as simple as sysctl.
Sounds good.
quoted
At task exit, despite doing a few rounds of sending of pending data, if
netlink were still reporting errors
then it would be a sign of unsustainable rate and the pending queue
could be dropped and a message like you suggest could be sent.
When you send inside the kernel - you will get an error if there's
problems sending to the socket queue. So you may wanna use that info
to release the kernel allocated entries or keep them for a little
longer.
Hopefully that helps.
Yes it does. Thanks for the tips.
Will code up something and send out so this can become more concrete.
--Shailabh
On Thu, 2006-29-06 at 23:01 -0400, Shailabh Nagar wrote:
jamal wrote:
quoted
quoted
As long as the user is willing to pay the price in terms of memory,
You may wanna draw a line to the upper limit - maybe even allocate slab
space.
Didn't quite understand...could you please elaborate ?
Today we have a slab cache from which the taskstats structure gets
allocated at the beginning
of the exit() path.
The upper limit to which you refer is the amount of slab memory the user
is willing to be used
to store the bursty traffic ?
I think you have it fine already if you have a slab - as long as you
know you will run out of space and have some strategy to deal with
such boundary conditions. I was only reacting to your statement
"As long as the user is willing to pay the price in terms of memory"
I think you meant that a user could adjust the slab size on bootup etc,
but it is finite in size.
cheers,
jamal