Here is a preview version. It provides restricted set of functionality.
I would like to collect feedback about this idea.
Currently we use the proc file system, where all information are
presented in text files, what is convenient for humans. But if we need
to get information about processes from code (e.g. in C), the procfs
doesn't look so cool.
From code we would prefer to get information in binary format and to be
able to specify which information and for which tasks are required. Here
is a new interface with all these features, which is called task_diag.
In addition it's much faster than procfs.
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
A respone is a set of netlink messages. Each message describes one task.
All task properties are divided on groups. A message contains the
TASK_DIAG_MSG group and other groups if they have been requested in
show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
response will contain the TASK_DIAG_CRED group which is described by the
task_diag_creds structure.
struct task_diag_msg {
__u32 tgid;
__u32 pid;
__u32 ppid;
__u32 tpid;
__u32 sid;
__u32 pgid;
__u8 state;
char comm[TASK_DIAG_COMM_LEN];
};
Another good feature of task_diag is an ability to request information
for a few processes. Currently here are two stratgies
TASK_DIAG_DUMP_ALL - get information for all tasks
TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
tasks
The task diag is much faster than the proc file system. We don't need to
create a new file descriptor for each task. We need to send a request
and get a response. It allows to get information for a few task in one
request-response iteration.
I have compared performance of procfs and task-diag for the
"ps ax -o pid,ppid" command.
A test stand contains 10348 processes.
$ ps ax -o pid,ppid | wc -l
10348
$ time ps ax -o pid,ppid > /dev/null
real 0m1.073s
user 0m0.086s
sys 0m0.903s
$ time ./task_diag_all > /dev/null
real 0m0.037s
user 0m0.004s
sys 0m0.020s
And here are statistics about syscalls which were called by each
command.
$ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid 2>&1 | grep syscalls | sort -n -r | head -n 5
20,713 syscalls:sys_exit_open
20,710 syscalls:sys_exit_close
20,708 syscalls:sys_exit_read
10,348 syscalls:sys_exit_newstat
31 syscalls:sys_exit_write
$ perf stat -e syscalls:sys_exit* -- ./task_diag_all 2>&1 | grep syscalls | sort -n -r | head -n 5
114 syscalls:sys_exit_recvfrom
49 syscalls:sys_exit_write
8 syscalls:sys_exit_mmap
4 syscalls:sys_exit_mprotect
3 syscalls:sys_exit_newfstat
You can find the test program from this experiment in the last patch.
The idea of this functionality was suggested by Pavel Emelyanov
(xemul@), when he found that operations with /proc forms a significant
part of a checkpointing time.
Ten years ago here was attempt to add a netlink interface to access to /proc
information:
http://lwn.net/Articles/99600/
Signed-off-by: Andrey Vagin <redacted>
git repo: https://github.com/avagin/linux-task-diag
Andrey Vagin (7):
[RFC] kernel: add a netlink interface to get information about tasks
kernel: move next_tgid from fs/proc
task-diag: add ability to get information about all tasks
task-diag: add a new group to get process credentials
kernel: add ability to iterate children of a specified task
task_diag: add ability to dump children
selftest: check the task_diag functinonality
fs/proc/array.c | 58 +---
fs/proc/base.c | 43 ---
include/linux/proc_fs.h | 13 +
include/uapi/linux/taskdiag.h | 89 ++++++
init/Kconfig | 12 +
kernel/Makefile | 1 +
kernel/pid.c | 94 ++++++
kernel/taskdiag.c | 343 +++++++++++++++++++++
tools/testing/selftests/task_diag/Makefile | 16 +
tools/testing/selftests/task_diag/task_diag.c | 59 ++++
tools/testing/selftests/task_diag/task_diag_all.c | 82 +++++
tools/testing/selftests/task_diag/task_diag_comm.c | 195 ++++++++++++
tools/testing/selftests/task_diag/task_diag_comm.h | 47 +++
tools/testing/selftests/task_diag/taskdiag.h | 1 +
14 files changed, 967 insertions(+), 86 deletions(-)
create mode 100644 include/uapi/linux/taskdiag.h
create mode 100644 kernel/taskdiag.c
create mode 100644 tools/testing/selftests/task_diag/Makefile
create mode 100644 tools/testing/selftests/task_diag/task_diag.c
create mode 100644 tools/testing/selftests/task_diag/task_diag_all.c
create mode 100644 tools/testing/selftests/task_diag/task_diag_comm.c
create mode 100644 tools/testing/selftests/task_diag/task_diag_comm.h
create mode 120000 tools/testing/selftests/task_diag/taskdiag.h
Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Cyrill Gorcunov <redacted>
Cc: Pavel Emelyanov <redacted>
Cc: Roger Luethi <redacted>
--
2.1.0
For that we need to set NLM_F_DUMP. Currently here are no
filters. Any suggestions are welcome.
I think we can add request for children, threads, session or group
members.
Signed-off-by: Andrey Vagin <redacted>
---
kernel/taskdiag.c | 41 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 41 insertions(+)
Now we can dump all task or children of a specified task.
It's an example how this interface can be expanded for different
use-cases.
Signed-off-by: Andrey Vagin <redacted>
---
include/uapi/linux/taskdiag.h | 1 +
kernel/taskdiag.c | 83 +++++++++++++++++++++++++++++++++++++------
2 files changed, 73 insertions(+), 11 deletions(-)
@@ -9,11 +9,14 @@enum{/* optional attributes which can be specified in show_flags */+TASK_DIAG_CRED,/* other attributes */TASK_DIAG_MSG=64,};+#define TASK_DIAG_SHOW_CRED (1ULL << TASK_DIAG_CRED)+enum{TASK_DIAG_RUNNING,TASK_DIAG_INTERRUPTIBLE,
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
task_diag is a new interface which is going to raplace the proc file
system in cases when we need to get information in a binary format.
A request messages is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags;
__u64 dump_stratagy;
__u32 pid;
};
A respone is a set of netlink messages. Each message describes one task.
All task properties are divided on groups. A message contains the
TASK_DIAG_MSG group, and other groups if they have been requested in
show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
response will contain the TASK_DIAG_CRED group which is described by the
task_diag_creds structure.
struct task_diag_msg {
__u32 tgid;
__u32 pid;
__u32 ppid;
__u32 tpid;
__u32 sid;
__u32 pgid;
__u8 state;
char comm[TASK_DIAG_COMM_LEN];
};
The dump_stratagy field will be used in following patches to request
information for a group of processes.
Signed-off-by: Andrey Vagin <redacted>
---
include/uapi/linux/taskdiag.h | 64 +++++++++++++++
init/Kconfig | 12 +++
kernel/Makefile | 1 +
kernel/taskdiag.c | 179 ++++++++++++++++++++++++++++++++++++++++++
4 files changed, 256 insertions(+)
create mode 100644 include/uapi/linux/taskdiag.h
create mode 100644 kernel/taskdiag.c
The interface is similar with the tgid iterator. It is used in
procfs and it will be used in task_diag.
Signed-off-by: Andrey Vagin <redacted>
---
fs/proc/array.c | 58 +++++++++++++------------------------------------
include/linux/proc_fs.h | 6 +++++
kernel/pid.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 76 insertions(+), 43 deletions(-)
@@ -606,6 +606,61 @@ retry:returniter;}+structchild_iternext_child(structchild_iteriter)+{+structtask_struct*task;+loff_tpos=iter.pos;++read_lock(&tasklist_lock);++/*+*Letstrytocontinuesearchingfirst,thisgives+*ussignificantspeeduponchildren-richprocesses.+*/+if(iter.task){+task=iter.task;+if(task&&task->real_parent==iter.parent&&+!(list_empty(&task->sibling))){+if(list_is_last(&task->sibling,&iter.parent->children)){+task=NULL;+gotoout;+}+task=list_first_entry(&task->sibling,+structtask_struct,sibling);+gotoout;+}+}++/*+*Slowsearchcase.+*+*Wemightmisssomechildrenhereifchildren+*areexitedwhilewewerenotholdingthelock,+*butitwasneverpromisedtobeaccuratethat+*much.+*+*"Just suppose that the parent sleeps, but N children+*exitafterweprintedtheirtids.Nowtheslowpaths+*skipsNextrachildren,wemissNtasks." (c)+*+*Sooneneedtostoporfreezetheleaderandall+*itschildrentogetapreciseresult.+*/+list_for_each_entry(task,&iter.parent->children,sibling){+if(pos--==0)+gotoout;+}+task=NULL;+out:+if(iter.task)+put_task_struct(iter.task);+if(task)+get_task_struct(task);+iter.task=task;+read_unlock(&tasklist_lock);+returniter;+}+/**Thepidhashtableisscaledaccordingtotheamountofmemoryinthe*machine.Fromaminimumof16slotsupto4096slotsatonegigabyteor
@@ -2795,49 +2795,6 @@ out:returnERR_PTR(result);}-/*-*Findthefirsttaskwithtgid>=tgid-*-*/-structtgid_iter{-unsignedinttgid;-structtask_struct*task;-};-staticstructtgid_iternext_tgid(structpid_namespace*ns,structtgid_iteriter)-{-structpid*pid;--if(iter.task)-put_task_struct(iter.task);-rcu_read_lock();-retry:-iter.task=NULL;-pid=find_ge_pid(iter.tgid,ns);-if(pid){-iter.tgid=pid_nr_ns(pid,ns);-iter.task=pid_task(pid,PIDTYPE_PID);-/* What we to know is if the pid we have find is the-*pidofathread_group_leader.Testingfortask-*beingathread_group_leaderistheobviousthing-*todobutthereisawindowwhenitfails,dueto-*thepidtransferlogicinde_thread.-*-*Soweperformthestraightforwardtestofseeing-*ifthepidwehavefoundisthepidofathread-*groupleader,anddon'tworryifthetaskwehave-*founddoesn'thappentobeathreadgroupleader.-*Aswedon'tcareinthecaseofreaddir.-*/-if(!iter.task||!has_group_leader_pid(iter.task)){-iter.tgid+=1;-gotoretry;-}-get_task_struct(iter.task);-}-rcu_read_unlock();-returniter;-}-#define TGID_OFFSET (FIRST_PROCESS_ENTRY + 2)/* for the /proc/ directory itself, after non-process stuff has been done */
@@ -568,6 +568,45 @@ struct pid *find_ge_pid(int nr, struct pid_namespace *ns)}/*+*Findthefirsttaskwithtgid>=tgid+*+*/+structtgid_iternext_tgid(structpid_namespace*ns,structtgid_iteriter)+{+structpid*pid;++if(iter.task)+put_task_struct(iter.task);+rcu_read_lock();+retry:+iter.task=NULL;+pid=find_ge_pid(iter.tgid,ns);+if(pid){+iter.tgid=pid_nr_ns(pid,ns);+iter.task=pid_task(pid,PIDTYPE_PID);+/* What we to know is if the pid we have find is the+*pidofathread_group_leader.Testingfortask+*beingathread_group_leaderistheobviousthing+*todobutthereisawindowwhenitfails,dueto+*thepidtransferlogicinde_thread.+*+*Soweperformthestraightforwardtestofseeing+*ifthepidwehavefoundisthepidofathread+*groupleader,anddon'tworryifthetaskwehave+*founddoesn'thappentobeathreadgroupleader.+*Aswedon'tcareinthecaseofreaddir.+*/+if(!iter.task||!has_group_leader_pid(iter.task)){+iter.tgid+=1;+gotoretry;+}+get_task_struct(iter.task);+}+rcu_read_unlock();+returniter;+}++/**Thepidhashtableisscaledaccordingtotheamountofmemoryinthe*machine.Fromaminimumof16slotsupto4096slotsatonegigabyteor*more.
On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
Can you explain how the interface relates to the 'taskstats' genetlink
API? Did you consider extending that interface to provide the
information you need instead of basing on the socket-diag?
Arnd
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-02-17 19:05:54
On Feb 17, 2015 12:40 AM, "Andrey Vagin" [off-list ref] wrote:
Here is a preview version. It provides restricted set of functionality.
I would like to collect feedback about this idea.
Currently we use the proc file system, where all information are
presented in text files, what is convenient for humans. But if we need
to get information about processes from code (e.g. in C), the procfs
doesn't look so cool.
From code we would prefer to get information in binary format and to be
able to specify which information and for which tasks are required. Here
is a new interface with all these features, which is called task_diag.
In addition it's much faster than procfs.
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
A respone is a set of netlink messages. Each message describes one task.
All task properties are divided on groups. A message contains the
TASK_DIAG_MSG group and other groups if they have been requested in
show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
response will contain the TASK_DIAG_CRED group which is described by the
task_diag_creds structure.
struct task_diag_msg {
__u32 tgid;
__u32 pid;
__u32 ppid;
__u32 tpid;
__u32 sid;
__u32 pgid;
__u8 state;
char comm[TASK_DIAG_COMM_LEN];
};
Another good feature of task_diag is an ability to request information
for a few processes. Currently here are two stratgies
TASK_DIAG_DUMP_ALL - get information for all tasks
TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
tasks
The task diag is much faster than the proc file system. We don't need to
create a new file descriptor for each task. We need to send a request
and get a response. It allows to get information for a few task in one
request-response iteration.
I have compared performance of procfs and task-diag for the
"ps ax -o pid,ppid" command.
A test stand contains 10348 processes.
$ ps ax -o pid,ppid | wc -l
10348
$ time ps ax -o pid,ppid > /dev/null
real 0m1.073s
user 0m0.086s
sys 0m0.903s
$ time ./task_diag_all > /dev/null
real 0m0.037s
user 0m0.004s
sys 0m0.020s
And here are statistics about syscalls which were called by each
command.
$ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid 2>&1 | grep syscalls | sort -n -r | head -n 5
20,713 syscalls:sys_exit_open
20,710 syscalls:sys_exit_close
20,708 syscalls:sys_exit_read
10,348 syscalls:sys_exit_newstat
31 syscalls:sys_exit_write
$ perf stat -e syscalls:sys_exit* -- ./task_diag_all 2>&1 | grep syscalls | sort -n -r | head -n 5
114 syscalls:sys_exit_recvfrom
49 syscalls:sys_exit_write
8 syscalls:sys_exit_mmap
4 syscalls:sys_exit_mprotect
3 syscalls:sys_exit_newfstat
You can find the test program from this experiment in the last patch.
The idea of this functionality was suggested by Pavel Emelyanov
(xemul@), when he found that operations with /proc forms a significant
part of a checkpointing time.
Ten years ago here was attempt to add a netlink interface to access to /proc
information:
http://lwn.net/Articles/99600/
I don't suppose this could use real syscalls instead of netlink. If
nothing else, netlink seems to conflate pid and net namespaces.
Also, using an asynchronous interface (send, poll?, recv) for
something that's inherently synchronous (as the kernel a local
question) seems awkward to me.
--Andy
From: Andrew Vagin <hidden> Date: 2015-02-17 21:33:26
On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
quoted
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
Can you explain how the interface relates to the 'taskstats' genetlink
API? Did you consider extending that interface to provide the
information you need instead of basing on the socket-diag?
It isn't based on the socket-diag, it looks like socket-diag.
Current task_diag registers a new genl family, but we can use the taskstats
family and add task_diag commands to it.
Thanks,
Andrew
On Wednesday 18 February 2015 00:33:13 Andrew Vagin wrote:
On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
quoted
On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
quoted
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
Can you explain how the interface relates to the 'taskstats' genetlink
API? Did you consider extending that interface to provide the
information you need instead of basing on the socket-diag?
It isn't based on the socket-diag, it looks like socket-diag.
Current task_diag registers a new genl family, but we can use the taskstats
family and add task_diag commands to it.
What I meant was more along the lines of making it look like taskstats
by adding new fields to 'struct taskstat' for what you want return.
I don't know if that is possible or a good idea for the information
you want to get out of the kernel, but it seems like a more natural
interface, as it already has some of the same data (comm, gid, pid,
ppid, ...).
Arnd
From: Andrew Vagin <hidden> Date: 2015-02-18 12:42:42
On Wed, Feb 18, 2015 at 12:06:40PM +0100, Arnd Bergmann wrote:
On Wednesday 18 February 2015 00:33:13 Andrew Vagin wrote:
quoted
On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
quoted
On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
quoted
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
Can you explain how the interface relates to the 'taskstats' genetlink
API? Did you consider extending that interface to provide the
information you need instead of basing on the socket-diag?
It isn't based on the socket-diag, it looks like socket-diag.
Current task_diag registers a new genl family, but we can use the taskstats
family and add task_diag commands to it.
What I meant was more along the lines of making it look like taskstats
by adding new fields to 'struct taskstat' for what you want return.
I don't know if that is possible or a good idea for the information
you want to get out of the kernel, but it seems like a more natural
interface, as it already has some of the same data (comm, gid, pid,
ppid, ...).
Now I see what you mean. task_diag has more flexible and universal
interface than taskstat. A response of taskstat only contains a
taskstats structure. A response of taskdiag can contains a few types of
properties. Each type is described by its own structure.
Curently here are only two groups of parameters: task_diag_msg and
task_diag_creds.
task_diag_msg contains a few basic parameters.
task_diag_creds contains credentials.
I'm going to add other groups to describe all kind of task properties
which currently are presented in procfs (e.g. /proc/pid/maps,
/proc/pid/fding/*, /proc/pid/status, etc).
One of features of task_diag is an ability to choose which information
are required. This allows to minimize a response size and a time, which
is requred to fill this response.
struct task_diag_msg {
__u32 tgid;
__u32 pid;
__u32 ppid;
__u32 tpid;
__u32 sid;
__u32 pgid;
__u8 state;
char comm[TASK_DIAG_COMM_LEN];
};
struct task_diag_creds {
struct task_diag_caps cap_inheritable;
struct task_diag_caps cap_permitted;
struct task_diag_caps cap_effective;
struct task_diag_caps cap_bset;
__u32 uid;
__u32 euid;
__u32 suid;
__u32 fsuid;
__u32 gid;
__u32 egid;
__u32 sgid;
__u32 fsgid;
};
Thanks,
Andrew
From: Andrew Vagin <hidden> Date: 2015-02-18 14:27:27
On Tue, Feb 17, 2015 at 11:05:31AM -0800, Andy Lutomirski wrote:
On Feb 17, 2015 12:40 AM, "Andrey Vagin" [off-list ref] wrote:
quoted
Here is a preview version. It provides restricted set of functionality.
I would like to collect feedback about this idea.
Currently we use the proc file system, where all information are
presented in text files, what is convenient for humans. But if we need
to get information about processes from code (e.g. in C), the procfs
doesn't look so cool.
From code we would prefer to get information in binary format and to be
able to specify which information and for which tasks are required. Here
is a new interface with all these features, which is called task_diag.
In addition it's much faster than procfs.
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
A respone is a set of netlink messages. Each message describes one task.
All task properties are divided on groups. A message contains the
TASK_DIAG_MSG group and other groups if they have been requested in
show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
response will contain the TASK_DIAG_CRED group which is described by the
task_diag_creds structure.
struct task_diag_msg {
__u32 tgid;
__u32 pid;
__u32 ppid;
__u32 tpid;
__u32 sid;
__u32 pgid;
__u8 state;
char comm[TASK_DIAG_COMM_LEN];
};
Another good feature of task_diag is an ability to request information
for a few processes. Currently here are two stratgies
TASK_DIAG_DUMP_ALL - get information for all tasks
TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
tasks
The task diag is much faster than the proc file system. We don't need to
create a new file descriptor for each task. We need to send a request
and get a response. It allows to get information for a few task in one
request-response iteration.
I have compared performance of procfs and task-diag for the
"ps ax -o pid,ppid" command.
A test stand contains 10348 processes.
$ ps ax -o pid,ppid | wc -l
10348
$ time ps ax -o pid,ppid > /dev/null
real 0m1.073s
user 0m0.086s
sys 0m0.903s
$ time ./task_diag_all > /dev/null
real 0m0.037s
user 0m0.004s
sys 0m0.020s
And here are statistics about syscalls which were called by each
command.
$ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid 2>&1 | grep syscalls | sort -n -r | head -n 5
20,713 syscalls:sys_exit_open
20,710 syscalls:sys_exit_close
20,708 syscalls:sys_exit_read
10,348 syscalls:sys_exit_newstat
31 syscalls:sys_exit_write
$ perf stat -e syscalls:sys_exit* -- ./task_diag_all 2>&1 | grep syscalls | sort -n -r | head -n 5
114 syscalls:sys_exit_recvfrom
49 syscalls:sys_exit_write
8 syscalls:sys_exit_mmap
4 syscalls:sys_exit_mprotect
3 syscalls:sys_exit_newfstat
You can find the test program from this experiment in the last patch.
The idea of this functionality was suggested by Pavel Emelyanov
(xemul@), when he found that operations with /proc forms a significant
part of a checkpointing time.
Ten years ago here was attempt to add a netlink interface to access to /proc
information:
http://lwn.net/Articles/99600/
I don't suppose this could use real syscalls instead of netlink. If
nothing else, netlink seems to conflate pid and net namespaces.
What do you mean by "conflate pid and net namespaces"?
Also, using an asynchronous interface (send, poll?, recv) for
something that's inherently synchronous (as the kernel a local
question) seems awkward to me.
Actually all requests are handled synchronously. We call sendmsg to send
a request and it is handled in this syscall.
2) | netlink_sendmsg() {
2) | netlink_unicast() {
2) | taskdiag_doit() {
2) 2.153 us | task_diag_fill();
2) | netlink_unicast() {
2) 0.185 us | netlink_attachskb();
2) 0.291 us | __netlink_sendskb();
2) 2.452 us | }
2) + 33.625 us | }
2) + 54.611 us | }
2) + 76.370 us | }
2) | netlink_recvmsg() {
2) 1.178 us | skb_recv_datagram();
2) + 46.953 us | }
If we request information for a group of tasks (NLM_F_DUMP), a first
portion of data is filled from the sendmsg syscall. And then when we read
it, the kernel fills the next portion.
3) | netlink_sendmsg() {
3) | __netlink_dump_start() {
3) | netlink_dump() {
3) | taskdiag_dumpid() {
3) 0.685 us | task_diag_fill();
...
3) 0.224 us | task_diag_fill();
3) + 74.028 us | }
3) + 88.757 us | }
3) + 89.296 us | }
3) + 98.705 us | }
3) | netlink_recvmsg() {
3) | netlink_dump() {
3) | taskdiag_dumpid() {
3) 0.594 us | task_diag_fill();
...
3) 0.242 us | task_diag_fill();
3) + 60.634 us | }
3) + 72.803 us | }
3) + 88.005 us | }
3) | netlink_recvmsg() {
3) | netlink_dump() {
3) 2.403 us | taskdiag_dumpid();
3) + 26.236 us | }
3) + 40.522 us | }
0) + 20.407 us | netlink_recvmsg();
netlink is really good for this type of tasks. It allows to create an
extendable interface which can be easy customized for different needs.
I don't think that we would want to create another similar interface
just to be independent from network subsystem.
Thanks,
Andrew
On Wednesday 18 February 2015 15:42:11 Andrew Vagin wrote:
On Wed, Feb 18, 2015 at 12:06:40PM +0100, Arnd Bergmann wrote:
quoted
On Wednesday 18 February 2015 00:33:13 Andrew Vagin wrote:
quoted
On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
quoted
On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
quoted
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
Can you explain how the interface relates to the 'taskstats' genetlink
API? Did you consider extending that interface to provide the
information you need instead of basing on the socket-diag?
It isn't based on the socket-diag, it looks like socket-diag.
Current task_diag registers a new genl family, but we can use the taskstats
family and add task_diag commands to it.
What I meant was more along the lines of making it look like taskstats
by adding new fields to 'struct taskstat' for what you want return.
I don't know if that is possible or a good idea for the information
you want to get out of the kernel, but it seems like a more natural
interface, as it already has some of the same data (comm, gid, pid,
ppid, ...).
Now I see what you mean. task_diag has more flexible and universal
interface than taskstat. A response of taskstat only contains a
taskstats structure. A response of taskdiag can contains a few types of
properties. Each type is described by its own structure.
Right, so the question is whether that flexibility is actually required
here. Independent of which design you personally prefer, what are the
downsides of extending the existing but less flexible interface?
If it's good enough, that would seem to provide a more consistent
API, which in turn helps users understand the interface and use it
correctly.
Curently here are only two groups of parameters: task_diag_msg and
task_diag_creds.
task_diag_msg contains a few basic parameters.
task_diag_creds contains credentials.
I'm going to add other groups to describe all kind of task properties
which currently are presented in procfs (e.g. /proc/pid/maps,
/proc/pid/fding/*, /proc/pid/status, etc).
One of features of task_diag is an ability to choose which information
are required. This allows to minimize a response size and a time, which
is requred to fill this response.
I realize that you are trying to optimize for performance, but it
would be nice to quantify this if you want to argue for requiring
a split interface.
I guess this part would be a very natural extension to the
existing taskstats structure, and we should only add a new
one here if there are extremely good reasons for it.
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-02-19 01:19:03
On Feb 18, 2015 6:27 AM, "Andrew Vagin" [off-list ref] wrote:
On Tue, Feb 17, 2015 at 11:05:31AM -0800, Andy Lutomirski wrote:
quoted
On Feb 17, 2015 12:40 AM, "Andrey Vagin" [off-list ref] wrote:
quoted
Here is a preview version. It provides restricted set of functionality.
I would like to collect feedback about this idea.
Currently we use the proc file system, where all information are
presented in text files, what is convenient for humans. But if we need
to get information about processes from code (e.g. in C), the procfs
doesn't look so cool.
From code we would prefer to get information in binary format and to be
able to specify which information and for which tasks are required. Here
is a new interface with all these features, which is called task_diag.
In addition it's much faster than procfs.
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
A respone is a set of netlink messages. Each message describes one task.
All task properties are divided on groups. A message contains the
TASK_DIAG_MSG group and other groups if they have been requested in
show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
response will contain the TASK_DIAG_CRED group which is described by the
task_diag_creds structure.
struct task_diag_msg {
__u32 tgid;
__u32 pid;
__u32 ppid;
__u32 tpid;
__u32 sid;
__u32 pgid;
__u8 state;
char comm[TASK_DIAG_COMM_LEN];
};
Another good feature of task_diag is an ability to request information
for a few processes. Currently here are two stratgies
TASK_DIAG_DUMP_ALL - get information for all tasks
TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
tasks
The task diag is much faster than the proc file system. We don't need to
create a new file descriptor for each task. We need to send a request
and get a response. It allows to get information for a few task in one
request-response iteration.
I have compared performance of procfs and task-diag for the
"ps ax -o pid,ppid" command.
A test stand contains 10348 processes.
$ ps ax -o pid,ppid | wc -l
10348
$ time ps ax -o pid,ppid > /dev/null
real 0m1.073s
user 0m0.086s
sys 0m0.903s
$ time ./task_diag_all > /dev/null
real 0m0.037s
user 0m0.004s
sys 0m0.020s
And here are statistics about syscalls which were called by each
command.
$ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid 2>&1 | grep syscalls | sort -n -r | head -n 5
20,713 syscalls:sys_exit_open
20,710 syscalls:sys_exit_close
20,708 syscalls:sys_exit_read
10,348 syscalls:sys_exit_newstat
31 syscalls:sys_exit_write
$ perf stat -e syscalls:sys_exit* -- ./task_diag_all 2>&1 | grep syscalls | sort -n -r | head -n 5
114 syscalls:sys_exit_recvfrom
49 syscalls:sys_exit_write
8 syscalls:sys_exit_mmap
4 syscalls:sys_exit_mprotect
3 syscalls:sys_exit_newfstat
You can find the test program from this experiment in the last patch.
The idea of this functionality was suggested by Pavel Emelyanov
(xemul@), when he found that operations with /proc forms a significant
part of a checkpointing time.
Ten years ago here was attempt to add a netlink interface to access to /proc
information:
http://lwn.net/Articles/99600/
I don't suppose this could use real syscalls instead of netlink. If
nothing else, netlink seems to conflate pid and net namespaces.
What do you mean by "conflate pid and net namespaces"?
A netlink socket is bound to a network namespace, but you should be
returning data specific to a pid namespace.
On a related note, how does this interact with hidepid? More
generally, what privileges are you requiring to obtain what data?
quoted
Also, using an asynchronous interface (send, poll?, recv) for
something that's inherently synchronous (as the kernel a local
question) seems awkward to me.
Actually all requests are handled synchronously. We call sendmsg to send
a request and it is handled in this syscall.
2) | netlink_sendmsg() {
2) | netlink_unicast() {
2) | taskdiag_doit() {
2) 2.153 us | task_diag_fill();
2) | netlink_unicast() {
2) 0.185 us | netlink_attachskb();
2) 0.291 us | __netlink_sendskb();
2) 2.452 us | }
2) + 33.625 us | }
2) + 54.611 us | }
2) + 76.370 us | }
2) | netlink_recvmsg() {
2) 1.178 us | skb_recv_datagram();
2) + 46.953 us | }
If we request information for a group of tasks (NLM_F_DUMP), a first
portion of data is filled from the sendmsg syscall. And then when we read
it, the kernel fills the next portion.
3) | netlink_sendmsg() {
3) | __netlink_dump_start() {
3) | netlink_dump() {
3) | taskdiag_dumpid() {
3) 0.685 us | task_diag_fill();
...
3) 0.224 us | task_diag_fill();
3) + 74.028 us | }
3) + 88.757 us | }
3) + 89.296 us | }
3) + 98.705 us | }
3) | netlink_recvmsg() {
3) | netlink_dump() {
3) | taskdiag_dumpid() {
3) 0.594 us | task_diag_fill();
...
3) 0.242 us | task_diag_fill();
3) + 60.634 us | }
3) + 72.803 us | }
3) + 88.005 us | }
3) | netlink_recvmsg() {
3) | netlink_dump() {
3) 2.403 us | taskdiag_dumpid();
3) + 26.236 us | }
3) + 40.522 us | }
0) + 20.407 us | netlink_recvmsg();
netlink is really good for this type of tasks. It allows to create an
extendable interface which can be easy customized for different needs.
I don't think that we would want to create another similar interface
just to be independent from network subsystem.
I guess this is a bit streamy in that you ask one question and get
multiple answers.
From: Andrew Vagin <hidden> Date: 2015-02-19 14:05:03
On Wed, Feb 18, 2015 at 03:46:31PM +0100, Arnd Bergmann wrote:
On Wednesday 18 February 2015 15:42:11 Andrew Vagin wrote:
quoted
On Wed, Feb 18, 2015 at 12:06:40PM +0100, Arnd Bergmann wrote:
quoted
On Wednesday 18 February 2015 00:33:13 Andrew Vagin wrote:
quoted
On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
quoted
On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
quoted
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
Can you explain how the interface relates to the 'taskstats' genetlink
API? Did you consider extending that interface to provide the
information you need instead of basing on the socket-diag?
It isn't based on the socket-diag, it looks like socket-diag.
Current task_diag registers a new genl family, but we can use the taskstats
family and add task_diag commands to it.
What I meant was more along the lines of making it look like taskstats
by adding new fields to 'struct taskstat' for what you want return.
I don't know if that is possible or a good idea for the information
you want to get out of the kernel, but it seems like a more natural
interface, as it already has some of the same data (comm, gid, pid,
ppid, ...).
Now I see what you mean. task_diag has more flexible and universal
interface than taskstat. A response of taskstat only contains a
taskstats structure. A response of taskdiag can contains a few types of
properties. Each type is described by its own structure.
Right, so the question is whether that flexibility is actually required
here. Independent of which design you personally prefer, what are the
downsides of extending the existing but less flexible interface?
I have looked at taskstat once again.
The format of response messages for taskstat and taskdiag are the same.
It's a netlink message with a set of nested attributes. New attributes
can be added without breaking backward compatibility.
The request can be expanded to be able to specified which information is
required and for which tasks.
These two features allow to significantly improve performance, because
in this case we don't need to do a system call for each task.
I have done a few experiments to prove these words.
task_proc_all reads /proc/pid/stat for each tast
$ time ./task_proc_all > /dev/null
real 0m1.528s
user 0m0.016s
sys 0m1.341s
task_diag uses task_diag and requests information for each task
separately.
$ time ./task_diag > /dev/null
real 0m1.166s
user 0m0.024s
sys 0m1.127s
task_diag_all uses task_diag and requests information for all tasks in
one request.
$ time ./task_diag_all > /dev/null
real 0m0.077s
user 0m0.018s
sys 0m0.053s
So you can see that the ability to request information for a group of
tasks allows to be more effective.
The summary of this message is that we can use the interface of
taskstats with some extensions.
Arnd, thank you for your opinion and suggestions.
If it's good enough, that would seem to provide a more consistent
API, which in turn helps users understand the interface and use it
correctly.
quoted
Curently here are only two groups of parameters: task_diag_msg and
task_diag_creds.
task_diag_msg contains a few basic parameters.
task_diag_creds contains credentials.
I'm going to add other groups to describe all kind of task properties
which currently are presented in procfs (e.g. /proc/pid/maps,
/proc/pid/fding/*, /proc/pid/status, etc).
One of features of task_diag is an ability to choose which information
are required. This allows to minimize a response size and a time, which
is requred to fill this response.
I realize that you are trying to optimize for performance, but it
would be nice to quantify this if you want to argue for requiring
a split interface.
I guess this part would be a very natural extension to the
existing taskstats structure, and we should only add a new
one here if there are extremely good reasons for it.
The task_diag_msg structure contains properties which are used more
frequently than statistics from the taststats structure.
The size of the task_diag_msg structure is 44 bytes, the size of the
taststats structure 328. If we have more data, we need to do more
system calls. So I have done one more experiment to look how it affects
perfomance:
If we use the task_diag_msg structure:
$ time ./task_diag_all > /dev/null
real 0m0.077s
user 0m0.018s
sys 0m0.053s
If we use the taststats structure:
$ time ./task_diag_all > /dev/null
real 0m0.117s
user 0m0.029s
sys 0m0.085s
Thanks,
Andrew
From: Andrew Vagin <hidden> Date: 2015-02-19 21:39:47
On Wed, Feb 18, 2015 at 05:18:38PM -0800, Andy Lutomirski wrote:
On Feb 18, 2015 6:27 AM, "Andrew Vagin" [off-list ref] wrote:
quoted
On Tue, Feb 17, 2015 at 11:05:31AM -0800, Andy Lutomirski wrote:
quoted
On Feb 17, 2015 12:40 AM, "Andrey Vagin" [off-list ref] wrote:
quoted
Here is a preview version. It provides restricted set of functionality.
I would like to collect feedback about this idea.
Currently we use the proc file system, where all information are
presented in text files, what is convenient for humans. But if we need
to get information about processes from code (e.g. in C), the procfs
doesn't look so cool.
From code we would prefer to get information in binary format and to be
able to specify which information and for which tasks are required. Here
is a new interface with all these features, which is called task_diag.
In addition it's much faster than procfs.
task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.
A request is described by the task_diag_pid structure:
struct task_diag_pid {
__u64 show_flags; /* specify which information are required */
__u64 dump_stratagy; /* specify a group of processes */
__u32 pid;
};
A respone is a set of netlink messages. Each message describes one task.
All task properties are divided on groups. A message contains the
TASK_DIAG_MSG group and other groups if they have been requested in
show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
response will contain the TASK_DIAG_CRED group which is described by the
task_diag_creds structure.
struct task_diag_msg {
__u32 tgid;
__u32 pid;
__u32 ppid;
__u32 tpid;
__u32 sid;
__u32 pgid;
__u8 state;
char comm[TASK_DIAG_COMM_LEN];
};
Another good feature of task_diag is an ability to request information
for a few processes. Currently here are two stratgies
TASK_DIAG_DUMP_ALL - get information for all tasks
TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
tasks
The task diag is much faster than the proc file system. We don't need to
create a new file descriptor for each task. We need to send a request
and get a response. It allows to get information for a few task in one
request-response iteration.
I have compared performance of procfs and task-diag for the
"ps ax -o pid,ppid" command.
A test stand contains 10348 processes.
$ ps ax -o pid,ppid | wc -l
10348
$ time ps ax -o pid,ppid > /dev/null
real 0m1.073s
user 0m0.086s
sys 0m0.903s
$ time ./task_diag_all > /dev/null
real 0m0.037s
user 0m0.004s
sys 0m0.020s
And here are statistics about syscalls which were called by each
command.
$ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid 2>&1 | grep syscalls | sort -n -r | head -n 5
20,713 syscalls:sys_exit_open
20,710 syscalls:sys_exit_close
20,708 syscalls:sys_exit_read
10,348 syscalls:sys_exit_newstat
31 syscalls:sys_exit_write
$ perf stat -e syscalls:sys_exit* -- ./task_diag_all 2>&1 | grep syscalls | sort -n -r | head -n 5
114 syscalls:sys_exit_recvfrom
49 syscalls:sys_exit_write
8 syscalls:sys_exit_mmap
4 syscalls:sys_exit_mprotect
3 syscalls:sys_exit_newfstat
You can find the test program from this experiment in the last patch.
The idea of this functionality was suggested by Pavel Emelyanov
(xemul@), when he found that operations with /proc forms a significant
part of a checkpointing time.
Ten years ago here was attempt to add a netlink interface to access to /proc
information:
http://lwn.net/Articles/99600/
I don't suppose this could use real syscalls instead of netlink. If
nothing else, netlink seems to conflate pid and net namespaces.
What do you mean by "conflate pid and net namespaces"?
A netlink socket is bound to a network namespace, but you should be
returning data specific to a pid namespace.
Here is a good question. When we mount a procfs instance, the current
pidns is saved on a superblock. Then if we read data from
this procfs from another pidns, we will see pid-s from the pidns where
this procfs has been mounted.
$ unshare -p -- bash -c '(bash)'
$ cat /proc/self/status | grep ^Pid:
Pid: 15770
$ echo $$
1
A similar situation with socket_diag. A socket_diag socket is bound to a
network namespace. If we open a socket_diag socket and change a network
namespace, it will return infromation about the initial netns.
In this version I always use a current pid namespace.
But to be consistant with other kernel logic, a socket diag has to be
linked with a pidns where it has been created.
On a related note, how does this interact with hidepid? More
Currently it always work as procfs with hidepid = 2 (highest level of
security).
generally, what privileges are you requiring to obtain what data?
It dumps information only if ptrace_may_access(tsk, PTRACE_MODE_READ) returns true
quoted
quoted
Also, using an asynchronous interface (send, poll?, recv) for
something that's inherently synchronous (as the kernel a local
question) seems awkward to me.
Actually all requests are handled synchronously. We call sendmsg to send
a request and it is handled in this syscall.
2) | netlink_sendmsg() {
2) | netlink_unicast() {
2) | taskdiag_doit() {
2) 2.153 us | task_diag_fill();
2) | netlink_unicast() {
2) 0.185 us | netlink_attachskb();
2) 0.291 us | __netlink_sendskb();
2) 2.452 us | }
2) + 33.625 us | }
2) + 54.611 us | }
2) + 76.370 us | }
2) | netlink_recvmsg() {
2) 1.178 us | skb_recv_datagram();
2) + 46.953 us | }
If we request information for a group of tasks (NLM_F_DUMP), a first
portion of data is filled from the sendmsg syscall. And then when we read
it, the kernel fills the next portion.
3) | netlink_sendmsg() {
3) | __netlink_dump_start() {
3) | netlink_dump() {
3) | taskdiag_dumpid() {
3) 0.685 us | task_diag_fill();
...
3) 0.224 us | task_diag_fill();
3) + 74.028 us | }
3) + 88.757 us | }
3) + 89.296 us | }
3) + 98.705 us | }
3) | netlink_recvmsg() {
3) | netlink_dump() {
3) | taskdiag_dumpid() {
3) 0.594 us | task_diag_fill();
...
3) 0.242 us | task_diag_fill();
3) + 60.634 us | }
3) + 72.803 us | }
3) + 88.005 us | }
3) | netlink_recvmsg() {
3) | netlink_dump() {
3) 2.403 us | taskdiag_dumpid();
3) + 26.236 us | }
3) + 40.522 us | }
0) + 20.407 us | netlink_recvmsg();
netlink is really good for this type of tasks. It allows to create an
extendable interface which can be easy customized for different needs.
I don't think that we would want to create another similar interface
just to be independent from network subsystem.
I guess this is a bit streamy in that you ask one question and get
multiple answers.
It's like seq_file in procfs. The kernel allocates a buffer then fills
it, copies it into userspace, fills it again, ... repeats these actions.
And we can read data from file by portions.
Actually here is one more analogy. When we open a file in procfs,
we sends a request to the kernel and a file path is a request body in
this case. But in case of procfs, we can't construct requests, we only
have a set of predefined requests.
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-02-20 20:33:55
On Thu, Feb 19, 2015 at 1:39 PM, Andrew Vagin [off-list ref] wrote:
On Wed, Feb 18, 2015 at 05:18:38PM -0800, Andy Lutomirski wrote:
quoted
quoted
quoted
I don't suppose this could use real syscalls instead of netlink. If
nothing else, netlink seems to conflate pid and net namespaces.
What do you mean by "conflate pid and net namespaces"?
A netlink socket is bound to a network namespace, but you should be
returning data specific to a pid namespace.
Here is a good question. When we mount a procfs instance, the current
pidns is saved on a superblock. Then if we read data from
this procfs from another pidns, we will see pid-s from the pidns where
this procfs has been mounted.
$ unshare -p -- bash -c '(bash)'
$ cat /proc/self/status | grep ^Pid:
Pid: 15770
$ echo $$
1
A similar situation with socket_diag. A socket_diag socket is bound to a
network namespace. If we open a socket_diag socket and change a network
namespace, it will return infromation about the initial netns.
In this version I always use a current pid namespace.
But to be consistant with other kernel logic, a socket diag has to be
linked with a pidns where it has been created.
Attaching a pidns to every freshly created netlink socket seems odd,
but I don't see a better solution that still uses netlink.
quoted
On a related note, how does this interact with hidepid? More
Currently it always work as procfs with hidepid = 2 (highest level of
security).
quoted
generally, what privileges are you requiring to obtain what data?
It dumps information only if ptrace_may_access(tsk, PTRACE_MODE_READ) returns true
Sounds good to me.
quoted
quoted
quoted
Also, using an asynchronous interface (send, poll?, recv) for
something that's inherently synchronous (as the kernel a local
question) seems awkward to me.
Actually all requests are handled synchronously. We call sendmsg to send
a request and it is handled in this syscall.
2) | netlink_sendmsg() {
2) | netlink_unicast() {
2) | taskdiag_doit() {
2) 2.153 us | task_diag_fill();
2) | netlink_unicast() {
2) 0.185 us | netlink_attachskb();
2) 0.291 us | __netlink_sendskb();
2) 2.452 us | }
2) + 33.625 us | }
2) + 54.611 us | }
2) + 76.370 us | }
2) | netlink_recvmsg() {
2) 1.178 us | skb_recv_datagram();
2) + 46.953 us | }
If we request information for a group of tasks (NLM_F_DUMP), a first
portion of data is filled from the sendmsg syscall. And then when we read
it, the kernel fills the next portion.
3) | netlink_sendmsg() {
3) | __netlink_dump_start() {
3) | netlink_dump() {
3) | taskdiag_dumpid() {
3) 0.685 us | task_diag_fill();
...
3) 0.224 us | task_diag_fill();
3) + 74.028 us | }
3) + 88.757 us | }
3) + 89.296 us | }
3) + 98.705 us | }
3) | netlink_recvmsg() {
3) | netlink_dump() {
3) | taskdiag_dumpid() {
3) 0.594 us | task_diag_fill();
...
3) 0.242 us | task_diag_fill();
3) + 60.634 us | }
3) + 72.803 us | }
3) + 88.005 us | }
3) | netlink_recvmsg() {
3) | netlink_dump() {
3) 2.403 us | taskdiag_dumpid();
3) + 26.236 us | }
3) + 40.522 us | }
0) + 20.407 us | netlink_recvmsg();
netlink is really good for this type of tasks. It allows to create an
extendable interface which can be easy customized for different needs.
I don't think that we would want to create another similar interface
just to be independent from network subsystem.
I guess this is a bit streamy in that you ask one question and get
multiple answers.
It's like seq_file in procfs. The kernel allocates a buffer then fills
it, copies it into userspace, fills it again, ... repeats these actions.
And we can read data from file by portions.
Actually here is one more analogy. When we open a file in procfs,
we sends a request to the kernel and a file path is a request body in
this case. But in case of procfs, we can't construct requests, we only
have a set of predefined requests.
Fair enough. Procfs is also a bit absurd and only makes sense because
it's compatible with lots of tools. In a totally sane world, I would
argue that you should issue one syscall asking questions about a bit
and you should get answers immediately.
--Andy