From: David Drysdale <hidden> Date: 2015-03-13 16:05:29
On Fri, Mar 13, 2015 at 1:40 AM, Josh Triplett [off-list ref] wrote:
This patch series introduces a new clone flag, CLONE_FD, which lets the caller
handle child process exit notification via a file descriptor rather than
SIGCHLD. CLONE_FD makes it possible for libraries to safely launch and manage
child processes on behalf of their caller, *without* taking over process-wide
SIGCHLD handling (either via signal handler or signalfd).
Hi Josh,
From the overall description (i.e. I haven't looked at the code yet)
this looks very interesting. However, it seems to cover a lot of the
same ground as the process descriptor feature that was added to FreeBSD
in 9.x/10.x:
https://www.freebsd.org/cgi/man.cgi?query=pdfork&sektion=2
I think it would ideally be nice for a userspace library developer to be
able to do subprocess management (without SIGCHLD) in a similar way
across both platforms, without lots of complicated autoconf shenanigans.
So could we look at the overlap and seeing if we can come up with
something that covers your requirements and also allows for something
that looks like FreeBSD's process descriptors?
(I've actually got some rough patches to add process descriptor
functionality on Linux, so I can look at how the two approaches compare
and contrast.)
Note that signalfd for SIGCHLD does not suffice here, because that still
receives notification for all child processes, and interferes with process-wide
signal handling.
The CLONE_FD file descriptor uniquely identifies a process on the system in a
race-free way, by holding a reference to the task_struct. In the future, we
may introduce APIs that support using process file descriptors instead of PIDs.
FreeBSD has pdkill(2) and (theoretically) pdwait4(2) along these lines.
I suspect we need either need pdkill(2) or a way to retrieve a PID from
a process file descriptor, so that there's a way to send signals to the
child.
Introducing CLONE_FD required two additional bits of yak shaving: Since clone
has no more usable flags (with the three currently unused flags unusable
because old kernels ignore them without EINVAL), also introduce a new clone4
system call with more flag bits and an extensible argument structure. And
since the magic pt_regs-based syscall argument processing for clone's tls
argument would otherwise prevent introducing a sane clone4 system call, fix
that too.
I tested the CLONE_SETTLS changes with a thread-local storage test program (two
threads independently reading and writing a __thread variable), on both 32-bit
and 64-bit, and I observed no issues there.
Worth preserving in tools/testing/selftests/ ?
I tested clone4 and the new CLONE_FD call with several additional test
programs, launching either a process or thread (in the former case using
syscall(), in the latter case by calling clone4 via assembly and returning to
C), sleeping in parent and child to test the case of either exiting first, and
then printing the received clone4_info structure. Thiago also tested clone4
with CLONE_FD with a modified version of libqt's process handling, which
includes a test suite.
I've also included the manpages patch at the end of this series. (Note that
the manpage documents the behavior of the future glibc wrapper as well as the
raw syscall.) Here's a formatted plain-text version of the manpage for
reference:
FYI, I've added some comparisons with the FreeBSD equivalents below.
CLONE4(2) Linux Programmer's Manual CLONE4(2)
NAME
clone4 - create a child process
SYNOPSIS
/* Prototype for the glibc wrapper function */
#define _GNU_SOURCE
#include <sched.h>
int clone4(uint64_t flags,
size_t args_size,
struct clone4_args *args,
int (*fn)(void *), void *arg);
/* Prototype for the raw system call */
int clone4(unsigned flags_high, unsigned flags_low,
unsigned long args_size,
struct clone4_args *args);
struct clone4_args {
pid_t *ptid;
pid_t *ctid;
unsigned long stack_start;
unsigned long stack_size;
unsigned long tls;
};
DESCRIPTION
clone4() creates a new process, similar to clone(2) and fork(2).
clone4() supports additional flags that clone(2) does not, and accepts
arguments via an extensible structure.
args points to a clone4_args structure, and args_size must contain the
size of that structure, as understood by the caller. If the caller
passes a shorter structure than the kernel expects, the remaining
fields will default to 0. If the caller passes a larger structure than
the kernel expects (such as one from a newer kernel), clone4() will
return EINVAL. The clone4_args structure may gain additional fields at
the end in the future, and callers must only pass a size that encom‐
passes the number of fields they understand. If the caller passes 0
for args_size, args is ignored and may be NULL.
In the clone4_args structure, ptid, ctid, stack_start, stack_size, and
tls have the same semantics as they do with clone(2) and clone2(2).
In the glibc wrapper, fn and arg have the same semantics as they do
with clone(2). As with clone(2), the underlying system call works more
like fork(2), returning 0 in the child process; the glibc wrapper sim‐
plifies thread execution by calling fn(arg) and exiting the child when
that function exits.
The 64-bit flags argument (split into the 32-bit flags_high and
flags_low arguments in the kernel interface) accepts all the same flags
as clone(2), with the exception of the obsolete CLONE_PID,
CLONE_DETACHED, and CLONE_STOPPED. In addition, flags accepts the fol‐
lowing flags:
CLONE_FD
Instead of returning a process ID, clone4() with the CLONE_FD
flag returns a file descriptor associated with the new process.
When the new process exits, the kernel will not send a signal to
the parent process, and will not keep the new process around as
a "zombie" process until a call to waitpid(2) or similar.
Instead, the file descriptor will become available for reading,
and the new process will be immediately reaped.
Just to confirm: presumably a waitpid(-1,...) call that's already in
progress won't return when one of these child processes exits?
Unlike using signalfd(2) for the SIGCHLD signal, the file
descriptor returned by clone4() with the CLONE_FD flag works
even with SIGCHLD unblocked in one or more threads of the parent
process, and allows the process to have different handlers for
different child processes, such as those created by a library,
without introducing race conditions around process-wide signal
handling.
clone4() will never return a file descriptor in the range 0-2 to
the caller, to avoid ambiguity with the return of 0 in the child
process. Only the calling process will have the new file
descriptor open; the child process will not.
FreeBSD's pdfork(2) returns a PID but also takes an int *fdp argument to
return the file descriptor separately, which avoids the need for special
case processing for low FD values (and means that POSIX's "lowest file
descriptor not currently open" behaviour can be preserved if desired).
Since the kernel does not send a termination signal when a child
process created with CLONE_FD exits, the low byte of flags does
not contain a signal number. Instead, the low byte of flags can
contain the following additional flags for use with CLONE_FD:
CLONEFD_CLOEXEC
Set the O_CLOEXEC flag on the new open file descriptor.
See the description of the O_CLOEXEC flag in open(2) for
reasons why this may be useful.
CLONEFD_NONBLOCK
Set the O_NONBLOCK flag on the new open file descriptor.
Using this flag saves extra calls to fcntl(2) to achieve
the same result.
clone4() with the CLONE_FD flag returns a file descriptor that
supports the following operations:
read(2) (and similar)
When the new process exits, reading from the file
descriptor produces a single clonefd_info structure:
struct clonefd_info {
uint32_t code; /* Signal code */
uint32_t status; /* Exit status or signal */
uint64_t utime; /* User CPU time */
uint64_t stime; /* System CPU time */
};
Presumably there is no way to get full rusage information for the exited
process?
[FreeBSD theoretically has pdwait4(2) to do wait4-like operations on a
process descriptor, including rusage retrieval. However, I don't think
they actually implemented it:
http://fxr.watson.org/fxr/source/kern/syscalls.master#L928]
If the new process has not yet exited, read(2) either
blocks until it does, or fails with the error EAGAIN if
the file descriptor has been made nonblocking.
Future kernels may extend clonefd_info by appending addi‐
tional fields to the end. Callers should read as many
bytes as they understand; unread data will be discarded,
and subsequent reads after the first will return 0 to
indicate end-of-file. Callers requesting more bytes than
the kernel provides (such as callers expecting a newer
clonefd_info structure) will receive a shorter structure
from older kernels.
FreeBSD also implements fstat(2) for its process descriptors, although
only a few of the fields get filled in.
poll(2), select(2), epoll(7) (and similar)
The file descriptor is readable (the select(2) readfds
argument; the poll(2) POLLIN flag) if the new process has
exited.
FreeBSD uses POLLHUP here.
close(2)
When the file descriptor is no longer required it should
be closed. If no process has a file descriptor open for
the new process, no process will receive any notification
when the new process exits. The new process will still
be immediately reaped.
FreeBSD has two different behaviours for close(2), depending on a flag
value (PD_DAEMON). With the flag set it's roughly like this, but
without PD_DAEMON a close(2) operation on the (last open) file
descriptor terminates the child process.
This can be quite useful, particularly for the use case where some
userspace library has an FD-controlled subprocess -- if the application
using the library terminates, the process descriptor is closed and so
the subprocess is automatically terminated.
C library/kernel ABI differences
As with clone(2), the raw clone4() system call corresponds more closely
to fork(2) in that execution in the child continues from the point of
the call.
Unlike clone(2), the raw system call interface for clone4() accepts
arguments in the same order on all architectures.
The raw system call accepts flags as two 32-bit arguments, flags_high
and flags_low, to simplify portability across 32-bit and 64-bit archi‐
tectures and calling conventions. The glibc wrapper accepts flags as a
single 64-bit argument for convenience.
RETURN VALUE
For the glibc wrapper, on success, clone4() returns the file descriptor
(with CLONE_FD) or new process ID (without CLONE_FD), and the child
process begins running at the specified function.
For the raw syscall, on success, clone4() returns the file descriptor
or new process ID to the calling process, and returns 0 in the new
child process.
On failure, clone4() returns -1 and sets errno accordingly.
ERRORS
clone4() can return any error from clone(2), as well as the following
additional errors:
EINVAL flags contained an unknown flag.
EINVAL flags included CLONE_FD, but the kernel configuration does not
have the CONFIG_CLONEFD option enabled.
EMFILE flags included CLONE_FD, but the new file descriptor would
exceed the process limit on open file descriptors.
ENFILE flags included CLONE_FD, but the new file descriptor would
exceed the system-wide limit on open file descriptors.
ENODEV flags included CLONE_FD, but clone4() could not mount the
(internal) anonymous inode device.
CONFORMING TO
clone4() is Linux-specific and should not be used in programs intended
to be portable.
SEE ALSO
clone(2), epoll(7), poll(2), pthreads(7), read(2), select(2)
Linux 2015-03-01 CLONE4(2)
On Fri, Mar 13, 2015 at 04:05:29PM +0000, David Drysdale wrote:
On Fri, Mar 13, 2015 at 1:40 AM, Josh Triplett [off-list ref] wrote:
quoted
This patch series introduces a new clone flag, CLONE_FD, which lets the caller
handle child process exit notification via a file descriptor rather than
SIGCHLD. CLONE_FD makes it possible for libraries to safely launch and manage
child processes on behalf of their caller, *without* taking over process-wide
SIGCHLD handling (either via signal handler or signalfd).
Hi Josh,
From the overall description (i.e. I haven't looked at the code yet)
this looks very interesting. However, it seems to cover a lot of the
same ground as the process descriptor feature that was added to FreeBSD
in 9.x/10.x:
https://www.freebsd.org/cgi/man.cgi?query=pdfork&sektion=2
Interesting.
I think it would ideally be nice for a userspace library developer to be
able to do subprocess management (without SIGCHLD) in a similar way
across both platforms, without lots of complicated autoconf shenanigans.
So could we look at the overlap and seeing if we can come up with
something that covers your requirements and also allows for something
that looks like FreeBSD's process descriptors?
Agreed; however, I think it's reasonable to provide appropriate Linux
system calls, and then let glibc or libbsd or similar provide the
BSD-compatible calls on top of those. I don't think the kernel
interface needs to exactly match FreeBSD's, as long as it's a superset
of the functionality.
For example, pdfork can just call clone4 with CLONE_FD and return the
resulting file descriptor.
In my further comments below, I'll suggest ways that the FreeBSD library
calls could be implemented on top of Linux system calls.
(I've actually got some rough patches to add process descriptor
functionality on Linux, so I can look at how the two approaches compare
and contrast.)
quoted
Note that signalfd for SIGCHLD does not suffice here, because that still
receives notification for all child processes, and interferes with process-wide
signal handling.
The CLONE_FD file descriptor uniquely identifies a process on the system in a
race-free way, by holding a reference to the task_struct. In the future, we
may introduce APIs that support using process file descriptors instead of PIDs.
FreeBSD has pdkill(2) and (theoretically) pdwait4(2) along these lines.
I suspect we need either need pdkill(2) or a way to retrieve a PID from
a process file descriptor, so that there's a way to send signals to the
child.
The original caller of clone4 with CLONE_FD can pass CLONE_PARENT_SETTID
to get the PID.
In the future, I plan to add an fd-based equivalent of
rt_{,tg}sigqueueinfo (likely a single syscall with a flag to determine
whether to kill a process or thread) which is a superset of pdkill.
pdkill could then call that and just not pass the extra info.
A fair bit of pdwait4 could be implemented on top of read(), other than
the full rusage information (see below), and the ability to wait for
STOP/CONT (which the CLONE_FD file descriptor could support if desired,
but it'd have to be set via a flag at clone time).
I think it's a feature to use read() rather than an additional magic
system call.
quoted
Introducing CLONE_FD required two additional bits of yak shaving: Since clone
has no more usable flags (with the three currently unused flags unusable
because old kernels ignore them without EINVAL), also introduce a new clone4
system call with more flag bits and an extensible argument structure. And
since the magic pt_regs-based syscall argument processing for clone's tls
argument would otherwise prevent introducing a sane clone4 system call, fix
that too.
I tested the CLONE_SETTLS changes with a thread-local storage test program (two
threads independently reading and writing a __thread variable), on both 32-bit
and 64-bit, and I observed no issues there.
Worth preserving in tools/testing/selftests/ ?
Not really; it's just the following trivial program, which was faster to
write than to attempt to find somewhere:
#include <pthread.h>
#include <stdio.h>
__thread unsigned x = 0;
void *thread_func(void *unused)
{
unsigned *tx = &x;
for (; *tx < 10; (*tx)++)
printf("child: tx=%p *tx=%u\n", tx, *tx);
return NULL;
}
int main(void)
{
unsigned *tx = &x;
pthread_t thread;
pthread_create(&thread, NULL, thread_func, NULL);
for (; *tx < 10; (*tx)++)
printf("main: tx=%p *tx=%u\n", tx, *tx);
pthread_join(thread, NULL);
return 0;
}
(I didn't bother with error handling, because I ran it under strace.)
quoted
I tested clone4 and the new CLONE_FD call with several additional test
programs, launching either a process or thread (in the former case using
syscall(), in the latter case by calling clone4 via assembly and returning to
C), sleeping in parent and child to test the case of either exiting first, and
then printing the received clone4_info structure. Thiago also tested clone4
with CLONE_FD with a modified version of libqt's process handling, which
includes a test suite.
I've also included the manpages patch at the end of this series. (Note that
the manpage documents the behavior of the future glibc wrapper as well as the
raw syscall.) Here's a formatted plain-text version of the manpage for
reference:
FYI, I've added some comparisons with the FreeBSD equivalents below.
Thanks!
quoted
CLONE4(2) Linux Programmer's Manual CLONE4(2)
NAME
clone4 - create a child process
SYNOPSIS
/* Prototype for the glibc wrapper function */
#define _GNU_SOURCE
#include <sched.h>
int clone4(uint64_t flags,
size_t args_size,
struct clone4_args *args,
int (*fn)(void *), void *arg);
/* Prototype for the raw system call */
int clone4(unsigned flags_high, unsigned flags_low,
unsigned long args_size,
struct clone4_args *args);
struct clone4_args {
pid_t *ptid;
pid_t *ctid;
unsigned long stack_start;
unsigned long stack_size;
unsigned long tls;
};
DESCRIPTION
clone4() creates a new process, similar to clone(2) and fork(2).
clone4() supports additional flags that clone(2) does not, and accepts
arguments via an extensible structure.
args points to a clone4_args structure, and args_size must contain the
size of that structure, as understood by the caller. If the caller
passes a shorter structure than the kernel expects, the remaining
fields will default to 0. If the caller passes a larger structure than
the kernel expects (such as one from a newer kernel), clone4() will
return EINVAL. The clone4_args structure may gain additional fields at
the end in the future, and callers must only pass a size that encom‐
passes the number of fields they understand. If the caller passes 0
for args_size, args is ignored and may be NULL.
In the clone4_args structure, ptid, ctid, stack_start, stack_size, and
tls have the same semantics as they do with clone(2) and clone2(2).
In the glibc wrapper, fn and arg have the same semantics as they do
with clone(2). As with clone(2), the underlying system call works more
like fork(2), returning 0 in the child process; the glibc wrapper sim‐
plifies thread execution by calling fn(arg) and exiting the child when
that function exits.
The 64-bit flags argument (split into the 32-bit flags_high and
flags_low arguments in the kernel interface) accepts all the same flags
as clone(2), with the exception of the obsolete CLONE_PID,
CLONE_DETACHED, and CLONE_STOPPED. In addition, flags accepts the fol‐
lowing flags:
CLONE_FD
Instead of returning a process ID, clone4() with the CLONE_FD
flag returns a file descriptor associated with the new process.
When the new process exits, the kernel will not send a signal to
the parent process, and will not keep the new process around as
a "zombie" process until a call to waitpid(2) or similar.
Instead, the file descriptor will become available for reading,
and the new process will be immediately reaped.
Just to confirm: presumably a waitpid(-1,...) call that's already in
progress won't return when one of these child processes exits?
I agree, I don't think it should. Because otherwise you'd also assume
you can waitpid() on the PID itself, and that'd be a race condition
since the process autoreaps.
quoted
Unlike using signalfd(2) for the SIGCHLD signal, the file
descriptor returned by clone4() with the CLONE_FD flag works
even with SIGCHLD unblocked in one or more threads of the parent
process, and allows the process to have different handlers for
different child processes, such as those created by a library,
without introducing race conditions around process-wide signal
handling.
clone4() will never return a file descriptor in the range 0-2 to
the caller, to avoid ambiguity with the return of 0 in the child
process. Only the calling process will have the new file
descriptor open; the child process will not.
FreeBSD's pdfork(2) returns a PID but also takes an int *fdp argument to
return the file descriptor separately, which avoids the need for special
case processing for low FD values (and means that POSIX's "lowest file
descriptor not currently open" behaviour can be preserved if desired).
That'd be easy to implement if desired, by adding an outbound pointer to
clone4_args.
The (very mild) reason I'd dropped the PID: with CLONE_FD and future
syscalls that use the fd as an identifier, PIDs can hopefully become
mostly unnecessary. However, I'm not that attached to changing the
return value; it'd be trivial to switch to an outbound parameter
instead, and then drop the "not 0-2".
quoted
Since the kernel does not send a termination signal when a child
process created with CLONE_FD exits, the low byte of flags does
not contain a signal number. Instead, the low byte of flags can
contain the following additional flags for use with CLONE_FD:
CLONEFD_CLOEXEC
Set the O_CLOEXEC flag on the new open file descriptor.
See the description of the O_CLOEXEC flag in open(2) for
reasons why this may be useful.
CLONEFD_NONBLOCK
Set the O_NONBLOCK flag on the new open file descriptor.
Using this flag saves extra calls to fcntl(2) to achieve
the same result.
clone4() with the CLONE_FD flag returns a file descriptor that
supports the following operations:
read(2) (and similar)
When the new process exits, reading from the file
descriptor produces a single clonefd_info structure:
struct clonefd_info {
uint32_t code; /* Signal code */
uint32_t status; /* Exit status or signal */
uint64_t utime; /* User CPU time */
uint64_t stime; /* System CPU time */
};
Presumably there is no way to get full rusage information for the exited
process?
I focused on the information available via SIGCHLD. Even utime and
stime are unnecessary for the primary use case of CLONE_FD, but I
included them because SIGCHLD does. I'd like to avoid sending the much
larger rusage over the file descriptor when the caller may not care.
However, given that the task_struct sticks around as long as the
CLONE_FD file descriptor does, if that information is normally still
available from a dead-but-not-waited-on process, it should be trivial to
add an operation that takes the file descriptor and returns the full
rusage, if someone needs that. I think that can be done as part of a
later patch series adding other operations for use with the file
descriptor, though.
That's a pretty good argument that we don't need to either, at least not
yet.
quoted
If the new process has not yet exited, read(2) either
blocks until it does, or fails with the error EAGAIN if
the file descriptor has been made nonblocking.
Future kernels may extend clonefd_info by appending addi‐
tional fields to the end. Callers should read as many
bytes as they understand; unread data will be discarded,
and subsequent reads after the first will return 0 to
indicate end-of-file. Callers requesting more bytes than
the kernel provides (such as callers expecting a newer
clonefd_info structure) will receive a shorter structure
from older kernels.
FreeBSD also implements fstat(2) for its process descriptors, although
only a few of the fields get filled in.
I looked at what they provide, and that seems like more of a novelty
than something particularly useful (since most of the stat fields aren't
meaningful), but if that's useful for compatibility then adding it seems
fine.
quoted
poll(2), select(2), epoll(7) (and similar)
The file descriptor is readable (the select(2) readfds
argument; the poll(2) POLLIN flag) if the new process has
exited.
FreeBSD uses POLLHUP here.
That makes sense given that they provide the information via a separate
call rather than read. Since the CLONE_FD file descriptor uses read, it
needs to provide POLLIN, but I have no objection to using *both* POLLIN
and POLLHUP if that'd be at all useful.
quoted
close(2)
When the file descriptor is no longer required it should
be closed. If no process has a file descriptor open for
the new process, no process will receive any notification
when the new process exits. The new process will still
be immediately reaped.
FreeBSD has two different behaviours for close(2), depending on a flag
value (PD_DAEMON). With the flag set it's roughly like this, but
without PD_DAEMON a close(2) operation on the (last open) file
descriptor terminates the child process.
This can be quite useful, particularly for the use case where some
userspace library has an FD-controlled subprocess -- if the application
using the library terminates, the process descriptor is closed and so
the subprocess is automatically terminated.
That's an interesting idea. I don't think it makes sense for that to be
the default behavior, but if someone wanted to add an additional flag
to implement that behavior, that seems fine. A FreeBSD-compatible
pdfork could then use that flag when not passed PD_DAEMON and not use it
when passed PD_DAEMON.
How does it kill the process when the last open descriptor closes?
SIGKILL? SIGTERM? The former seems unfriendly (preventing graceful
termination), and the latter blockable. There's a reason init systems
send TERM, then wait, then KILL.
- Josh Triplett
On Friday 13 March 2015 12:42:52 Josh Triplett wrote:
quoted
Hi Josh,
From the overall description (i.e. I haven't looked at the code yet)
this looks very interesting. However, it seems to cover a lot of the
same ground as the process descriptor feature that was added to FreeBSD
in 9.x/10.x:
https://www.freebsd.org/cgi/man.cgi?query=pdfork&sektion=2
Interesting.
Hi Josh, David
I wasn't aware of the FreeBSD implementation of pdfork(). It is actually
exactly what I need in userspace. The only difference between pdfork() and and
my proposed forkfd() is where the PID and where the file descriptor are
returned (meaning, which is optional and which isn't).
Josh and I opted to return the file descriptor in the regular return value in
forkfd and in clone4 because getting the file descriptor the whole objective of
using the forkfd or clone4-with-CLONE_FD in the first place: the file descriptor
is not optional, but the PID is.
Agreed; however, I think it's reasonable to provide appropriate Linux
system calls, and then let glibc or libbsd or similar provide the
BSD-compatible calls on top of those. I don't think the kernel
interface needs to exactly match FreeBSD's, as long as it's a superset
of the functionality.
For example, pdfork can just call clone4 with CLONE_FD and return the
resulting file descriptor.
Agreed, we should recommend libc implement pdfork(), pdkill() and pdwait4().
I'm not too attached to the forkfd() interface, but I find it slightly superior
for the reasons above.
If we want the PD_DAEMON flag, it will have to translate to a clone flag, like
CLONEFD_DAEMON or inverted like CLONEFD_KILL_ON_CLOSE.
In the future, I plan to add an fd-based equivalent of
rt_{,tg}sigqueueinfo (likely a single syscall with a flag to determine
whether to kill a process or thread) which is a superset of pdkill.
pdkill could then call that and just not pass the extra info.
A fair bit of pdwait4 could be implemented on top of read(), other than
the full rusage information (see below), and the ability to wait for
STOP/CONT (which the CLONE_FD file descriptor could support if desired,
but it'd have to be set via a flag at clone time).
I think it's a feature to use read() rather than an additional magic
system call.
Indeed, even if the libc provides a wrapper for you, like glibc does for
eventfd (eventfd_read, eventfd_write).
Josh and I didn't want to submit "killfd" (or pdkill in the FreeBSD name) in
the initial patch set, but it was part of the plans.
quoted
quoted
clone4() will never return a file descriptor in the range
0-2 to
the caller, to avoid ambiguity with the return of 0 in the
child
process. Only the calling process will have the new
file
descriptor open; the child process will not.
FreeBSD's pdfork(2) returns a PID but also takes an int *fdp argument to
return the file descriptor separately, which avoids the need for special
case processing for low FD values (and means that POSIX's "lowest file
descriptor not currently open" behaviour can be preserved if desired).
That'd be easy to implement if desired, by adding an outbound pointer to
clone4_args.
The (very mild) reason I'd dropped the PID: with CLONE_FD and future
syscalls that use the fd as an identifier, PIDs can hopefully become
mostly unnecessary. However, I'm not that attached to changing the
return value; it'd be trivial to switch to an outbound parameter
instead, and then drop the "not 0-2".
See above for more motivation on making the PID optional.
As for the file descriptor range, if we need to be able to return 0, we can
implement a magic constant to mean the child process, like the userspace
forkfd() does (FFD_CHILD_PROCESS). We'd probably choose the value -4096 on
Linux, since that is neither a valid file descriptor nor a valid errno value.
That's a pretty good argument that we don't need to either, at least not
yet.
pdwait4() can be implemented on top of read(), with the WNOHANG flag being just
toggling the O_NONBLOCK bit. The problem is with the rest of the flags. We
could implement it via more ioctls to be done prior to read() if we don't want
to add a syscall...
Another alternative is to add a P_PD flag that can be passed as the first
argument to waitid(), making the second argument a file descriptor instead of a
PID or pgrp.
quoted
FreeBSD also implements fstat(2) for its process descriptors, although
only a few of the fields get filled in.
I looked at what they provide, and that seems like more of a novelty
than something particularly useful (since most of the stat fields aren't
meaningful), but if that's useful for compatibility then adding it seems
fine.
I don't think we need to do anything: anon_inode will do it for us.
If I stat an eventfd:
stat("/proc/107751/fd/4", {st_dev=makedev(0, 9), st_ino=3943,
st_mode=0600, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
And just out of curiosity, in the following order: epoll, signalfd, timerfd
and inotify:
stat("/proc/1462/fd/4", {st_dev=makedev(0, 9), st_ino=3943, st_mode=0600,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0,
st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
stat("/proc/1462/fd/5", {st_dev=makedev(0, 9), st_ino=3943, st_mode=0600,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0,
st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
stat("/proc/1462/fd/7", {st_dev=makedev(0, 9), st_ino=3943, st_mode=0600,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0,
st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
stat("/proc/1462/fd/8", {st_dev=makedev(0, 9), st_ino=3943, st_mode=0600,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0,
st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
(that process is systemd --user)
quoted
quoted
poll(2), select(2), epoll(7) (and similar)
The file descriptor is readable (the select(2)
readfds
argument; the poll(2) POLLIN flag) if the new
process has
exited.
FreeBSD uses POLLHUP here.
That makes sense given that they provide the information via a separate
call rather than read. Since the CLONE_FD file descriptor uses read, it
needs to provide POLLIN, but I have no objection to using *both* POLLIN
and POLLHUP if that'd be at all useful.
I think we should provide both, since we're notifying that there are things to
be read and that the file descriptor has closed.
quoted
FreeBSD has two different behaviours for close(2), depending on a flag
value (PD_DAEMON). With the flag set it's roughly like this, but
without PD_DAEMON a close(2) operation on the (last open) file
descriptor terminates the child process.
This can be quite useful, particularly for the use case where some
userspace library has an FD-controlled subprocess -- if the application
using the library terminates, the process descriptor is closed and so
the subprocess is automatically terminated.
That's an interesting idea. I don't think it makes sense for that to be
the default behavior, but if someone wanted to add an additional flag
to implement that behavior, that seems fine. A FreeBSD-compatible
pdfork could then use that flag when not passed PD_DAEMON and not use it
when passed PD_DAEMON.
How does it kill the process when the last open descriptor closes?
SIGKILL? SIGTERM? The former seems unfriendly (preventing graceful
termination), and the latter blockable. There's a reason init systems
send TERM, then wait, then KILL.
I was wondering if it shouldn't be a SIGHUP, since we're talking about a file
descriptor closing. We could make it configurable too, but I'd rather not use
the current CSIGNAL field -- better move to the arguments structure, just in
case someone is passing SIGCHLD there, they should get EINVAL instead of
silently sending SIGCHLD to the child process to ask it to terminate.
--
Thiago Macieira - thiago.macieira (AT) intel.com
Software Architect - Intel Open Source Technology Center
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-03-13 21:33:44
On Fri, Mar 13, 2015 at 12:42 PM, Josh Triplett [off-list ref] wrote:
On Fri, Mar 13, 2015 at 04:05:29PM +0000, David Drysdale wrote:
quoted
On Fri, Mar 13, 2015 at 1:40 AM, Josh Triplett [off-list ref] wrote:
quoted
This patch series introduces a new clone flag, CLONE_FD, which lets the caller
handle child process exit notification via a file descriptor rather than
SIGCHLD. CLONE_FD makes it possible for libraries to safely launch and manage
child processes on behalf of their caller, *without* taking over process-wide
SIGCHLD handling (either via signal handler or signalfd).
Hi Josh,
From the overall description (i.e. I haven't looked at the code yet)
this looks very interesting. However, it seems to cover a lot of the
same ground as the process descriptor feature that was added to FreeBSD
in 9.x/10.x:
https://www.freebsd.org/cgi/man.cgi?query=pdfork&sektion=2
Interesting.
quoted
I think it would ideally be nice for a userspace library developer to be
able to do subprocess management (without SIGCHLD) in a similar way
across both platforms, without lots of complicated autoconf shenanigans.
So could we look at the overlap and seeing if we can come up with
something that covers your requirements and also allows for something
that looks like FreeBSD's process descriptors?
Agreed; however, I think it's reasonable to provide appropriate Linux
system calls, and then let glibc or libbsd or similar provide the
BSD-compatible calls on top of those. I don't think the kernel
interface needs to exactly match FreeBSD's, as long as it's a superset
of the functionality.
We need to be careful with things like read(2), though. It's hard to
write a glibc function that makes read(2) do something other than what
the kernel thinks. Similarly, poll(2) is defined by the kernel. It
would be really nice to be consistent here.
--Andy
On Fri, Mar 13, 2015 at 02:16:07PM -0700, Thiago Macieira wrote:
On Friday 13 March 2015 12:42:52 Josh Triplett wrote:
quoted
quoted
Hi Josh,
From the overall description (i.e. I haven't looked at the code yet)
this looks very interesting. However, it seems to cover a lot of the
same ground as the process descriptor feature that was added to FreeBSD
in 9.x/10.x:
https://www.freebsd.org/cgi/man.cgi?query=pdfork&sektion=2
Interesting.
I wasn't aware of the FreeBSD implementation of pdfork(). It is actually
exactly what I need in userspace.
Right; libqt should be able to use pdfork on FreeBSD and CLONE_FD on
Linux.
The only difference between pdfork() and and
my proposed forkfd() is where the PID and where the file descriptor are
returned (meaning, which is optional and which isn't).
Josh and I opted to return the file descriptor in the regular return value in
forkfd and in clone4 because getting the file descriptor the whole objective of
using the forkfd or clone4-with-CLONE_FD in the first place: the file descriptor
is not optional, but the PID is.
And as long as you can get the fd, where it's returned really doesn't
matter.
quoted
Agreed; however, I think it's reasonable to provide appropriate Linux
system calls, and then let glibc or libbsd or similar provide the
BSD-compatible calls on top of those. I don't think the kernel
interface needs to exactly match FreeBSD's, as long as it's a superset
of the functionality.
For example, pdfork can just call clone4 with CLONE_FD and return the
resulting file descriptor.
Agreed, we should recommend libc implement pdfork(), pdkill() and pdwait4().
I'm not too attached to the forkfd() interface, but I find it slightly superior
for the reasons above.
Agreed.
If we want the PD_DAEMON flag, it will have to translate to a clone flag, like
CLONEFD_DAEMON or inverted like CLONEFD_KILL_ON_CLOSE.
I think the inverted version makes more sense, so that the default
behavior just changes exit notification without adding the kill-on-close
behavior. And that kill-on-close behavior can come in a later patch. :)
quoted
In the future, I plan to add an fd-based equivalent of
rt_{,tg}sigqueueinfo (likely a single syscall with a flag to determine
whether to kill a process or thread) which is a superset of pdkill.
pdkill could then call that and just not pass the extra info.
A fair bit of pdwait4 could be implemented on top of read(), other than
the full rusage information (see below), and the ability to wait for
STOP/CONT (which the CLONE_FD file descriptor could support if desired,
but it'd have to be set via a flag at clone time).
I think it's a feature to use read() rather than an additional magic
system call.
Indeed, even if the libc provides a wrapper for you, like glibc does for
eventfd (eventfd_read, eventfd_write).
Josh and I didn't want to submit "killfd" (or pdkill in the FreeBSD name) in
the initial patch set, but it was part of the plans.
quoted
quoted
quoted
clone4() will never return a file descriptor in the range
0-2 to
the caller, to avoid ambiguity with the return of 0 in the
child
process. Only the calling process will have the new
file
descriptor open; the child process will not.
FreeBSD's pdfork(2) returns a PID but also takes an int *fdp argument to
return the file descriptor separately, which avoids the need for special
case processing for low FD values (and means that POSIX's "lowest file
descriptor not currently open" behaviour can be preserved if desired).
That'd be easy to implement if desired, by adding an outbound pointer to
clone4_args.
The (very mild) reason I'd dropped the PID: with CLONE_FD and future
syscalls that use the fd as an identifier, PIDs can hopefully become
mostly unnecessary. However, I'm not that attached to changing the
return value; it'd be trivial to switch to an outbound parameter
instead, and then drop the "not 0-2".
See above for more motivation on making the PID optional.
As for the file descriptor range, if we need to be able to return 0, we can
implement a magic constant to mean the child process, like the userspace
forkfd() does (FFD_CHILD_PROCESS). We'd probably choose the value -4096 on
Linux, since that is neither a valid file descriptor nor a valid errno value.
I don't think that logic is worth implementing, though, since it would
require changing all the architecture-specific copy_thread
implementations. If we really want to go this path, we should just
return the fd via an out parameter in the clone4_args structure.
That's a pretty good argument that we don't need to either, at least not
yet.
pdwait4() can be implemented on top of read(), with the WNOHANG flag being just
toggling the O_NONBLOCK bit. The problem is with the rest of the flags. We
could implement it via more ioctls to be done prior to read() if we don't want
to add a syscall...
Another alternative is to add a P_PD flag that can be passed as the first
argument to waitid(), making the second argument a file descriptor instead of a
PID or pgrp.
Or a flag that can be added to the options argument of wait4 to indicate
that the first argument is really a file descriptor.
quoted
quoted
FreeBSD also implements fstat(2) for its process descriptors, although
only a few of the fields get filled in.
I looked at what they provide, and that seems like more of a novelty
than something particularly useful (since most of the stat fields aren't
meaningful), but if that's useful for compatibility then adding it seems
fine.
I don't think we need to do anything: anon_inode will do it for us.
If I stat an eventfd:
stat("/proc/107751/fd/4", {st_dev=makedev(0, 9), st_ino=3943,
st_mode=0600, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0,
st_size=0, st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
And just out of curiosity, in the following order: epoll, signalfd, timerfd
and inotify:
stat("/proc/1462/fd/4", {st_dev=makedev(0, 9), st_ino=3943, st_mode=0600,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0,
st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
stat("/proc/1462/fd/5", {st_dev=makedev(0, 9), st_ino=3943, st_mode=0600,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0,
st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
stat("/proc/1462/fd/7", {st_dev=makedev(0, 9), st_ino=3943, st_mode=0600,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0,
st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
stat("/proc/1462/fd/8", {st_dev=makedev(0, 9), st_ino=3943, st_mode=0600,
st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0,
st_atime=2015/03/07-16:40:28, st_mtime=2015/03/12-16:12:00,
st_ctime=2015/03/12-16:12:00}) = 0
(that process is systemd --user)
Interesting. What does stat on a CLONE_FD file descriptor return?
quoted
quoted
quoted
poll(2), select(2), epoll(7) (and similar)
The file descriptor is readable (the select(2)
readfds
argument; the poll(2) POLLIN flag) if the new
process has
exited.
FreeBSD uses POLLHUP here.
That makes sense given that they provide the information via a separate
call rather than read. Since the CLONE_FD file descriptor uses read, it
needs to provide POLLIN, but I have no objection to using *both* POLLIN
and POLLHUP if that'd be at all useful.
I think we should provide both, since we're notifying that there are things to
be read and that the file descriptor has closed.
"closed" in the "other end of the not-quite-a-pipe" sense, sure. I'll
add that in a v2.
quoted
quoted
FreeBSD has two different behaviours for close(2), depending on a flag
value (PD_DAEMON). With the flag set it's roughly like this, but
without PD_DAEMON a close(2) operation on the (last open) file
descriptor terminates the child process.
This can be quite useful, particularly for the use case where some
userspace library has an FD-controlled subprocess -- if the application
using the library terminates, the process descriptor is closed and so
the subprocess is automatically terminated.
That's an interesting idea. I don't think it makes sense for that to be
the default behavior, but if someone wanted to add an additional flag
to implement that behavior, that seems fine. A FreeBSD-compatible
pdfork could then use that flag when not passed PD_DAEMON and not use it
when passed PD_DAEMON.
How does it kill the process when the last open descriptor closes?
SIGKILL? SIGTERM? The former seems unfriendly (preventing graceful
termination), and the latter blockable. There's a reason init systems
send TERM, then wait, then KILL.
I was wondering if it shouldn't be a SIGHUP, since we're talking about a file
descriptor closing. We could make it configurable too, but I'd rather not use
the current CSIGNAL field -- better move to the arguments structure, just in
case someone is passing SIGCHLD there, they should get EINVAL instead of
silently sending SIGCHLD to the child process to ask it to terminate.
That sounds like several good reasons right there to defer "kill on
close" to a future patch, the author of which should research how
FreeBSD implements this.
- Josh Triplett
From: David Drysdale <hidden> Date: 2015-03-15 08:55:33
On Fri, Mar 13, 2015 at 7:42 PM, Josh Triplett [off-list ref] wrote:
On Fri, Mar 13, 2015 at 04:05:29PM +0000, David Drysdale wrote:
quoted
On Fri, Mar 13, 2015 at 1:40 AM, Josh Triplett [off-list ref] wrote:
quoted
This patch series introduces a new clone flag, CLONE_FD, which lets the caller
handle child process exit notification via a file descriptor rather than
SIGCHLD. CLONE_FD makes it possible for libraries to safely launch and manage
child processes on behalf of their caller, *without* taking over process-wide
SIGCHLD handling (either via signal handler or signalfd).
Hi Josh,
From the overall description (i.e. I haven't looked at the code yet)
this looks very interesting. However, it seems to cover a lot of the
same ground as the process descriptor feature that was added to FreeBSD
in 9.x/10.x:
https://www.freebsd.org/cgi/man.cgi?query=pdfork&sektion=2
Interesting.
quoted
I think it would ideally be nice for a userspace library developer to be
able to do subprocess management (without SIGCHLD) in a similar way
across both platforms, without lots of complicated autoconf shenanigans.
So could we look at the overlap and seeing if we can come up with
something that covers your requirements and also allows for something
that looks like FreeBSD's process descriptors?
Agreed; however, I think it's reasonable to provide appropriate Linux
system calls, and then let glibc or libbsd or similar provide the
BSD-compatible calls on top of those. I don't think the kernel
interface needs to exactly match FreeBSD's, as long as it's a superset
of the functionality.
Agreed -- if it's possible to implement equivalent process descriptor
functionality with a wrapper library, but the kernel interface is more
comprehensive and consistent with the rest of the Linux kernel, then
that's a big win. So thanks for your work and for being willing to look
at the overlap!