kdbus is a kernel-level IPC implementation that aims for resemblance to
the the protocol layer with the existing userspace D-Bus daemon while
enabling some features that couldn't be implemented before in userspace.
The documentation in the first patch in this series explains the
protocol and the API details.
Full details on what has changed from the v2 submission are at the
bottom of this email.
Reasons why this should be done in the kernel, instead of userspace as
it is currently done today include the following:
- performance: fewer process context switches, fewer copies, fewer
syscalls, larger memory chunks via memfd. This is really important
for a whole class of userspace programs that are ported from other
operating systems that are run on tiny ARM systems that rely on
hundreds of thousands of messages passed at boot time, and at
"critical" times in their user interaction loops.
- security: the peers which communicate do not have to trust each other,
as the only trustworthy compoenent in the game is the kernel which
adds metadata and ensures that all data passed as payload is either
copied or sealed, so that the receiver can parse the data without
having to protect against changing memory while parsing buffers. Also,
all the data transfer is controlled by the kernel, so that LSMs can
track and control what is going on, without involving userspace.
Because of the LSM issue, security people are much happier with this
model than the current scheme of having to hook into dbus to mediate
things.
- more metadata can be attached to messages than in userspace
- semantics for apps with heavy data payloads (media apps, for instance)
with optinal priority message dequeuing, and global message ordering.
Some "crazy" people are playing with using kdbus for audio data in the
system. I'm not saying that this is the best model for this, but
until now, there wasn't any other way to do this without having to
create custom "busses", one for each application library.
- being in the kernle closes a lot of races which can't be fixed with
the current userspace solutions. For example, with kdbus, there is a
way a client can disconnect from a bus, but do so only if no further
messages present in its queue, which is crucial for implementing
race-free "exit-on-idle" services
- eavesdropping on the kernel level, so privileged users can hook into
the message stream without hacking support for that into their
userspace processes
- a number of smaller benefits: for example kdbus learned a way to peek
full messages without dequeing them, which is really useful for
logging metadata when handling bus-activation requests.
Of course, some of the bits above could be implemented in userspace
alone, for example with more sophisticated memory management APIs, but
this is usually done by losing out on the other details. For example,
for many of the memory management APIs, it's hard to not require the
communicating peers to fully trust each other. And we _really_ don't
want peers to have to trust each other.
Another benefit of having this in the kernel, rather than as a userspace
daemon, is that you can now easily use the bus from the initrd, or up to
the very end when the system shuts down. On current userspace D-Bus,
this is not really possible, as this requires passing the bus instance
around between initrd and the "real" system. Such a transition of all
fds also requires keeping full state of what has already been read from
the connection fds. kdbus makes this much simpler, as we can change the
ownership of the bus, just by passing one fd over from one part to the
other.
Regarding binder: binder and kdbus follow very different design
concepts. Binder implies the use of thread-pools to dispatch incoming
method calls. This is a very efficient scheme, and completely natural
in programming languages like Java. On most Linux programs, however,
there's a much stronger focus on central poll() loops that dispatch all
sources a program cares about. kdbus is much more usable in such
environments, as it doesn't enforce a threading model, and it is happy
with serialized dispatching. In fact, this major difference had an
effect on much of the design decisions: binder does not guarantee global
message ordering due to the parallel dispatching in the thread-pools,
but kdbus does. Moreover, there's also a difference in the way message
handling. In kdbus, every message is basically taken and dispatched as
one blob, while in binder, continious connections to other peers are
created, which are then used to send messages on. Hence, the models are
quite different, and they serve different needs. I believe that the
D-Bus/kdbus model is more compatible and friendly with how Linux
programs are usually implemented.
This can also be found in a git tree, the kdbus branch of char-misc.git at:
https://git.kernel.org/cgit/linux/kernel/git/gregkh/char-misc.git/
Changes since v2:
* Add FS_USERNS_MOUNT to the file system flags, so users can mount
their own kdbusfs instances without being root in the parent
user-ns. Spotted by Andy Lutomirski.
* Rewrite major parts of the metadata implementation to allow for
per-recipient namespace translations. For this, namespaces are
now not pinned by domains anymore. Instead, metadata is recorded
in kernel scope, and exported into the currently active namespaces
at the time of message installing.
* Split PID and TID from KDBUS_ITEM_CREDS into KDBUS_ITEM_PIDS.
The starttime is there to detect re-used PIDs, so move it to that
new item type as well. Consequently, introduce struct kdbus_pids
to accommodate the information. Requested by Andy Lutomirski.
* Add {e,s,fs}{u,g}id to KDBUS_ITEM_CREDS, so users have a way to
get more fine-grained credential information.
* Removed KDBUS_CMD_CANCEL. The interface was not usable from
threaded userspace implementation due to inherent races. Instead,
add an item type CANCEL_FD which can be used to pass a file
descriptor to the CMD_SEND ioctl. When the SEND is done
synchronously, it will get cancelled as soon as the passed
FD signals POLLIN.
* Dropped startttime from KDBUS_ITEM_PIDS
* Restrict names of custom endpoints to names with a "<uid>-" prefix,
just like we do for buses.
* Provide module-parameter "kdbus.attach_flags_mask" to specify the
a mask of metadata items that is applied on all exported items.
* Monitors are now entirely invisible (IOW, there won't be any
notification when they are created) and they don't need to install
filters for broadcast messages anymore.
* All information exposed via a connection's pool now also reports
the length in addition to the offset. That way, userspace
applications can mmap() only parts of the pool on demand.
* Due to the metadata rework, KDBUS_ITEM_PAYLOAD_OFF items now
describe the offset relative to the pool, where they used to be
relative to the message header.
* Added return_flags bitmask to all kdbus_cmd_* structs, so the
kernel can report details of the command processing. This is
mostly reserved for future extensions.
* Some fixes in kdbus.txt and tests, spotted by Harald Hoyer, Andy
Lutomirski, Michele Curti, Sergei Zviagintsev, Sheng Yong, Torstein
Husebø and Hristo Venev.
* Fixed compiler warnings in test-message by Michele Curti
* Unexpected items are now rejected with -EINVAL
* Split signal and broadcast handling. Unicast signals are now
supported, and messages have a new KDBUS_MSG_SIGNAL flag.
* KDBUS_CMD_MSG_SEND was renamed to KDBUS_CMD_SEND, and now takes
a struct kdbus_cmd_send instead of a kdbus_msg.
* KDBUS_CMD_MSG_RECV was renamed to KDBUS_CMD_RECV.
* Test case memory leak plugged, and various other cleanups and
fixes, by Rui Miguel Silva.
* Build fix for s390
* Test case fix for 32bit archs
* The test framework now supports mount, pid and user namespaces.
* The test framework learned a --tap command line parameter to
format its output in the "Test Anything Protocol". This format
is chosen by default when "make kselftest" is invoked.
* Fixed buses and custom endpoints name validation, reported by
Andy Lutomirski.
* copy_from_user() return code issue fixed, reported by
Dan Carpenter.
* Avoid signed int overflow on archs without atomic_sub
* Avoid variable size stack items. Fixes a sparse warning in queue.c.
* New test case for kernel notification quota
* Switched back to enums for the list of ioctls. This has advantages
for userspace code as gdb, for instance, is able to resolve the
numbers into names. Added features can easily be detected with
autotools, and new iotcls can get #defines as well. Having #defines
for the initial set of ioctls is uncecessary.
Daniel Mack (13):
kdbus: add documentation
kdbus: add header file
kdbus: add driver skeleton, ioctl entry points and utility functions
kdbus: add connection pool implementation
kdbus: add connection, queue handling and message validation code
kdbus: add node and filesystem implementation
kdbus: add code to gather metadata
kdbus: add code for notifications and matches
kdbus: add code for buses, domains and endpoints
kdbus: add name registry implementation
kdbus: add policy database implementation
kdbus: add Makefile, Kconfig and MAINTAINERS entry
kdbus: add selftests
Documentation/ioctl/ioctl-number.txt | 1 +
Documentation/kdbus.txt | 2107 +++++++++++++++++++++
MAINTAINERS | 12 +
include/uapi/linux/Kbuild | 1 +
include/uapi/linux/kdbus.h | 1049 ++++++++++
include/uapi/linux/magic.h | 2 +
init/Kconfig | 12 +
ipc/Makefile | 2 +-
ipc/kdbus/Makefile | 22 +
ipc/kdbus/bus.c | 553 ++++++
ipc/kdbus/bus.h | 103 +
ipc/kdbus/connection.c | 2004 ++++++++++++++++++++
ipc/kdbus/connection.h | 262 +++
ipc/kdbus/domain.c | 350 ++++
ipc/kdbus/domain.h | 84 +
ipc/kdbus/endpoint.c | 232 +++
ipc/kdbus/endpoint.h | 68 +
ipc/kdbus/fs.c | 519 +++++
ipc/kdbus/fs.h | 25 +
ipc/kdbus/handle.c | 1134 +++++++++++
ipc/kdbus/handle.h | 20 +
ipc/kdbus/item.c | 309 +++
ipc/kdbus/item.h | 57 +
ipc/kdbus/limits.h | 95 +
ipc/kdbus/main.c | 72 +
ipc/kdbus/match.c | 535 ++++++
ipc/kdbus/match.h | 32 +
ipc/kdbus/message.c | 598 ++++++
ipc/kdbus/message.h | 133 ++
ipc/kdbus/metadata.c | 1066 +++++++++++
ipc/kdbus/metadata.h | 52 +
ipc/kdbus/names.c | 891 +++++++++
ipc/kdbus/names.h | 82 +
ipc/kdbus/node.c | 910 +++++++++
ipc/kdbus/node.h | 87 +
ipc/kdbus/notify.c | 244 +++
ipc/kdbus/notify.h | 30 +
ipc/kdbus/policy.c | 481 +++++
ipc/kdbus/policy.h | 51 +
ipc/kdbus/pool.c | 784 ++++++++
ipc/kdbus/pool.h | 47 +
ipc/kdbus/queue.c | 505 +++++
ipc/kdbus/queue.h | 108 ++
ipc/kdbus/reply.c | 262 +++
ipc/kdbus/reply.h | 68 +
ipc/kdbus/util.c | 317 ++++
ipc/kdbus/util.h | 133 ++
tools/testing/selftests/Makefile | 1 +
tools/testing/selftests/kdbus/.gitignore | 11 +
tools/testing/selftests/kdbus/Makefile | 46 +
tools/testing/selftests/kdbus/kdbus-enum.c | 95 +
tools/testing/selftests/kdbus/kdbus-enum.h | 14 +
tools/testing/selftests/kdbus/kdbus-test.c | 920 +++++++++
tools/testing/selftests/kdbus/kdbus-test.h | 85 +
tools/testing/selftests/kdbus/kdbus-util.c | 1646 ++++++++++++++++
tools/testing/selftests/kdbus/kdbus-util.h | 216 +++
tools/testing/selftests/kdbus/test-activator.c | 319 ++++
tools/testing/selftests/kdbus/test-attach-flags.c | 751 ++++++++
tools/testing/selftests/kdbus/test-benchmark.c | 427 +++++
tools/testing/selftests/kdbus/test-bus.c | 174 ++
tools/testing/selftests/kdbus/test-chat.c | 123 ++
tools/testing/selftests/kdbus/test-connection.c | 611 ++++++
tools/testing/selftests/kdbus/test-daemon.c | 66 +
tools/testing/selftests/kdbus/test-endpoint.c | 344 ++++
tools/testing/selftests/kdbus/test-fd.c | 710 +++++++
tools/testing/selftests/kdbus/test-free.c | 36 +
tools/testing/selftests/kdbus/test-match.c | 442 +++++
tools/testing/selftests/kdbus/test-message.c | 658 +++++++
tools/testing/selftests/kdbus/test-metadata-ns.c | 507 +++++
tools/testing/selftests/kdbus/test-monitor.c | 158 ++
tools/testing/selftests/kdbus/test-names.c | 184 ++
tools/testing/selftests/kdbus/test-policy-ns.c | 633 +++++++
tools/testing/selftests/kdbus/test-policy-priv.c | 1270 +++++++++++++
tools/testing/selftests/kdbus/test-policy.c | 81 +
tools/testing/selftests/kdbus/test-race.c | 313 +++
tools/testing/selftests/kdbus/test-sync.c | 368 ++++
tools/testing/selftests/kdbus/test-timeout.c | 99 +
77 files changed, 27818 insertions(+), 1 deletion(-)
create mode 100644 Documentation/kdbus.txt
create mode 100644 include/uapi/linux/kdbus.h
create mode 100644 ipc/kdbus/Makefile
create mode 100644 ipc/kdbus/bus.c
create mode 100644 ipc/kdbus/bus.h
create mode 100644 ipc/kdbus/connection.c
create mode 100644 ipc/kdbus/connection.h
create mode 100644 ipc/kdbus/domain.c
create mode 100644 ipc/kdbus/domain.h
create mode 100644 ipc/kdbus/endpoint.c
create mode 100644 ipc/kdbus/endpoint.h
create mode 100644 ipc/kdbus/fs.c
create mode 100644 ipc/kdbus/fs.h
create mode 100644 ipc/kdbus/handle.c
create mode 100644 ipc/kdbus/handle.h
create mode 100644 ipc/kdbus/item.c
create mode 100644 ipc/kdbus/item.h
create mode 100644 ipc/kdbus/limits.h
create mode 100644 ipc/kdbus/main.c
create mode 100644 ipc/kdbus/match.c
create mode 100644 ipc/kdbus/match.h
create mode 100644 ipc/kdbus/message.c
create mode 100644 ipc/kdbus/message.h
create mode 100644 ipc/kdbus/metadata.c
create mode 100644 ipc/kdbus/metadata.h
create mode 100644 ipc/kdbus/names.c
create mode 100644 ipc/kdbus/names.h
create mode 100644 ipc/kdbus/node.c
create mode 100644 ipc/kdbus/node.h
create mode 100644 ipc/kdbus/notify.c
create mode 100644 ipc/kdbus/notify.h
create mode 100644 ipc/kdbus/policy.c
create mode 100644 ipc/kdbus/policy.h
create mode 100644 ipc/kdbus/pool.c
create mode 100644 ipc/kdbus/pool.h
create mode 100644 ipc/kdbus/queue.c
create mode 100644 ipc/kdbus/queue.h
create mode 100644 ipc/kdbus/reply.c
create mode 100644 ipc/kdbus/reply.h
create mode 100644 ipc/kdbus/util.c
create mode 100644 ipc/kdbus/util.h
create mode 100644 tools/testing/selftests/kdbus/.gitignore
create mode 100644 tools/testing/selftests/kdbus/Makefile
create mode 100644 tools/testing/selftests/kdbus/kdbus-enum.c
create mode 100644 tools/testing/selftests/kdbus/kdbus-enum.h
create mode 100644 tools/testing/selftests/kdbus/kdbus-test.c
create mode 100644 tools/testing/selftests/kdbus/kdbus-test.h
create mode 100644 tools/testing/selftests/kdbus/kdbus-util.c
create mode 100644 tools/testing/selftests/kdbus/kdbus-util.h
create mode 100644 tools/testing/selftests/kdbus/test-activator.c
create mode 100644 tools/testing/selftests/kdbus/test-attach-flags.c
create mode 100644 tools/testing/selftests/kdbus/test-benchmark.c
create mode 100644 tools/testing/selftests/kdbus/test-bus.c
create mode 100644 tools/testing/selftests/kdbus/test-chat.c
create mode 100644 tools/testing/selftests/kdbus/test-connection.c
create mode 100644 tools/testing/selftests/kdbus/test-daemon.c
create mode 100644 tools/testing/selftests/kdbus/test-endpoint.c
create mode 100644 tools/testing/selftests/kdbus/test-fd.c
create mode 100644 tools/testing/selftests/kdbus/test-free.c
create mode 100644 tools/testing/selftests/kdbus/test-match.c
create mode 100644 tools/testing/selftests/kdbus/test-message.c
create mode 100644 tools/testing/selftests/kdbus/test-metadata-ns.c
create mode 100644 tools/testing/selftests/kdbus/test-monitor.c
create mode 100644 tools/testing/selftests/kdbus/test-names.c
create mode 100644 tools/testing/selftests/kdbus/test-policy-ns.c
create mode 100644 tools/testing/selftests/kdbus/test-policy-priv.c
create mode 100644 tools/testing/selftests/kdbus/test-policy.c
create mode 100644 tools/testing/selftests/kdbus/test-race.c
create mode 100644 tools/testing/selftests/kdbus/test-sync.c
create mode 100644 tools/testing/selftests/kdbus/test-timeout.c
From: Daniel Mack <daniel@zonque.org>
kdbus is a system for low-latency, low-overhead, easy to use
interprocess communication (IPC).
The interface to all functions in this driver is implemented via ioctls
on files exposed through a filesystem called 'kdbusfs'. The default
mount point of kdbusfs is /sys/fs/kdbus. This patch adds detailed
documentation about the kernel level API design.
Signed-off-by: Daniel Mack <daniel@zonque.org>
Signed-off-by: David Herrmann <redacted>
Signed-off-by: Djalal Harouni <redacted>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
---
Documentation/kdbus.txt | 2107 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 2107 insertions(+)
create mode 100644 Documentation/kdbus.txt
@@ -0,0 +1,2107 @@+D-Bus is a system for powerful, easy to use interprocess communication (IPC).++The focus of this document is an overview of the low-level, native kernel D-Bus+transport called kdbus. Kdbus exposes its functionality via files in a+filesystem called 'kdbusfs'. All communication between processes takes place+via ioctls on files exposed through the mount point of a kdbusfs. The default+mount point of kdbusfs is /sys/fs/kdbus.++Please note that kdbus was designed as transport layer for D-Bus, but is in no+way limited, nor controlled by the D-Bus protocol specification. The D-Bus+protocol is one possible application layer on top of kdbus.++For the general D-Bus protocol specification, the payload format, the+marshaling, and the communication semantics, please refer to:+ http://dbus.freedesktop.org/doc/dbus-specification.html++For a kdbus specific userspace library implementation please refer to:+ http://cgit.freedesktop.org/systemd/systemd/tree/src/systemd/sd-bus.h++Articles about D-Bus and kdbus:+ http://lwn.net/Articles/580194/+++1. Terminology+===============================================================================++ Domain:+ A domain is created each time a kdbusfs is mounted. Each process that is+ capable to mount a new instance of a kdbusfs will have its own kdbus+ hierarchy. Each domain (ie, each mount point) offers its own "control"+ file to create new buses. Domains have no connection to each other and+ cannot see nor talk to each other. See section 5 for more details.++ Bus:+ A bus is a named object inside a domain. Clients exchange messages+ over a bus. Multiple buses themselves have no connection to each other;+ messages can only be exchanged on the same bus. The default endpoint of+ a bus, where clients establish the connection to, is the "bus" file+ /sys/fs/kdbus/<bus name>/bus.+ Common operating system setups create one "system bus" per system, and one+ "user bus" for every logged-in user. Applications or services may create+ their own private buses. See section 5 for more details.++ Endpoint:+ An endpoint provides a file to talk to a bus. Opening an endpoint+ creates a new connection to the bus to which the endpoint belongs. All+ endpoints have unique names and are accessible as files underneath the+ directory of a bus, e.g., /sys/fs/kdbus/<bus>/<endpoint>+ Every bus has a default endpoint called "bus". A bus can optionally offer+ additional endpoints with custom names to provide restricted access to the+ bus. Custom endpoints carry additional policy which can be used to create+ sandboxes with locked-down, limited, filtered access to a bus. See+ section 5 for more details.++ Connection:+ A connection to a bus is created by opening an endpoint file of a bus and+ becoming an active client with the HELLO exchange. Every ordinary client+ connection has a unique identifier on the bus and can address messages to+ every other connection on the same bus by using the peer's connection id+ as the destination. See section 6 for more details.++ Pool:+ Each connection allocates a piece of shmem-backed memory that is used+ to receive messages and answers to ioctl commands from the kernel. It is+ never used to send anything to the kernel. In order to access that memory,+ userspace must mmap() it into its address space.+ See section 12 for more details.++ Well-known Name:+ A connection can, in addition to its implicit unique connection id, request+ the ownership of a textual well-known name. Well-known names are noted in+ reverse-domain notation, such as com.example.service1. Connections offering+ a service on a bus are usually reached by its well-known name. The analogy+ of connection id and well-known name is an IP address and a DNS name+ associated with that address.++ Message:+ Connections can exchange messages with other connections by addressing+ the peers with their connection id or well-known name. A message consists+ of a message header with kernel-specific information on how to route the+ message, and the message payload, which is a logical byte stream of+ arbitrary size. Messages can carry additional file descriptors to be passed+ from one connection to another. Every connection can specify which set of+ metadata the kernel should attach to the message when it is delivered+ to the receiving connection. Metadata contains information like: system+ timestamps, uid, gid, tid, proc-starttime, well-known-names, process comm,+ process exe, process argv, cgroup, capabilities, seclabel, audit session,+ loginuid and the connection's human-readable name.+ See section 7 and 13 for more details.++ Item:+ The API of kdbus implements a notion of items, submitted through and+ returned by most ioctls, and stored inside data structures in the+ connection's pool. See section 4 for more details.++ Broadcast and Match:+ Broadcast messages are potentially sent to all connections of a bus. By+ default, the connections will not actually receive any of the sent+ broadcast messages; only after installing a match for specific message+ properties, a broadcast message passes this filter.+ See section 10 for more details.++ Policy:+ A policy is a set of rules that define which connections can see, talk to,+ or register a well-know name on the bus. A policy is attached to buses and+ custom endpoints, and modified by policy holder connections or owners of+ custom endpoints. See section 11 for more details.+ See section 11 for more details.++ Privileged bus users:+ A user connecting to the bus is considered privileged if it is either the+ creator of the bus, or if it has the CAP_IPC_OWNER capability flag set.+++2. Control Files Layout+===============================================================================++The kdbus interface is exposed through files in its kdbusfs mount point+(defaults to /sys/fs/kdbus):++ /sys/fs/kdbus (mount point of kdbusfs)+ |-- control (domain control-file)+ |-- 0-system (bus of user uid=0)+ | |-- bus (default endpoint of bus '0-system')+ | `-- ep.apache (custom endpoint of bus '0-system')+ |-- 1000-user (bus of user uid=1000)+ | `-- bus (default endpoint of bus '1000-user')+ `-- 2702-user (bus of user uid=2702)+ |-- bus (default endpoint of bus '2702-user')+ `-- ep.app (custom endpoint of bus '2702-user')+++3. Data Structures and flags+===============================================================================++3.1 Data structures and interconnections+----------------------------------------++ +--------------------------------------------------------------------------++ | Domain (Mount Point) |+ | /sys/fs/kdbus/control |+ | +----------------------------------------------------------------------+ |+ | | Bus (System Bus) | |+ | | /sys/fs/kdbus/0-system/ | |+ | | +-------------------------------+ +--------------------------------+ | |+ | | | Endpoint | | Endpoint | | |+ | | | /sys/fs/kdbus/0-system/bus | | /sys/fs/kdbus/0-system/ep.app | | |+ | | +-------------------------------+ +--------------------------------+ | |+ | | +--------------+ +--------------+ +--------------+ +---------------+ | |+ | | | Connection | | Connection | | Connection | | Connection | | |+ | | | :1.22 | | :1.25 | | :1.55 | | :1.81 | | |+ | | +--------------+ +--------------+ +--------------+ +---------------+ | |+ | +----------------------------------------------------------------------+ |+ | |+ | +----------------------------------------------------------------------+ |+ | | Bus (User Bus for UID 2702) | |+ | | /sys/fs/kdbus/2702-user/ | |+ | | +-------------------------------+ +--------------------------------+ | |+ | | | Endpoint | | Endpoint | | |+ | | | /sys/fs/kdbus/2702-user/bus | | /sys/fs/kdbus/2702-user/ep.app | | |+ | | +-------------------------------+ +--------------------------------+ | |+ | | +--------------+ +--------------+ +--------------+ +---------------+ | |+ | | | Connection | | Connection | | Connection | | Connection | | |+ | | | :1.22 | | :1.25 | | :1.55 | | :1.81 | | |+ | | +--------------+ +--------------+ +--------------------------------+ | |+ | +----------------------------------------------------------------------+ |+ +--------------------------------------------------------------------------+++The above description uses the D-Bus notation of unique connection names that+adds a ":1." prefix to the connection's unique ID. kdbus itself doesn't+use that notation, neither internally nor externally. However, libraries and+other userspace code that aims for compatibility to D-Bus might.++3.2 Flags+---------++All ioctls used in the communication with the driver contain three 64-bit+fields: 'flags', 'kernel_flags' and 'return_flags'. All of them are specific+to the ioctl used.++In 'flags', the behavior of the command can be tweaked. All bits that are not+recognized by the kernel in this field are rejected, and the ioctl fails with+-EINVAL.++In 'kernel_flags', the kernel driver writes back the mask of supported bits+upon each call, and sets the KDBUS_FLAGS_KERNEL bit. This is a way to probe+possible kernel features and make userspace code forward and backward+compatible.++In 'return_flags', the kernel can return results of the command, in addition+to the actual return value. This is mostly to inform userspace about non-fatal+conditions that occurred during the execution of the command.+++4. Items+===============================================================================++To flexibly augment transport structures, data blobs of type struct kdbus_item+can be attached to the structs passed into the ioctls. Some ioctls make items+of certain types mandatory, others are optional. Unsupported items will cause+the ioctl to fail -EINVAL.++The total size of an item is variable and is in some cases defined by the item+type. In other cases, they can be of arbitrary length (for instance, a string).++Items are also used for information stored in a connection's pool, such as+received messages, name lists or requested connection or bus owner information.++Whenever items are used as part of the kdbus kernel API, they are embedded in+structs that have an overall size of their own, so there can be multiple items+per ioctl.++The kernel expects all items to be aligned to 8-byte boundaries. Unaligned+items or such that are unsupported by the ioctl are rejected.++A simple iterator in userspace would iterate over the items until the items+have reached the embedding structure's overall size. An example implementation+of such an iterator can be found in tools/testing/selftests/kdbus/kdbus-util.h.+++5. Creation of new domains, buses and endpoints+===============================================================================+++5.1 Domains+-----------++A domain is a container of buses. Domains themselves do not provide any IPC+functionality. Their sole purpose is to manage buses allocated in their+domain. Each time kdbusfs is mounted, a new kdbus domain is created, with its+own 'control' file. The lifetime of the domain ends once the user has unmounted+the kdbusfs. If you mount kdbusfs multiple times, each will have its own kdbus+domain internally. Operations performed on one domain do not affect any+other domain.++The full kdbusfs hierarchy, any sub-directory, or file can be bind-mounted to+an external mount point and will remain fully functional. The kdbus domain and+any linked resources stay available until the original mount and all subsequent+bind-mounts have been unmounted.++During creation, domains pin the user-namespace of the creator and use+it as controlling user-namespace for this domain. Any user accounting is done+relative to that user-namespace.++Newly created kdbus domains do not have any bus pre-created. The only resource+available is a 'control' file, which is used to manage kdbus domains.+Currently, 'control' files are exclusively used to create buses via+KDBUS_CMD_BUS_MAKE, but further ioctls might be added in the future.+++5.2 Buses+---------++A bus is a shared resource between connections to transmit messages. Each bus+is independent and operations on the bus will not have any effect on other+buses. A bus is a management entity, that controls the addresses of its+connections, their policies and message transactions performed via this bus.++Each bus is bound to the domain it was created on. It has a custom name that is+unique across all buses of a domain. In kdbusfs, a bus is presented as a+directory. No operations can be performed on the bus itself, instead you need+to perform those on an endpoint associated with the bus. Endpoints are+accessible as files underneath the bus directory. A default endpoint called+"bus" is provided on each bus.++Bus names may be chosen freely except for one restriction: the name+must be prefixed with the numeric UID of the creator and a dash. This+is required to avoid namespace clashes between different users. When+creating a bus the name must be passed in properly formatted, or the+kernel will refuse creation of the bus. Example: "1047-foobar" is an+OK name for a bus registered by a user with UID 1047. However,+"1024-foobar" is not, and neither is "foobar".+The UID must be provided in the user-namespace of the parent domain.++To create a new bus, you need to open the control file of a domain and run the+KDBUS_CMD_BUS_MAKE ioctl. The control file descriptor that was used to issue+KDBUS_CMD_BUS_MAKE must not have been used for any other control-ioctl before+and needs to be kept open for the entire life-time of the created bus. Closing+it will immediately cleanup the entire bus and all its associated resources and+endpoints. Every control file descriptor can only be used to create a single+new bus; from that point on, it is not used for any further communication until+the final close().++Each bus will generate a random, 128-bit UUID upon creation. It will be+returned to creators of connections through kdbus_cmd_hello.id128 and can+be used by userspace to uniquely identify buses, even across different machines+or containers. The UUID will have its variant bits set to 'DCE', and denote+version 4 (random).++When creating buses, a variable list of items that must be passed in+the items array is expected otherwise bus creation will fail.+++5.3 Endpoints+-------------++Endpoints are entry points to a bus. By default, each bus has a default+endpoint called 'bus'. The bus owner has the ability to create custom+endpoints with specific names, permissions, and policy databases (see below).+An endpoint is presented as file underneath the directory of the parent bus.++To create a custom endpoint, open the default endpoint ('bus') and use the+KDBUS_CMD_ENDPOINT_MAKE ioctl with "struct kdbus_cmd_make". Custom endpoints+always have a policy database that, by default, forbids any operation. You have+to explicitly install policy entries to allow any operation on this endpoint.+Once KDBUS_CMD_ENDPOINT_MAKE succeeded, this file descriptor will manage the+newly created endpoint resource. It cannot be used to manage further resources.++Endpoint names may be chosen freely except for one restriction: the name+must be prefixed with the numeric UID of the creator and a dash. This+is required to avoid namespace clashes between different users. When+creating an endpoint the name must be passed in properly formatted, or the+kernel will refuse creation of the endpoint. Example: "1047-foobar" is an+OK name for an endpoint registered by a user with UID 1047. However,+"1024-foobar" is not, and neither is "foobar".+The UID must be provided in the user-namespace of the parent domain.++To create connections to a bus, you use KDBUS_CMD_HELLO. See section 6 for+details. Note that once KDBUS_CMD_HELLO succeeded, this file descriptor manages+the newly created connection resource. It cannot be used to manage further+resources.+++5.4 Creating buses and endpoints+--------------------------------++KDBUS_CMD_BUS_MAKE, and KDBUS_CMD_ENDPOINT_MAKE take a+struct kdbus_cmd_make argument.++struct kdbus_cmd_make {+ __u64 size;+ The overall size of the struct, including its items.++ __u64 flags;+ The flags for creation.++ KDBUS_MAKE_ACCESS_GROUP+ Make the bus or endpoint file group-accessible++ KDBUS_MAKE_ACCESS_WORLD+ Make the bus or endpoint file world-accessible++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ struct kdbus_item items[0];+ A list of items that has specific meanings for KDBUS_CMD_BUS_MAKE+ and KDBUS_CMD_ENDPOINT_MAKE (see above).++ Following items are expected for KDBUS_CMD_BUS_MAKE:+ KDBUS_ITEM_MAKE_NAME+ Contains a string to identify the bus name.++ KDBUS_ITEM_BLOOM_PARAMETER+ Bus-wide bloom parameters passed in a dbus_bloom_parameter struct++ KDBUS_ITEM_ATTACH_FLAGS_RECV+ An optional item that contains a set of required attach flags+ that connections must allow. This item is used as a negotiation+ measure during connection creation. If connections do not satisfy+ the bus requirements, they are not allowed on the bus.+ If not set, the bus does not require any metadata to be attached,+ in this case connections are free to set their own attach flags.++ KDBUS_ITEM_ATTACH_FLAGS_SEND+ An optional item that contains a set of attach flags that are+ returned to connections when they query the bus creator metadata.+ If not set, no metadata is returned.++ Unrecognized items are rejected, and the ioctl will fail with -EINVAL.+};+++6. Connections+===============================================================================+++6.1 Connection IDs and well-known connection names+--------------------------------------------------++Connections are identified by their connection id, internally implemented as a+uint64_t counter. The IDs of every newly created bus start at 1, and every new+connection will increment the counter by 1. The ids are not reused.++In higher level tools, the user visible representation of a connection is+defined by the D-Bus protocol specification as ":1.<id>".++Messages with a specific uint64_t destination id are directly delivered to+the connection with the corresponding id. Messages with the special destination+id KDBUS_DST_ID_BROADCAST are broadcast messages and are potentially delivered+to all known connections on the bus; clients interested in broadcast messages+need to subscribe to the specific messages they are interested, though before+any broadcast message reaches them.++Messages synthesized and sent directly by the kernel will carry the special+source id KDBUS_SRC_ID_KERNEL (0).++In addition to the unique uint64_t connection id, established connections can+request the ownership of well-known names, under which they can be found and+addressed by other bus clients. A well-known name is associated with one and+only one connection at a time. See section 8 on name acquisition and the+name registry, and the validity of names.++Messages can specify the special destination id 0 and carry a well-known name+in the message data. Such a message is delivered to the destination connection+which owns that well-known name.++ +-------------------------------------------------------------------------++ | +---------------+ +---------------------------+ |+ | | Connection | | Message | -----------------+ |+ | | :1.22 | --> | src: 22 | | |+ | | | | dst: 25 | | |+ | | | | | | |+ | | | | | | |+ | | | +---------------------------+ | |+ | | | | |+ | | | <--------------------------------------+ | |+ | +---------------+ | | |+ | | | |+ | +---------------+ +---------------------------+ | | |+ | | Connection | | Message | -----+ | |+ | | :1.25 | --> | src: 25 | | |+ | | | | dst: 0xffffffffffffffff | -------------+ | |+ | | | | (KDBUS_DST_ID_BROADCAST) | | | |+ | | | | | ---------+ | | |+ | | | +---------------------------+ | | | |+ | | | | | | |+ | | | <--------------------------------------------------+ |+ | +---------------+ | | |+ | | | |+ | +---------------+ +---------------------------+ | | |+ | | Connection | | Message | --+ | | |+ | | :1.55 | --> | src: 55 | | | | |+ | | | | dst: 0 / org.foo.bar | | | | |+ | | | | | | | | |+ | | | | | | | | |+ | | | +---------------------------+ | | | |+ | | | | | | |+ | | | <------------------------------------------+ | |+ | +---------------+ | | |+ | | | |+ | +---------------+ | | |+ | | Connection | | | |+ | | :1.81 | | | |+ | | org.foo.bar | | | |+ | | | | | |+ | | | | | |+ | | | <-----------------------------------+ | |+ | | | | |+ | | | <----------------------------------------------+ |+ | +---------------+ |+ +-------------------------------------------------------------------------++++6.2 Creating connections+------------------------++A connection to a bus is created by opening an endpoint file of a bus and+becoming an active client with the KDBUS_CMD_HELLO ioctl. Every connected client+connection has a unique identifier on the bus and can address messages to every+other connection on the same bus by using the peer's connection id as the+destination.++The KDBUS_CMD_HELLO ioctl takes the following struct as argument.++struct kdbus_cmd_hello {+ __u64 size;+ The overall size of the struct, including all attached items.++ __u64 flags;+ Flags to apply to this connection:++ KDBUS_HELLO_ACCEPT_FD+ When this flag is set, the connection can be sent file descriptors+ as message payload. If it's not set, any attempt of doing so will+ result in -ECOMM on the sender's side.++ KDBUS_HELLO_ACTIVATOR+ Make this connection an activator (see below). With this bit set,+ an item of type KDBUS_ITEM_NAME has to be attached which describes+ the well-known name this connection should be an activator for.++ KDBUS_HELLO_POLICY_HOLDER+ Make this connection a policy holder (see below). With this bit set,+ an item of type KDBUS_ITEM_NAME has to be attached which describes+ the well-known name this connection should hold a policy for.++ KDBUS_HELLO_MONITOR+ Make this connection an eaves-dropping connection. See section 6.8 for+ more information.++To also receive broadcast messages,+ the connection has to upload appropriate matches as well.+ This flag is only valid for privileged bus connections.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ __u64 attach_flags_send;+ Set the bits for metadata this connection permits to be sent to the+ receiving peer. Only metadata items that are both allowed to be sent by+ the sender and that are requested by the receiver will effectively be+ attached to the message eventually. Note, however, that the bus may+ optionally enforce some of those bits to be set. If the match fails,+ -ECONNREFUSED will be returned. In either case, this field will be set+ to the mask of metadata items that are enforced by the bus. The+ KDBUS_FLAGS_KERNEL bit will as well be set.++ __u64 attach_flags_recv;+ Request the attachment of metadata for each message received by this+ connection. The metadata actually attached may actually augment the list+ of requested items. See section 13 for more details.++ __u64 bus_flags;+ Upon successful completion of the ioctl, this member will contain the+ flags of the bus it connected to.++ __u64 id;+ Upon successful completion of the ioctl, this member will contain the+ id of the new connection.++ __u64 pool_size;+ The size of the communication pool, in bytes. The pool can be accessed+ by calling mmap() on the file descriptor that was used to issue the+ KDBUS_CMD_HELLO ioctl.++ __u64 offset;+ The kernel will return the offset in the pool where returned details+ will be stored.++ __u8 id128[16];+ Upon successful completion of the ioctl, this member will contain the+ 128 bit wide UUID of the connected bus.++ struct kdbus_item items[0];+ Variable list of items to add optional additional information. The+ following items are currently expected/valid:++ KDBUS_ITEM_CONN_DESCRIPTION+ Contains a string to describes this connection's name, so it can be+ identified later.++ KDBUS_ITEM_NAME+ KDBUS_ITEM_POLICY_ACCESS+ For activators and policy holders only, combinations of these two+ items describe policy access entries (see section about policy).++ KDBUS_ITEM_CREDS+ KDBUS_ITEM_PIDS+ KDBUS_ITEM_SECLABEL+ Privileged bus users may submit these types in order to create+ connections with faked credentials. This information will be returned+ when peer information is queried by KDBUS_CMD_CONN_INFO. See section+ 13 for more information.++ Items of other types are rejected, and the ioctl will fail with -EINVAL.+};++At the offset returned in the 'offset' field of struct kdbus_cmd_hello, the+kernel will store items of the following types:++ KDBUS_ITEM_BLOOM_PARAMETER+ Bloom filter parameter as defined by the bus creator (see below).++The offset in the pool has to be freed with the KDBUS_CMD_FREE ioctl.++6.3 Activator and policy holder connection+------------------------------------------++An activator connection is a placeholder for a well-known name. Messages sent+to such a connection can be used by userspace to start an implementer+connection, which will then get all the messages from the activator copied+over. An activator connection cannot be used to send any message.++A policy holder connection only installs a policy for one or more names.+These policy entries are kept active as long as the connection is alive, and+are removed once it terminates. Such a policy connection type can be used to+deploy restrictions for names that are not yet active on the bus. A policy+holder connection cannot be used to send any message.++The creation of activator, policy holder or monitor connections is an operation+restricted to privileged users on the bus (see section "Terminology").+++6.4 Retrieving information on a connection+------------------------------------------++The KDBUS_CMD_CONN_INFO ioctl can be used to retrieve credentials and+properties of the initial creator of a connection. This ioctl uses the+following struct:++struct kdbus_cmd_info {+ __u64 size;+ The overall size of the struct, including the name with its 0-byte string+ terminator.++ __u64 flags;+ Specify which metadata items should be attached to the answer.+ See section 13 for more details.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ __u64 id;+ The connection's numerical ID to retrieve information for. If set to+ non-zero value, the 'name' field is ignored.++ __u64 offset;+ When the ioctl returns, this value will yield the offset of the connection+ information inside the caller's pool.++ __u64 info_size;+ The kernel will return the size of the returned information, so applications+ can optionally mmap specific parts of the pool.++ struct kdbus_item items[0];+ The optional item list, containing the well-known name to look up as+ a KDBUS_ITEM_OWNED_NAME. Only required if the 'id' field is set to 0.+ Items of other types are rejected, and the ioctl will fail with -EINVAL.+};++After the ioctl returns, the following struct will be stored in the caller's+pool at 'offset'.++struct kdbus_info {+ __u64 size;+ The overall size of the struct, including all its items.++ __u64 id;+ The connection's unique ID.++ __u64 flags;+ The connection's flags as specified when it was created.++ struct kdbus_item items[0];+ Depending on the 'flags' field in struct kdbus_cmd_info, items of+ types KDBUS_ITEM_OWNED_NAME and KDBUS_ITEM_CONN_DESCRIPTION are followed+ here.+};++Once the caller is finished with parsing the return buffer, it needs to call+KDBUS_CMD_FREE for the offset.+++6.5 Getting information about a connection's bus creator+--------------------------------------------------------++The KDBUS_CMD_BUS_CREATOR_INFO ioctl takes the same struct as+KDBUS_CMD_CONN_INFO but is used to retrieve information about the creator of+the bus the connection is attached to. The metadata returned by this call is+collected during the creation of the bus and is never altered afterwards, so+it provides pristine information on the task that created the bus, at the+moment when it did so.++In response to this call, a slice in the connection's pool is allocated and+filled with an object of type struct kdbus_info, pointed to by the ioctl's+'offset' field.++struct kdbus_info {+ __u64 size;+ The overall size of the struct, including all its items.++ __u64 id;+ The bus ID++ __u64 flags;+ The bus flags as specified when it was created.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ struct kdbus_item items[0];+ Metadata information is stored in items here. The item list contains+ a KDBUS_ITEM_MAKE_NAME item that indicates the bus name of the+ calling connection.+};++Once the caller is finished with parsing the return buffer, it needs to call+KDBUS_CMD_FREE for the offset.+++6.6 Updating connection details+-------------------------------++Some of a connection's details can be updated with the KDBUS_CMD_CONN_UPDATE+ioctl, using the file descriptor that was used to create the connection.+The update command uses the following struct.++struct kdbus_cmd_update {+ __u64 size;+ The overall size of the struct, including all its items.++ __u64 flags;+ Currently no flags are supported. Reserved for future use.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ struct kdbus_item items[0];+ Items to describe the connection details to be updated. The following item+ types are supported:++ KDBUS_ITEM_ATTACH_FLAGS_SEND+ Supply a new set of items that this connection permits to be sent along+ with messages.++ KDBUS_ITEM_ATTACH_FLAGS_RECV+ Supply a new set of items to be attached to each message.++ KDBUS_ITEM_NAME+ KDBUS_ITEM_POLICY_ACCESS+ Policy holder connections may supply a new set of policy information+ with these items. For other connection types, -EOPNOTSUPP is returned.++ Items of other types are rejected, and the ioctl will fail with -EINVAL.+};+++6.7 Termination+---------------++A connection can be terminated by simply closing its file descriptor. All+pending incoming messages will be discarded, and the memory in the pool will+be freed.++An alternative way of closing down a connection is calling the KDBUS_CMD_BYEBYE+ioctl on it, which will only succeed if the message queue of the connection is+empty at the time of closing, otherwise, -EBUSY is returned.++When this ioctl returns successfully, the connection has been terminated and+won't accept any new messages from remote peers. This way, a connection can+be terminated race-free, without losing any messages.+++6.8 Monitor connections ('eavesdropper')+----------------------------------------++Eavesdropping connections are created by setting the KDBUS_HELLO_MONITOR flag+in struct kdbus_hello.flags. Such connections have all properties of any other,+regular connection, except for the following details:++ * They will get every message sent over the bus, both unicasts and broadcasts++ * Installing matches for broadcast messages is neither necessary nor allowed++ * They cannot send messages or be directly addressed as receiver++ * Their creation and destruction will not cause KDBUS_ITEM_ID_{ADD,REMOVE}+ notifications to be generated, so other connections cannot detect the+ presence of an eavesdropper.+++7. Messages+===============================================================================++Messages consist of a fixed-size header followed directly by a list of+variable-sized data 'items'. The overall message size is specified in the+header of the message. The chain of data items can contain well-defined+message metadata fields, raw data, references to data, or file descriptors.+++7.1 Sending messages+--------------------++Messages are passed to the kernel with the KDBUS_CMD_SEND ioctl. Depending+on the destination address of the message, the kernel delivers the message to+the specific destination connection or to all connections on the same bus.+Sending messages across buses is not possible. Messages are always queued in+the memory pool of the destination connection (see below).++The KDBUS_CMD_SEND ioctl uses struct kdbus_cmd_send to describe the message+transfer.++struct kdbus_cmd_send {+ __u64 size;+ The overall size of the struct, including the attached items.++ __u64 flags;+ Flags for message delivery:++ KDBUS_SEND_SYNC_REPLY+ By default, all calls to kdbus are considered asynchronous,+ non-blocking. However, as there are many use cases that need to wait+ for a remote peer to answer a method call, there's a way to send a+ message and wait for a reply in a synchronous fashion. This is what+ the KDBUS_MSG_SYNC_REPLY controls. The KDBUS_CMD_SEND ioctl will block+ until the reply has arrived, the timeout limit is reached, in case the+ remote connection was shut down, or if interrupted by a signal before+ any reply; see signal(7).++ The offset of the reply message in the sender's pool is stored in in+ 'offset_reply' when the ioctl has returned without error. Hence, there+ is no need for another KDBUS_CMD_RECV ioctl or anything else to receive+ the reply.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call of+ KDBUS_CMD_SEND.++ __u64 kernel_msg_flags;+ Valid bits for message flags, returned by the kernel upon each call of+ KDBUS_CMD_SEND.++ __u64 return_flags;+ Kernel-provided flags, returning non-fatal errors that occurred during+ send. Currently unused.++ __u64 msg_address;+ Userspace has to provide a pointer to a message (struct kdbus_msg) to send.++ struct kdbus_msg_info reply;+ Only used for synchronous replies. See description of struct kdbus_cmd_recv+ for more details.++ struct kdbus_item items[0];+ The following items are currently recognized:++ KDBUS_ITEM_CANCEL_FD+ When this optional item is passed in, and the call is executed as SYNC+ call, the passed in file descriptor can be used as alternative+ cancellation point. The kernel will call poll() on this file descriptor,+ and if it reports any incoming bytes, the blocking send operation will+ be canceled, and the call will return -ECANCELED. Any type of file+ descriptor that implements poll() can be used as payload to this item.+ For asynchronous message sending, this item is accepted but ignored.++ All other items are rejected, and the ioctl will fail with -EINVAL.+};++The message referenced by 'msg_address' above has the following layout.++struct kdbus_msg {+ __u64 size;+ The overall size of the struct, including the attached items.++ __u64 flags;+ KDBUS_MSG_EXPECT_REPLY+ Expect a reply from the remote peer to this message. With this bit set,+ the timeout_ns field must be set to a non-zero number of nanoseconds in+ which the receiving peer is expected to reply. If such a reply is not+ received in time, the sender will be notified with a timeout message+ (see below). The value must be an absolute value, in nanoseconds and+ based on CLOCK_MONOTONIC.++ For a message to be accepted as reply, it must be a direct message to+ the original sender (not a broadcast), and its kdbus_msg.reply_cookie+ must match the previous message's kdbus_msg.cookie.++ Expected replies also temporarily open the policy of the sending+ connection, so the other peer is allowed to respond within the given+ time window.++ KDBUS_MSG_NO_AUTO_START+ By default, when a message is sent to an activator connection, the+ activator notified and will start an implementer. This flag inhibits+ that behavior. With this bit set, and the remote being an activator,+ -EADDRNOTAVAIL is returned from the ioctl.++ __s64 priority;+ The priority of this message. Receiving messages (see below) may+ optionally be constrained to messages of a minimal priority. This+ allows for use cases where timing critical data is interleaved with+ control data on the same connection. If unused, the priority should be+ set to zero.++ __u64 dst_id;+ The numeric ID of the destination connection, or KDBUS_DST_ID_BROADCAST+ (~0ULL) to address every peer on the bus, or KDBUS_DST_ID_NAME (0) to look+ it up dynamically from the bus' name registry. In the latter case, an item+ of type KDBUS_ITEM_DST_NAME is mandatory.++ __u64 src_id;+ Upon return of the ioctl, this member will contain the sending+ connection's numerical ID. Should be 0 at send time.++ __u64 payload_type;+ Type of the payload in the actual data records. Currently, only+ KDBUS_PAYLOAD_DBUS is accepted as input value of this field. When+ receiving messages that are generated by the kernel (notifications),+ this field will yield KDBUS_PAYLOAD_KERNEL.++ __u64 cookie;+ Cookie of this message, for later recognition. Also, when replying+ to a message (see above), the cookie_reply field must match this value.++ __u64 timeout_ns;+ If the message sent requires a reply from the remote peer (see above),+ this field contains the timeout in absolute nanoseconds based on+ CLOCK_MONOTONIC.++ __u64 cookie_reply;+ If the message sent is a reply to another message, this field must+ match the cookie of the formerly received message.++ struct kdbus_item items[0];+ A dynamically sized list of items to contain additional information.+ The following items are expected/valid:++ KDBUS_ITEM_PAYLOAD_VEC+ KDBUS_ITEM_PAYLOAD_MEMFD+ KDBUS_ITEM_FDS+ Actual data records containing the payload. See section "Passing of+ Payload Data".++ KDBUS_ITEM_BLOOM_FILTER+ Bloom filter for matches (see below).++ KDBUS_ITEM_DST_NAME+ Well-known name to send this message to. Required if dst_id is set+ to KDBUS_DST_ID_NAME. If a connection holding the given name can't+ be found, -ESRCH is returned.+ For messages to a unique name (ID), this item is optional. If present,+ the kernel will make sure the name owner matches the given unique name.+ This allows userspace tie the message sending to the condition that a+ name is currently owned by a certain unique name.+};++The message will be augmented by the requested metadata items when queued into+the receiver's pool. See also section 13.2 ("Metadata and namespaces").+++7.2 Message layout+------------------++The layout of a message is shown below.++ +-------------------------------------------------------------------------++ | Message |+ | +---------------------------------------------------------------------+ |+ | | Header | |+ | | size: overall message size, including the data records | |+ | | destination: connection id of the receiver | |+ | | source: connection id of the sender (set by kernel) | |+ | | payload_type: "DBusDBus" textual identifier stored as uint64_t | |+ | +---------------------------------------------------------------------+ |+ | +---------------------------------------------------------------------+ |+ | | Data Record | |+ | | size: overall record size (without padding) | |+ | | type: type of data | |+ | | data: reference to data (address or file descriptor) | |+ | +---------------------------------------------------------------------+ |+ | +---------------------------------------------------------------------+ |+ | | padding bytes to the next 8 byte alignment | |+ | +---------------------------------------------------------------------+ |+ | +---------------------------------------------------------------------+ |+ | | Data Record | |+ | | size: overall record size (without padding) | |+ | | ... | |+ | +---------------------------------------------------------------------+ |+ | +---------------------------------------------------------------------+ |+ | | padding bytes to the next 8 byte alignment | |+ | +---------------------------------------------------------------------+ |+ | +---------------------------------------------------------------------+ |+ | | Data Record | |+ | | size: overall record size | |+ | | ... | |+ | +---------------------------------------------------------------------+ |+ | ... further data records ... |+ +-------------------------------------------------------------------------++++7.3 Passing of Payload Data+---------------------------++When connecting to the bus, receivers request a memory pool of a given size,+large enough to carry all backlog of data enqueued for the connection. The+pool is internally backed by a shared memory file which can be mmap()ed by+the receiver.++KDBUS_MSG_PAYLOAD_VEC:+ Messages are directly copied by the sending process into the receiver's pool,+ that way two peers can exchange data by effectively doing a single-copy from+ one process to another, the kernel will not buffer the data anywhere else.++KDBUS_MSG_PAYLOAD_MEMFD:+ Messages can reference memfd files which contain the data.+ memfd files are tmpfs-backed files that allow sealing of the content of the+ file, which prevents all writable access to the file content.+ Only memfds that have (F_SEAL_SHRINK|F_SEAL_GROW|F_SEAL_WRITE|F_SEAL_SEAL) set+ are accepted as payload data, which enforces reliable passing of data.+ The receiver can assume that neither the sender nor anyone else can alter the+ content after the message is sent.+ Apart from the sender filling-in the content into memfd files, the data will+ be passed as zero-copy from one process to another, read-only, shared between+ the peers.++The sender must not make any assumptions on the type how data is received by the+remote peer. The kernel is free to re-pack multiple VEC and MEMFD payloads. For+instance, the kernel may decide to merge multiple VECs into a single VEC, inline+MEMFD payloads into memory or merge all passed VECs into a single MEMFD.+However, the kernel preserves the order of passed data. This means, the order of+all VEC and MEMFD items is not changed in respect to each other.++In other words: All passed VEC and MEMFD data payloads are treated as a single+stream of data that may be received by the remote peer in a different set of+hunks than it was sent as.+++7.4 Receiving messages+----------------------++Messages are received by the client with the KDBUS_CMD_RECV ioctl. The endpoint+file of the bus supports poll() to wake up the receiving process when new+messages are queued up to be received.++With the KDBUS_CMD_RECV ioctl, a struct kdbus_cmd_recv is used.++struct kdbus_cmd_recv {+ __u64 size;+ The overall size of the struct, including the attached items.++ __u64 flags;+ Flags to control the receive command.++ KDBUS_RECV_PEEK+ Just return the location of the next message. Do not install file+ descriptors or anything else. This is usually used to determine the+ sender of the next queued message.++ KDBUS_RECV_DROP+ Drop the next message without doing anything else with it, and free the+ pool slice. This a short-cut for KDBUS_RECV_PEEK and KDBUS_CMD_FREE.++ KDBUS_RECV_USE_PRIORITY+ Use the priority field (see below).++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Kernel-provided flags, returning non-fatal errors that occurred during+ send. Currently unused.++ __s64 priority;+ With KDBUS_RECV_USE_PRIORITY set in flags, receive the next message in+ the queue with at least the given priority. If no such message is waiting+ in the queue, -ENOMSG is returned.++ __u64 dropped_msgs;+ If the CMD_RECV ioctl fails with EOVERFLOW, this field is filled by+ the kernel with the number of messages that couldn't be transmitted to+ this connection. In that case, the @offset member must not be accessed.++ struct kdbus_msg_info msg;+ Embedded struct to be filled when the command succeeded (see below).++ struct kdbus_item items[0];+ Items to specify further details for the receive command. Currently unused.+};++Both 'struct kdbus_cmd_recv' and 'struct kdbus_cmd_send' embed 'struct+kdbus_msg_info'. For the SEND ioctl, it is used to catch synchronous replies,+if one was requested, and is unused otherwise.++struct kdbus_msg_info {+ __u64 offset;+ Upon return of the ioctl, this field contains the offset in the receiver's+ memory pool. The memory must be freed with KDBUS_CMD_FREE.++ __u64 msg_size;+ Upon successful return of the ioctl, this field contains the size of the+ allocated slice at offset @offset. It is the combination of the size of+ the stored kdbus_msg object plus all appended VECs. You can use it in+ combination with @offset to map a single message, instead of mapping the+ whole pool.++ __u64 return_flags;+ Kernel-provided return flags. Currently, the following flags are defined.++ KDBUS_RECV_RETURN_INCOMPLETE_FDS+ The message contained file descriptors which couldn't be installed+ into the receiver's task. Most probably that happened because the+ maximum number of file descriptors for that task were exceeded.+ The message is still delivered, so this is not a fatal condition.+ File descriptors inside the KDBUS_ITEM_FDS item that could not be+ installed will be set to -1.+};++Unless KDBUS_RECV_DROP was passed, and given that the ioctl succeeded, the+offset field contains the location of the new message inside the receiver's+pool. The message is stored as struct kdbus_msg at this offset, and can be+interpreted with the semantics described above.++Also, if the connection allowed for file descriptor to be passed+(KDBUS_HELLO_ACCEPT_FD), and if the message contained any, they will be+installed into the receiving process after the KDBUS_CMD_RECV ioctl returns.+The receiving task is obliged to close all of them appropriately. If+KDBUS_RECV_PEEK is set, no file descriptors are installed. This allows for+peeking at a message and dropping it via KDBUS_RECV_DROP, without installing+the passed file descriptors into the receiving process.++The caller is obliged to call KDBUS_CMD_FREE with the returned offset when+the memory is no longer needed.+++8. Name registry+===============================================================================++Each bus instantiates a name registry to resolve well-known names into unique+connection IDs for message delivery. The registry will be queried when a+message is sent with kdbus_msg.dst_id set to KDBUS_DST_ID_NAME, or when a+registry dump is requested.++All of the below is subject to policy rules for SEE and OWN permissions.+++8.1 Name validity+-----------------++A name has to comply to the following rules to be considered valid:++ - The name has two or more elements separated by a period ('.') character+ - All elements must contain at least one character+ - Each element must only contain the ASCII characters "[A-Z][a-z][0-9]_"+ and must not begin with a digit+ - The name must contain at least one '.' (period) character+ (and thus at least two elements)+ - The name must not begin with a '.' (period) character+ - The name must not exceed KDBUS_NAME_MAX_LEN (255)+++8.2 Acquiring a name+--------------------++To acquire a name, a client uses the KDBUS_CMD_NAME_ACQUIRE ioctl with the+following data structure.++struct kdbus_cmd_name {+ __u64 size;+ The overall size of this struct, including the name with its 0-byte string+ terminator.++ __u64 flags;+ Flags to control details in the name acquisition.++ KDBUS_NAME_REPLACE_EXISTING+ Acquiring a name that is already present usually fails, unless this flag+ is set in the call, and KDBUS_NAME_ALLOW_REPLACEMENT or (see below) was+ set when the current owner of the name acquired it, or if the current+ owner is an activator connection (see below).++ KDBUS_NAME_ALLOW_REPLACEMENT+ Allow other connections to take over this name. When this happens, the+ former owner of the connection will be notified of the name loss.++ KDBUS_NAME_QUEUE (acquire)+ A name that is already acquired by a connection, and which wasn't+ requested with the KDBUS_NAME_ALLOW_REPLACEMENT flag set can not be+ acquired again. However, a connection can put itself in a queue of+ connections waiting for the name to be released. Once that happens, the+ first connection in that queue becomes the new owner and is notified+ accordingly.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ struct kdbus_item items[0];+ Items to submit the name. Currently, one item of type KDBUS_ITEM_NAME is+ expected and allowed, and the contained string must be a valid bus name.+ Items of other types are rejected, and the ioctl will fail with -EINVAL.+};+++8.3 Releasing a name+--------------------++A connection may release a name explicitly with the KDBUS_CMD_NAME_RELEASE+ioctl. If the connection was an implementer of an activatable name, its+pending messages are moved back to the activator. If there are any connections+queued up as waiters for the name, the oldest one of them will become the new+owner. The same happens implicitly for all names once a connection terminates.++The KDBUS_CMD_NAME_RELEASE ioctl uses the same data structure as the+acquisition call, but with slightly different field usage.++struct kdbus_cmd_name {+ __u64 size;+ The overall size of this struct, including the name with its 0-byte string+ terminator.++ __u64 flags;+ Flags to the command. Currently unused.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ struct kdbus_item items[0];+ Items to submit the name. Currently, one item of type KDBUS_ITEM_NAME is+ expected and allowed, and the contained string must be a valid bus name.+};+++8.4 Dumping the name registry+-----------------------------++A connection may request a complete or filtered dump of currently active bus+names with the KDBUS_CMD_NAME_LIST ioctl, which takes a struct+kdbus_cmd_name_list as argument.++struct kdbus_cmd_name_list {+ __u64 flags;+ Any combination of flags to specify which names should be dumped.++ KDBUS_NAME_LIST_UNIQUE+ List the unique (numeric) IDs of the connection, whether it owns a name+ or not.++ KDBUS_NAME_LIST_NAMES+ List well-known names stored in the database which are actively owned by+ a real connection (not an activator).++ KDBUS_NAME_LIST_ACTIVATORS+ List names that are owned by an activator.++ KDBUS_NAME_LIST_QUEUED+ List connections that are not yet owning a name but are waiting for it+ to become available.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ __u64 offset;+ When the ioctl returns successfully, the offset to the name registry dump+ inside the connection's pool will be stored in this field.+};++The returned list of names is stored in a struct kdbus_name_list that in turn+contains a dynamic number of struct kdbus_cmd_name that carry the actual+information. The fields inside that struct kdbus_cmd_name is described next.++struct kdbus_name_info {+ __u64 size;+ The overall size of this struct, including the name with its 0-byte string+ terminator.++ __u64 owner_id;+ The owning connection's unique ID.++ __u64 conn_flags;+ The flags of the owning connection.++ struct kdbus_item items[0];+ Items containing the actual name. Currently, one item of type+ KDBUS_ITEM_OWNED_NAME will be attached, including the name's flags. In that+ item, the flags field of the name may carry the following bits:++ KDBUS_NAME_ALLOW_REPLACEMENT+ Other connections are allowed to take over this name from the+ connection that owns it.++ KDBUS_NAME_IN_QUEUE (list)+ When retrieving a list of currently acquired name in the registry, this+ flag indicates whether the connection actually owns the name or is+ currently waiting for it to become available.++ KDBUS_NAME_ACTIVATOR (list)+ An activator connection owns a name as a placeholder for an implementer,+ which is started on demand as soon as the first message arrives. There's+ some more information on this topic below. In contrast to+ KDBUS_NAME_REPLACE_EXISTING, when a name is taken over from an activator+ connection, all the messages that have been queued in the activator+ connection will be moved over to the new owner. The activator connection+ will still be tracked for the name and will take control again if the+ implementer connection terminates.+ This flag can not be used when acquiring a name, but is implicitly set+ through KDBUS_CMD_HELLO with KDBUS_HELLO_ACTIVATOR set in+ kdbus_cmd_hello.conn_flags.+};++The returned buffer must be freed with the KDBUS_CMD_FREE ioctl when the user+is finished with it.+++9. Notifications+===============================================================================++The kernel will notify its users of the following events.++ * When connection A is terminated while connection B is waiting for a reply+ from it, connection B is notified with a message with an item of type+ KDBUS_ITEM_REPLY_DEAD.++ * When connection A does not receive a reply from connection B within the+ specified timeout window, connection A will receive a message with an item+ of type KDBUS_ITEM_REPLY_TIMEOUT.++ * When an ordinary connection (not a monitor) is created on or removed from+ a bus, messages with an item of type KDBUS_ITEM_ID_ADD or+ KDBUS_ITEM_ID_REMOVE, respectively, are sent to all bus members that match+ these messages through their match database. Eavesdroppers (monitor+ connections) do not cause such notifications to be sent. They are invisible+ on the bus.++ * When a connection gains or loses ownership of a name, messages with an item+ of type KDBUS_ITEM_NAME_ADD, KDBUS_ITEM_NAME_REMOVE or+ KDBUS_ITEM_NAME_CHANGE are sent to all bus members that match these+ messages through their match database.++A kernel notification is a regular kdbus message with the following details.++ * kdbus_msg.src_id == KDBUS_SRC_ID_KERNEL+ * kdbus_msg.dst_id == KDBUS_DST_ID_BROADCAST+ * kdbus_msg.payload_type == KDBUS_PAYLOAD_KERNEL+ * Has exactly one of the aforementioned items attached++Kernel notifications have an item of type KDBUS_ITEM_TIMESTAMP attached.+++10. Message Matching, Bloom filters+===============================================================================++10.1 Matches for broadcast messages from other connections+----------------------------------------------------------++A message addressed at the connection ID KDBUS_DST_ID_BROADCAST (~0ULL) is a+broadcast message, delivered to all connected peers which installed a rule to+match certain properties of the message. Without any rules installed in the+connection, no broadcast message or kernel-side notifications will be delivered+to the connection. Broadcast messages are subject to policy rules and TALK+access checks.++See section 11 for details on policies, and section 11.5 for more+details on implicit policies.++Matches for messages from other connections (not kernel notifications) are+implemented as bloom filters. The sender adds certain properties of the message+as elements to a bloom filter bit field, and sends that along with the+broadcast message.++The connection adds the message properties it is interested as elements to a+bloom mask bit field, and uploads the mask to the match rules of the+connection.++The kernel will match the broadcast message's bloom filter against the+connections bloom mask (simply by &-ing it), and decide whether the message+should be delivered to the connection.++The kernel has no notion of any specific properties of the message, all it+sees are the bit fields of the bloom filter and mask to match against. The+use of bloom filters allows simple and efficient matching, without exposing+any message properties or internals to the kernel side. Clients need to deal+with the fact that they might receive broadcasts which they did not subscribe+to, as the bloom filter might allow false-positives to pass the filter.++To allow the future extension of the set of elements in the bloom filter, the+filter specifies a "generation" number. A later generation must always contain+all elements of the set of the previous generation, but can add new elements+to the set. The match rules mask can carry an array with all previous+generations of masks individually stored. When the filter and mask are matched+by the kernel, the mask with the closest matching "generation" is selected+as the index into the mask array.+++10.2 Matches for kernel notifications+------------------------------------++To receive kernel generated notifications (see section 9), a connection must+install special match rules that are different from the bloom filter matches+described in the section above. They can be filtered by a sender connection's+ID, by one of the name the sender connection owns at the time of sending the+message, or by type of the notification (id/name add/remove/change).++10.3 Adding a match+-------------------++To add a match, the KDBUS_CMD_MATCH_ADD ioctl is used, which takes a struct+of the struct described below.++Note that each of the items attached to this command will internally create+one match 'rule', and the collection of them, which is submitted as one block+via the ioctl is called a 'match'. To allow a message to pass, all rules of a+match have to be satisfied. Hence, adding more items to the command will only+narrow the possibility of a match to effectively let the message pass, and will+cause the connection's user space process to wake up less likely.++Multiple matches can be installed per connection. As long as one of it has a+set of rules which allows the message to pass, this one will be decisive.++struct kdbus_cmd_match {+ __u64 size;+ The overall size of the struct, including its items.++ __u64 cookie;+ A cookie which identifies the match, so it can be referred to at removal+ time.++ __u64 flags;+ Flags to control the behavior of the ioctl.++ KDBUS_MATCH_REPLACE:+ Remove all entries with the given cookie before installing the new one.+ This allows for race-free replacement of matches.++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ struct kdbus_item items[0];+ Items to define the actual rules of the matches. The following item types+ are expected. Each item will cause one new match rule to be created.++ KDBUS_ITEM_BLOOM_MASK+ An item that carries the bloom filter mask to match against in its+ data field. The payload size must match the bloom filter size that+ was specified when the bus was created.+ See section 10.4 for more information.++ KDBUS_ITEM_NAME+ Specify a name that a sending connection must own at a time of sending+ a broadcast message in order to match this rule.++ KDBUS_ITEM_ID+ Specify a sender connection's ID that will match this rule.++ KDBUS_ITEM_NAME_ADD+ KDBUS_ITEM_NAME_REMOVE+ KDBUS_ITEM_NAME_CHANGE+ These items request delivery of broadcast messages that describe a name+ acquisition, loss, or change. The details are stored in the item's+ kdbus_notify_name_change member. All information specified must be+ matched in order to make the message pass. Use KDBUS_MATCH_ID_ANY to+ match against any unique connection ID.++ KDBUS_ITEM_ID_ADD+ KDBUS_ITEM_ID_REMOVE+ These items request delivery of broadcast messages that are generated+ when a connection is created or terminated. struct kdbus_notify_id_change+ is used to store the actual match information. This item can be used to+ monitor one particular connection ID, or, when the id field is set to+ KDBUS_MATCH_ID_ANY, all of them.++ Items of other types are rejected, and the ioctl will fail with -EINVAL.+};+++10.4 Bloom filters+------------------++Bloom filters allow checking whether a given word is present in a dictionary.+This allows connections to set up a mask for information it is interested in,+and will be delivered signal messages that have a matching filter.++For general information on bloom filters, see++ https://en.wikipedia.org/wiki/Bloom_filter++The size of the bloom filter is defined per bus when it is created, in+kdbus_bloom_parameter.size. All bloom filters attached to signals on the bus+must match this size, and all bloom filter matches uploaded by connections must+also match the size, or a multiple thereof (see below).++The calculation of the mask has to be done on the userspace side. The kernel+just checks the bitmasks to decide whether or not to let the message pass. All+bits in the mask must match the filter in and bit-wise AND logic, but the+mask may have more bits set than the filter. Consequently, false positive+matches are expected to happen, and userspace must deal with that fact.++Masks are entities that are always passed to the kernel as part of a match+(with an item of type KDBUS_ITEM_BLOOM_MASK), and filters can be attached to+signals, with an item of type KDBUS_ITEM_BLOOM_FILTER.++For a filter to match, all its bits have to be set in the match mask as well.+For example, consider a bus has a bloom size of 8 bytes, and the following+mask/filter combinations:++ filter 0x0101010101010101+ mask 0x0101010101010101+ -> matches++ filter 0x0303030303030303+ mask 0x0101010101010101+ -> doesn't match++ filter 0x0101010101010101+ mask 0x0303030303030303+ -> matches++Hence, in order to catch all messages, a mask filled with 0xff bytes can be+installed as a wildcard match rule.++Uploaded matches may contain multiple masks, each of which in the size of the+bloom size defined by the bus. Each block of a mask is called a 'generation',+starting at index 0.++At match time, when a signal is about to be delivered, a bloom mask generation+is passed, which denotes which of the bloom masks the filter should be matched+against. This allows userspace to provide backward compatible masks at upload+time, while older clients can still match against older versions of filters.+++10.5 Removing a match+--------------------++Matches can be removed through the KDBUS_CMD_MATCH_REMOVE ioctl, which again+takes struct kdbus_cmd_match as argument, but its fields are used slightly+differently.++struct kdbus_cmd_match {+ __u64 size;+ The overall size of the struct. As it has no items in this use case, the+ value should yield 16.++ __u64 cookie;+ The cookie of the match, as it was passed when the match was added.+ All matches that have this cookie will be removed.++ __u64 flags;+ Unused for this use case,++ __u64 kernel_flags;+ Valid flags for this command, returned by the kernel upon each call.++ __u64 return_flags;+ Flags returned by the kernel. Currently unused.++ struct kdbus_item items[0];+ Unused und not allowed for this use case.+};+++11. Policy+===============================================================================++A policy databases restrict the possibilities of connections to own, see and+talk to well-known names. It can be associated with a bus (through a policy+holder connection) or a custom endpoint.++See section 8.1 for more details on the validity of well-known names.++Default endpoints of buses always have a policy database. The default+policy is to deny all operations except for operations that are covered by+implicit policies. Custom endpoints always have a policy, and by default,+a policy database is empty. Therefore, unless policy rules are added, all+operations will also be denied by default.++See section 11.5 for more details on implicit policies.++A set of policy rules is described by a name and multiple access rules, defined+by the following struct.++struct kdbus_policy_access {+ __u64 type; /* USER, GROUP, WORLD */+ One of the following.++ KDBUS_POLICY_ACCESS_USER+ Grant access to a user with the uid stored in the 'id' field.++ KDBUS_POLICY_ACCESS_GROUP+ Grant access to a user with the gid stored in the 'id' field.++ KDBUS_POLICY_ACCESS_WORLD+ Grant access to everyone. The 'id' field is ignored.++ __u64 access; /* OWN, TALK, SEE */+ The access to grant.++ KDBUS_POLICY_SEE+ Allow the name to be seen.++ KDBUS_POLICY_TALK+ Allow the name to be talked to.++ KDBUS_POLICY_OWN+ Allow the name to be owned.++ __u64 id;+ For KDBUS_POLICY_ACCESS_USER, stores the uid.+ For KDBUS_POLICY_ACCESS_GROUP, stores the gid.+};++Policies are set through KDBUS_CMD_HELLO (when creating a policy holder+connection), KDBUS_CMD_CONN_UPDATE (when updating a policy holder connection),+KDBUS_CMD_ENDPOINT_MAKE (creating a custom endpoint) or+KDBUS_CMD_ENDPOINT_UPDATE (updating a custom endpoint). In all cases, the name+and policy access information is stored in items of type KDBUS_ITEM_NAME and+KDBUS_ITEM_POLICY_ACCESS. For this transport, the following rules apply.++ * An item of type KDBUS_ITEM_NAME must be followed by at least one+ KDBUS_ITEM_POLICY_ACCESS item+ * An item of type KDBUS_ITEM_NAME can be followed by an arbitrary number of+ KDBUS_ITEM_POLICY_ACCESS items+ * An arbitrary number of groups of names and access levels can be passed++uids and gids are internally always stored in the kernel's view of global ids,+and are translated back and forth on the ioctl level accordingly.+++11.2 Wildcard names+-------------------++Policy holder connections may upload names that contain the wild card suffix+(".*"). That way, a policy can be uploaded that is effective for every+well-known name that extends the provided name by exactly one more level.++For example, if an item of a set up uploaded policy rules contains the name+"foo.bar.*", both "foo.bar.baz" and "foo.bar.bazbaz" are valid, but+"foo.bar.baz.baz" is not.++This allows connections to take control over multiple names that the policy+holder doesn't need to know about when uploading the policy.++Such wildcard entries are not allowed for custom endpoints.+++11.3 Policy example+-------------------++For example, a set of policy rules may look like this:++ KDBUS_ITEM_NAME: str='org.foo.bar'+ KDBUS_ITEM_POLICY_ACCESS: type=USER, access=OWN, id=1000+ KDBUS_ITEM_POLICY_ACCESS: type=USER, access=TALK, id=1001+ KDBUS_ITEM_POLICY_ACCESS: type=WORLD, access=SEE+ KDBUS_ITEM_NAME: str='org.blah.baz'+ KDBUS_ITEM_POLICY_ACCESS: type=USER, access=OWN, id=0+ KDBUS_ITEM_POLICY_ACCESS: type=WORLD, access=TALK++That means that 'org.foo.bar' may only be owned by uid 1000, but every user on+the bus is allowed to see the name. However, only uid 1001 may actually send+a message to the connection and receive a reply from it.++The second rule allows 'org.blah.baz' to be owned by uid 0 only, but every user+may talk to it.+++11.4 TALK access and multiple well-known names per connection+-------------------------------------------------------------++Note that TALK access is checked against all names of a connection.+For example, if a connection owns both 'org.foo.bar' and 'org.blah.baz', and+the policy database allows 'org.blah.baz' to be talked to by WORLD, then this+permission is also granted to 'org.foo.bar'. That might sound illogical, but+after all, we allow messages to be directed to either the ID or a well-known+name, and policy is applied to the connection, not the name. In other words,+the effective TALK policy for a connection is the most permissive of all names+the connection owns.++For broadcast messages, the receiver needs TALK permissions to the sender to+receive the broadcast.++If a policy database exists for a bus (because a policy holder created one on+demand) or for a custom endpoint (which always has one), each one is consulted+during name registry listing, name owning or message delivery. If either one+fails, the operation is failed with -EPERM.++For best practices, connections that own names with a restricted TALK+access should not install matches. This avoids cases where the sent+message may pass the bloom filter due to false-positives and may also+satisfy the policy rules.+++11.5 Implicit policies+----------------------++Depending on the type of the endpoint, a set of implicit rules that+override installed policies might be enforced.++On default endpoints, the following set is enforced and checked before+any user-supplied policy is checked.++ * Privileged connections always override any installed policy. Those+ connections could easily install their own policies, so there is no+ reason to enforce installed policies.+ * Connections can always talk to connections of the same user. This+ includes broadcast messages.++Custom endpoints have stricter policies. The following rules apply:++ * Policy rules are always enforced, even if the connection is a privileged+ connection.+ * Policy rules are always enforced for TALK access, even if both ends are+ running under the same user. This includes broadcast messages.+ * To restrict the set of names that can be seen, endpoint policies can+ install "SEE" policies.+++12. Pool+===============================================================================++A pool for data received from the kernel is installed for every connection of+the bus, and is sized according to the information stored in the+KDBUS_ITEM_BLOOM_PARAMETER item that is returned by KDBUS_CMD_HELLO.++The pool is written to by the kernel when one of the following ioctls is issued:++ * KDBUS_CMD_HELLO, to receive details about the bus the connection was made to+ * KDBUS_CMD_RECV, to receive a message+ * KDBUS_CMD_NAME_LIST, to dump the name registry+ * KDBUS_CMD_CONN_INFO, to retrieve information on a connection++The offsets returned by either one of the aforementioned ioctls describe offsets+inside the pool. In order to make the slice available for subsequent calls,+KDBUS_CMD_FREE has to be called on the offset.++To access the memory, the caller is expected to mmap() it to its task, like+this:++ /*+ * POOL_SIZE has to be a multiple of PAGE_SIZE, and it must match the+ * value that was previously returned through the KDBUS_ITEM_BLOOM_PARAMETER+ * item field when the KDBUS_CMD_HELLO ioctl returned.+ */++ buf = mmap(NULL, POOL_SIZE, PROT_READ, MAP_SHARED, conn_fd, 0);++Alternatively, instead of mapping the entire pool buffer, only parts of it can+be mapped. The length of the response is returned by the kernel along with the+offset for each of the ioctls listed above.+++13. Metadata+===============================================================================++kdbus records data about the system in certain situations. Such metadata can+refer to the currently active process (creds, PIDs, current user groups, process+names and its executable path, cgroup membership, capabilities, security label+and audit information), connection information (description string, currently+owned names) and the timestamp.++Metadata is collected in the following circumstances:++ * When a bus is created (KDBUS_CMD_MAKE), information about the calling task+ is collected. This data is returned by the kernel via the+ KDBUS_CMD_BUS_CREATOR_INFO call-++ * When a connection is created (KDBUS_CMD_HELLO), information about the+ calling task is collected. Alternatively, a privileged connection may+ provide 'faked' information about credentials, PIDs and a security labels+ which will be taken instead. This data is returned by the kernel as+ information on a connection (KDBUS_CMD_CONN_INFO).++ * When a message is sent (KDBUS_CMD_SEND), information about the sending task+ and the sending connection are collected. This metadata will be attached+ to the message when it arrives in the receiver's pool. If the connection+ sending the message installed faked credentials (see above), the message+ will not be augmented by any information about the currently sending task.++Which metadata items are actually delivered depends on the following sets and+masks:++ (a) the system-wide kmod creds mask (module parameter 'attach_flags_mask')+ (b) the per-connection send creds mask, set by the connecting client+ (c) the per-connection receive creds mask, set by the connecting client+ (d) the per-bus minimal creds mask, set by the bus creator+ (e) the per-bus owner creds mask, set by the bus creator+ (f) the mask specified when querying creds of a bus peer+ (g) the mask specified when querying creds of a bus owner++With the following rules:++ [1] The creds attached to messages are determined as (a & b & c).+ [2] When connecting to a bus (KDBUS_CMD_HELLO), and (~b & d) != 0, the call+ will fail, the connection is refused.+ [3] When querying creds of a bus peer, the creds returned are (a & b & f)+ [4] When querying creds of a bus owner, the creds returned are (a & e & g)+ [5] When creating a new bus, and (d & ~a) != 0, then bus creation will fail++Hence, userspace might not always get all requested metadata items that it+requested. Code must be written so that it can cope with this fact.+++13.1 Known item types+---------------------++The following attach flags are currently supported.++ KDBUS_ATTACH_TIMESTAMP+ Attaches an item of type KDBUS_ITEM_TIMESTAMP which contains both the+ monotonic and the realtime timestamp, taken when the message was+ processed on the kernel side.++ KDBUS_ATTACH_CREDS+ Attaches an item of type KDBUS_ITEM_CREDS, containing credentials as+ described in struct kdbus_creds: the user and group IDs in the usual four+ flavors: real, effective, saved and file-system related.++ KDBUS_ATTACH_PIDS+ Attaches an item of type KDBUS_ITEM_PIDS, containing information on the+ process. In particular, the PID (process ID), TID (thread ID), and PPID+ (PID of the parent process).++ KDBUS_ATTACH_AUXGROUPS+ Attaches an item of type KDBUS_ITEM_AUXGROUPS, containing a dynamic+ number of auxiliary groups the sending task was a member of.++ KDBUS_ATTACH_NAMES+ Attaches items of type KDBUS_ITEM_OWNED_NAME, one for each name the sending+ connection currently owns. The name and flags are stored in kdbus_item.name+ for each of them.++ KDBUS_ATTACH_TID_COMM [*]+ Attaches an items of type KDBUS_ITEM_TID_COMM, transporting the sending+ task's 'comm', for the tid. The string is stored in kdbus_item.str.++ KDBUS_ATTACH_PID_COMM [*]+ Attaches an items of type KDBUS_ITEM_PID_COMM, transporting the sending+ task's 'comm', for the pid. The string is stored in kdbus_item.str.++ KDBUS_ATTACH_EXE [*]+ Attaches an item of type KDBUS_ITEM_EXE, containing the path to the+ executable of the sending task, stored in kdbus_item.str.++ KDBUS_ATTACH_CMDLINE [*]+ Attaches an item of type KDBUS_ITEM_CMDLINE, containing the command line+ arguments of the sending task, as an array of strings, stored in+ kdbus_item.str.++ KDBUS_ATTACH_CGROUP+ Attaches an item of type KDBUS_ITEM_CGROUP with the task's cgroup path.++ KDBUS_ATTACH_CAPS+ Attaches an item of type KDBUS_ITEM_CAPS, carrying sets of capabilities+ that should be accessed via kdbus_item.caps.caps. Also, userspace should+ be written in a way that it takes kdbus_item.caps.last_cap into account,+ and derive the number of sets and rows from the item size and the reported+ number of valid capability bits.++ KDBUS_ATTACH_SECLABEL+ Attaches an item of type KDBUS_ITEM_SECLABEL, which contains the SELinux+ security label of the sending task. SELinux and other MACs might want to+ do additional per-service security checks. For example, a service manager+ might want to check the security label of a service file against the+ security label of the client process checking the SELinux database before+ allowing access. The label can be accessed via kdbus_item->str.++ KDBUS_ATTACH_AUDIT+ Attaches an item of type KDBUS_ITEM_AUDIT, which contains the audit+ sessionid and loginuid of the sending task. Access via kdbus_item->audit.++ KDBUS_ATTACH_CONN_DESCRIPTION+ Attaches an item of type KDBUS_ITEM_CONN_DESCRIPTION that contains a+ descriptive string of the sending peer. That string can be supplied+ during HELLO by attaching an item of type KDBUS_ITEM_CONN_DESCRIPTION.+++[*] Note that the content stored in these items can easily be tampered by+ the sending tasks. Therefore, they should NOT be used for any sort of+ security relevant assumptions. The only reason why they are transmitted is+ to let receivers know about details that were set when metadata was+ collected, even though the task they were collected from is not active any+ longer when the items are received.+++13.2 Metadata and namespaces+----------------------------++Metadata such as PIDs, UIDs or GIDs are automatically translated to the+namespaces of the task that receives them.+++14. Error codes+===============================================================================++Below is a list of error codes that might be returned by the individual+ioctl commands. The list focuses on the return values from kdbus code itself,+and might not cover those of all kernel internal functions.++For all ioctls:++ -ENOMEM The kernel memory is exhausted+ -ENOTTY Illegal ioctl command issued for the file descriptor+ -ENOSYS The requested functionality is not available+ -EINVAL Unsupported item attached to command++For all ioctls that carry a struct as payload:++ -EFAULT The supplied data pointer was not 64-bit aligned, or was+ inaccessible from the kernel side.+ -EINVAL The size inside the supplied struct was smaller than expected+ -EMSGSIZE The size inside the supplied struct was bigger than expected+ -ENAMETOOLONG A supplied name is larger than the allowed maximum size++For KDBUS_CMD_BUS_MAKE:++ -EINVAL The flags supplied in the kdbus_cmd_make struct are invalid or+ the supplied name does not start with the current uid and a '-'+ -EEXIST A bus of that name already exists+ -ESHUTDOWN The domain for the bus is already shut down+ -EMFILE The maximum number of buses for the current user is exhausted++For KDBUS_CMD_ENDPOINT_MAKE:++ -EPERM The calling user is not privileged (see Terminology)+ -EINVAL The flags supplied in the kdbus_cmd_make struct are invalid+ -EEXIST An endpoint of that name already exists++For KDBUS_CMD_HELLO:++ -EFAULT The supplied pool size was 0 or not a multiple of the page size+ -EINVAL The flags supplied in the kdbus_cmd_make struct are invalid, or+ an illegal combination of KDBUS_HELLO_MONITOR,+ KDBUS_HELLO_ACTIVATOR and KDBUS_HELLO_POLICY_HOLDER was passed+ in the flags, or an invalid set of items was supplied+ -ECONNREFUSED The attach_flags_send field did not satisfy the requirements of+ the bus+ -EPERM An KDBUS_ITEM_CREDS items was supplied, but the current user is+ not privileged+ -ESHUTDOWN The bus has already been shut down+ -EMFILE The maximum number of connection on the bus has been reached+ -EOPNOTSUPP The endpoint does not support the connection flags+ supplied in the kdbus_cmd_hello struct++For KDBUS_CMD_BYEBYE:++ -EALREADY The connection has already been shut down+ -EBUSY There are still messages queued up in the connection's pool++For KDBUS_CMD_SEND:++ -EOPNOTSUPP The connection is not an ordinary connection, or the passed+ file descriptors are either kdbus handles or unix domain+ sockets. Both are currently unsupported+ -EINVAL The submitted payload type is KDBUS_PAYLOAD_KERNEL,+ KDBUS_MSG_EXPECT_REPLY was set without timeout or cookie+ values, KDBUS_MSG_SYNC_REPLY was set without+ KDBUS_MSG_EXPECT_REPLY, an invalid item was supplied,+ src_id was != 0 and different from the current connection's ID,+ a supplied memfd had a size of 0, a string was not properly+ null-terminated+ -ENOTUNIQ The supplied destination is KDBUS_DST_ID_BROADCAST, a file+ descriptor was passed, KDBUS_MSG_EXPECT_REPLY was set,+ or a timeout was given for a broadcast message+ -E2BIG Too many items+ -EMSGSIZE The size of the message header and items or the payload vector+ is too big.+ -EEXIST Multiple KDBUS_ITEM_FDS, KDBUS_ITEM_BLOOM_FILTER or+ KDBUS_ITEM_DST_NAME items were supplied+ -EBADF The supplied KDBUS_ITEM_FDS or KDBUS_MSG_PAYLOAD_MEMFD items+ contained an illegal file descriptor+ -EMEDIUMTYPE The supplied memfd is not a sealed kdbus memfd+ -EMFILE Too many file descriptors inside a KDBUS_ITEM_FDS+ -EBADMSG An item had illegal size, both a dst_id and a+ KDBUS_ITEM_DST_NAME was given, or both a name and a bloom+ filter was given+ -ETXTBSY The supplied kdbus memfd file cannot be sealed or the seal+ was removed, because it is shared with other processes or+ still mmap()ed+ -ECOMM A peer does not accept the file descriptors addressed to it+ -EFAULT The supplied bloom filter size was not 64-bit aligned+ -EDOM The supplied bloom filter size did not match the bloom filter+ size of the bus+ -EDESTADDRREQ dst_id was set to KDBUS_DST_ID_NAME, but no KDBUS_ITEM_DST_NAME+ was attached+ -ESRCH The name to look up was not found in the name registry+ -EADDRNOTAVAIL KDBUS_MSG_NO_AUTO_START was given but the destination+ connection is an activator.+ -ENXIO The passed numeric destination connection ID couldn't be found,+ or is not connected+ -ECONNRESET The destination connection is no longer active+ -ETIMEDOUT Timeout while synchronously waiting for a reply+ -EINTR System call interrupted while synchronously waiting for a reply+ -EPIPE When sending a message, a synchronous reply from the receiving+ connection was expected but the connection died before+ answering+ -ENOBUFS Too many pending messages on the receiver side+ -EREMCHG Both a well-known name and a unique name (ID) was given, but+ the name is not currently owned by that connection.+ -EXFULL The memory pool of the receiver is full+ -EREMOTEIO While synchronously waiting for a reply, the remote peer+ failed with an I/O error.++For KDBUS_CMD_RECV:++ -EINVAL Invalid flags or offset+ -EAGAIN No message found in the queue+ -ENOMSG No message of the requested priority found+ -EOVERFLOW Broadcast messages have been lost++For KDBUS_CMD_FREE:++ -ENXIO No pool slice found at given offset+ -EINVAL Invalid flags provided, the offset is valid, but the user is+ not allowed to free the slice. This happens, for example, if+ the offset was retrieved with KDBUS_RECV_PEEK.++For KDBUS_CMD_NAME_ACQUIRE:++ -EINVAL Illegal command flags, illegal name provided, or an activator+ tried to acquire a second name+ -EPERM Policy prohibited name ownership+ -EALREADY Connection already owns that name+ -EEXIST The name already exists and can not be taken over+ -E2BIG The maximum number of well-known names per connection+ is exhausted+ -ECONNRESET The connection was reset during the call++For KDBUS_CMD_NAME_RELEASE:++ -EINVAL Invalid command flags, or invalid name provided+ -ESRCH Name is not found found in the registry+ -EADDRINUSE Name is owned by a different connection and can't be released++For KDBUS_CMD_NAME_LIST:++ -EINVAL Invalid flags+ -ENOBUFS No available memory in the connection's pool.++For KDBUS_CMD_CONN_INFO:++ -EINVAL Invalid flags, or neither an ID nor a name was provided,+ or the name is invalid.+ -ESRCH Connection lookup by name failed+ -ENXIO No connection with the provided connection ID found++For KDBUS_CMD_CONN_UPDATE:++ -EINVAL Illegal flags or items+ -EOPNOTSUPP Operation not supported by connection.+ -E2BIG Too many policy items attached+ -EINVAL Wildcards submitted in policy entries, or illegal sequence+ of policy items++For KDBUS_CMD_ENDPOINT_UPDATE:++ -E2BIG Too many policy items attached+ -EINVAL Invalid flags, or wildcards submitted in policy entries,+ or illegal sequence of policy items++For KDBUS_CMD_MATCH_ADD:++ -EINVAL Illegal flags or items+ -EDOM Illegal bloom filter size+ -EMFILE Too many matches for this connection++For KDBUS_CMD_MATCH_REMOVE:++ -EINVAL Illegal flags+ -ENOENT A match entry with the given cookie could not be found.+++15. Internal object relations+===============================================================================++This is a simplified outline of the internal kdbus object relations, for+those interested in the inner life of the driver implementation.++From the a mount point's (domain's) perspective:++struct kdbus_domain+ |» struct kdbus_domain_user *user (many, owned)+ '» struct kdbus_node node (embedded)+ |» struct kdbus_node children (many, referenced)+ |» struct kdbus_node *parent (pinned)+ '» struct kdbus_bus (many, pinned)+ |» struct kdbus_node node (embedded)+ '» struct kdbus_ep (many, pinned)+ |» struct kdbus_node node (embedded)+ |» struct kdbus_bus *bus (pinned)+ |» struct kdbus_conn conn_list (many, pinned)+ | |» struct kdbus_ep *ep (pinned)+ | |» struct kdbus_name_entry *activator_of (owned)+ | |» struct kdbus_match_db *match_db (owned)+ | |» struct kdbus_meta *meta (owned)+ | |» struct kdbus_match_db *match_db (owned)+ | | '» struct kdbus_match_entry (many, owned)+ | |+ | |» struct kdbus_pool *pool (owned)+ | | '» struct kdbus_pool_slice *slices (many, owned)+ | | '» struct kdbus_pool *pool (pinned)+ | |+ | |» struct kdbus_domain_user *user (pinned)+ | `» struct kdbus_queue_entry entries (many, embedded)+ | |» struct kdbus_pool_slice *slice (pinned)+ | |» struct kdbus_conn_reply *reply (owned)+ | '» struct kdbus_domain_user *user (pinned)+ |+ '» struct kdbus_domain_user *user (pinned)+ '» struct kdbus_policy_db policy_db (embedded)+ |» struct kdbus_policy_db_entry (many, owned)+ | |» struct kdbus_conn (pinned)+ | '» struct kdbus_ep (pinned)+ |+ '» struct kdbus_policy_db_cache_entry (many, owned)+ '» struct kdbus_conn (pinned)+++For the life-time of a file descriptor derived from calling open() on a file+inside the mount point:++struct kdbus_handle+ |» struct kdbus_meta *meta (owned)+ |» struct kdbus_ep *ep (pinned)+ |» struct kdbus_conn *conn (owned)+ '» struct kdbus_ep *ep (owned)
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-20 13:53:18
On 01/16/2015 08:16 PM, Greg Kroah-Hartman wrote:
From: Daniel Mack <redacted>
kdbus is a system for low-latency, low-overhead, easy to use
interprocess communication (IPC).
The interface to all functions in this driver is implemented via ioctls
on files exposed through a filesystem called 'kdbusfs'. The default
mount point of kdbusfs is /sys/fs/kdbus. This patch adds detailed
documentation about the kernel level API design.
I have some details feedback on the contents of this file, and some
bigger questions. I'll split them out into separate mails.
So here, the bigger, general questions to start with. I've arrived late
to this, so sorry if they've already been discussed, but the answers to
some of the questions should actually be in this file, I would have
expected.
This is an enormous and complex API. Why is the API ioctl() based,
rather than system-call-based? Have we learned nothing from the hydra
that the futex() multiplexing syscall became? (And kdbus is an order
of magnitude more complex, by the look of things.) At the very least,
a *good* justification of why the API is ioctl()-based should be part
of this documentation file.
An observation: The documentation below is substantial, but this API is
enormous, so the documentation still feels rather thin. What would
really help would be some example code in the doc.
And on the subject of code examples... Are there any (prototype)
working user-space applications that exercise the current kdbus
implementation? That is, can I install these kdbus patches, and
then find a simple example application somewhere that does
something to exercise kdbus?
And then: is there any substantial real-world application (e.g., a
full D-Bus port) that is being developed in tandem with this kernel
side patch? (I don't mean a user-space library; I mean a seriously
large application.) This is an incredibly complex API whose
failings are only going to become evident through real-world use.
Solidifying an API in the kernel and then discovering the API
problems later when writing real-world applications would make for
a sad story. A story something like that of inotify, an API which
is an order of magnitude less complex than kdbus. (I can't help but
feel that many of inotify problems that I discuss at
https://lwn.net/Articles/605128/ might have been fixed or mitigated
if a few real-world applications had been implemented before the
API was set in stone.)
Is this library intended just for systemd? More generally, is there an
intention to provide a general purpose library API for kdbus? Or is the
intention that each application will roll a library suitable to its
needs? I think an answer to that question would be useful in this
Documentation file.
Cheers,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
From: David Herrmann <hidden> Date: 2015-01-20 14:31:55
Hi Michael
On Tue, Jan 20, 2015 at 2:53 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
On 01/16/2015 08:16 PM, Greg Kroah-Hartman wrote:
quoted
From: Daniel Mack <redacted>
kdbus is a system for low-latency, low-overhead, easy to use
interprocess communication (IPC).
The interface to all functions in this driver is implemented via ioctls
on files exposed through a filesystem called 'kdbusfs'. The default
mount point of kdbusfs is /sys/fs/kdbus. This patch adds detailed
documentation about the kernel level API design.
I have some details feedback on the contents of this file, and some
bigger questions. I'll split them out into separate mails.
So here, the bigger, general questions to start with. I've arrived late
to this, so sorry if they've already been discussed, but the answers to
some of the questions should actually be in this file, I would have
expected.
This is an enormous and complex API. Why is the API ioctl() based,
rather than system-call-based? Have we learned nothing from the hydra
that the futex() multiplexing syscall became? (And kdbus is an order
of magnitude more complex, by the look of things.) At the very least,
a *good* justification of why the API is ioctl()-based should be part
of this documentation file.
An observation: The documentation below is substantial, but this API is
enormous, so the documentation still feels rather thin. What would
really help would be some example code in the doc.
And on the subject of code examples... Are there any (prototype)
working user-space applications that exercise the current kdbus
implementation? That is, can I install these kdbus patches, and
then find a simple example application somewhere that does
something to exercise kdbus?
If you run a 3.18 kernel, you can install kdbus.ko from our repository
and boot a full Fedora system running Gnome3 with kdbus, given that
you compiled systemd with --enable-kdbus (which is still
experimental). No legacy dbus1 daemon is running. Instead, we have a
bus-proxy that converts classic dbus1 to kdbus, so all
bus-communication runs on kdbus.
And then: is there any substantial real-world application (e.g., a
full D-Bus port) that is being developed in tandem with this kernel
side patch? (I don't mean a user-space library; I mean a seriously
large application.) This is an incredibly complex API whose
failings are only going to become evident through real-world use.
Solidifying an API in the kernel and then discovering the API
problems later when writing real-world applications would make for
a sad story. A story something like that of inotify, an API which
is an order of magnitude less complex than kdbus. (I can't help but
feel that many of inotify problems that I discuss at
https://lwn.net/Articles/605128/ might have been fixed or mitigated
if a few real-world applications had been implemented before the
API was set in stone.)
I think running a whole Gnome3 stack counts as "substantial real-world
application", right? Sure, it's a dbus1-to-kdbus layer, but all the
systemd tools use kdbus natively and it works just fine. In fact, we
all run kdbus on our main-systems every day.
We've spent over a year fixing races and API misdesigns, we've talked
to other toolkit developers (glib, qt, ..) and made sure we're
backwards compatible to dbus1. I don't think the API is perfect,
everyone makes mistakes. But with bus-proxy and systemd we have two
huge users of kdbus that put a lot of pressure on API design.
Is this library intended just for systemd? More generally, is there an
intention to provide a general purpose library API for kdbus? Or is the
intention that each application will roll a library suitable to its
needs? I think an answer to that question would be useful in this
Documentation file.
kdbus is in no way bound to systemd. There are ongoing efforts to port
glib and qt to kdbus natively. The API is pretty simple and I don't
see how a libkdbus would simplify things. In fact, even our tests only
have slim wrappers around the ioctls to simplify error-handling in
test-scenarios.
Note that most of the toolkit work is on the marshaling level, which
is independent of kdbus. kdbus just provides the transport level. DBus
is just one, yet significant, application-layer on top of kdbus. Our
test-cases use kdbus exclusively to transport raw byte streams.
Thanks
David
On Tue, Jan 20, 2015 at 9:31 AM, David Herrmann [off-list ref] wrote:
Hi Michael
On Tue, Jan 20, 2015 at 2:53 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
On 01/16/2015 08:16 PM, Greg Kroah-Hartman wrote:
quoted
From: Daniel Mack <redacted>
kdbus is a system for low-latency, low-overhead, easy to use
interprocess communication (IPC).
The interface to all functions in this driver is implemented via ioctls
on files exposed through a filesystem called 'kdbusfs'. The default
mount point of kdbusfs is /sys/fs/kdbus. This patch adds detailed
documentation about the kernel level API design.
I have some details feedback on the contents of this file, and some
bigger questions. I'll split them out into separate mails.
So here, the bigger, general questions to start with. I've arrived late
to this, so sorry if they've already been discussed, but the answers to
some of the questions should actually be in this file, I would have
expected.
This is an enormous and complex API. Why is the API ioctl() based,
rather than system-call-based? Have we learned nothing from the hydra
that the futex() multiplexing syscall became? (And kdbus is an order
of magnitude more complex, by the look of things.) At the very least,
a *good* justification of why the API is ioctl()-based should be part
of this documentation file.
An observation: The documentation below is substantial, but this API is
enormous, so the documentation still feels rather thin. What would
really help would be some example code in the doc.
And on the subject of code examples... Are there any (prototype)
working user-space applications that exercise the current kdbus
implementation? That is, can I install these kdbus patches, and
then find a simple example application somewhere that does
something to exercise kdbus?
If you run a 3.18 kernel, you can install kdbus.ko from our repository
and boot a full Fedora system running Gnome3 with kdbus, given that
you compiled systemd with --enable-kdbus (which is still
experimental). No legacy dbus1 daemon is running. Instead, we have a
bus-proxy that converts classic dbus1 to kdbus, so all
bus-communication runs on kdbus.
FWIW, we've been building a "playground" repository for the kernel
that contains this already for Fedora. If you have a stock Fedora 21
or rawhide install, you can use:
https://copr.fedoraproject.org/coprs/jwboyer/kernel-playground/
which has the kernel+kdbus and systemd built with --enable-kdbus
already. Easy enough to throw in a VM for testing.
josh
Hi,
On Tue, Jan 20, 2015 at 09:42:59AM -0500, Josh Boyer wrote:
On Tue, Jan 20, 2015 at 9:31 AM, David Herrmann [off-list ref] wrote:
quoted
Hi Michael
On Tue, Jan 20, 2015 at 2:53 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
On 01/16/2015 08:16 PM, Greg Kroah-Hartman wrote:
quoted
From: Daniel Mack <redacted>
kdbus is a system for low-latency, low-overhead, easy to use
interprocess communication (IPC).
The interface to all functions in this driver is implemented via ioctls
on files exposed through a filesystem called 'kdbusfs'. The default
mount point of kdbusfs is /sys/fs/kdbus. This patch adds detailed
documentation about the kernel level API design.
I have some details feedback on the contents of this file, and some
bigger questions. I'll split them out into separate mails.
So here, the bigger, general questions to start with. I've arrived late
to this, so sorry if they've already been discussed, but the answers to
some of the questions should actually be in this file, I would have
expected.
This is an enormous and complex API. Why is the API ioctl() based,
rather than system-call-based? Have we learned nothing from the hydra
that the futex() multiplexing syscall became? (And kdbus is an order
of magnitude more complex, by the look of things.) At the very least,
a *good* justification of why the API is ioctl()-based should be part
of this documentation file.
An observation: The documentation below is substantial, but this API is
enormous, so the documentation still feels rather thin. What would
really help would be some example code in the doc.
And on the subject of code examples... Are there any (prototype)
working user-space applications that exercise the current kdbus
implementation? That is, can I install these kdbus patches, and
then find a simple example application somewhere that does
something to exercise kdbus?
If you run a 3.18 kernel, you can install kdbus.ko from our repository
and boot a full Fedora system running Gnome3 with kdbus, given that
you compiled systemd with --enable-kdbus (which is still
experimental). No legacy dbus1 daemon is running. Instead, we have a
bus-proxy that converts classic dbus1 to kdbus, so all
bus-communication runs on kdbus.
FWIW, we've been building a "playground" repository for the kernel
that contains this already for Fedora. If you have a stock Fedora 21
or rawhide install, you can use:
https://copr.fedoraproject.org/coprs/jwboyer/kernel-playground/
which has the kernel+kdbus and systemd built with --enable-kdbus
already. Easy enough to throw in a VM for testing.
josh
Yes thanks josh!
Another addition, if kdbus is installed and loaded, you could also use
systemd-nspawn to boot a full system (systemd compiled with
--enable-kdbus) in a container [1], kdbusfs will be mounted in the
container.
There is also the busctl tool to query kdbus...
http://www.freedesktop.org/wiki/Software/systemd/VirtualizedTesting/
--
Djalal Harouni
http://opendz.org
From: Johannes Stezenbach <hidden> Date: 2015-01-20 16:09:45
Hi all,
On Tue, Jan 20, 2015 at 03:53:53PM +0100, Djalal Harouni wrote:
On Tue, Jan 20, 2015 at 09:42:59AM -0500, Josh Boyer wrote:
quoted
On Tue, Jan 20, 2015 at 9:31 AM, David Herrmann [off-list ref] wrote:
quoted
If you run a 3.18 kernel, you can install kdbus.ko from our repository
and boot a full Fedora system running Gnome3 with kdbus, given that
you compiled systemd with --enable-kdbus (which is still
experimental). No legacy dbus1 daemon is running. Instead, we have a
bus-proxy that converts classic dbus1 to kdbus, so all
bus-communication runs on kdbus.
FWIW, we've been building a "playground" repository for the kernel
that contains this already for Fedora. If you have a stock Fedora 21
or rawhide install, you can use:
https://copr.fedoraproject.org/coprs/jwboyer/kernel-playground/
which has the kernel+kdbus and systemd built with --enable-kdbus
already. Easy enough to throw in a VM for testing.
Another addition, if kdbus is installed and loaded, you could also use
systemd-nspawn to boot a full system (systemd compiled with
--enable-kdbus) in a container [1], kdbusfs will be mounted in the
container.
There is also the busctl tool to query kdbus...
http://www.freedesktop.org/wiki/Software/systemd/VirtualizedTesting/
It is reassuring that kdbus actually works :)
However, let me repeat and rephrase my previous questions:
Is there a noticable or measurable improvement from using kdbus?
IOW, is the added complexity of kdbus worth the result?
I have stated my believe that current usage of D-Bus is not
performance sensitive and the number of messages exchanged
is low. I would love it if you would prove me wrong.
Or if you could show that any D-Bus related bug in Gnome3
is fixed by kdbus.
I would sooo love it if someone would finally post some
data that proves kdbus is useful beyond systemd.
Thanks,
Johannes
From: David Herrmann <hidden> Date: 2015-01-20 17:00:38
Hi
On Tue, Jan 20, 2015 at 5:08 PM, Johannes Stezenbach [off-list ref] wrote:
However, let me repeat and rephrase my previous questions:
Is there a noticable or measurable improvement from using kdbus?
IOW, is the added complexity of kdbus worth the result?
I have stated my believe that current usage of D-Bus is not
performance sensitive and the number of messages exchanged
is low. I would love it if you would prove me wrong.
Or if you could show that any D-Bus related bug in Gnome3
is fixed by kdbus.
DBus is not used for performance sensitive applications because DBus
is slow. We want to make it fast so we can finally use it for
low-latency, high-throughput applications. A simple DBus
method-call+reply takes 200us here, with kdbus it takes 8us (with UDS
about 2us). If I increase the packet size from 8k to 128k, kdbus even
tops UDS thanks to single-copy transfers.
The fact that there is no performance-critical application using DBus
is, imho, an argument *pro* kdbus. People haven't been capable of
making classic dbus1 fast enough for low-latency audio, thus, we
present kdbus.
Starting up 'gdm' sends ~5k dbus messages on my machine. It takes >1s
to transmit the messages alone. Each dbus1 message has to be copied 4
times for each direction. With kdbus, each message is copied only once
for each transmission (or not at all, if you use memfds, though that
doesn't mean it's necessarily faster). No intermediate context-switch
is needed. This makes kdbus capable to transmit low-latency audio data
*inline*.
DBus marshaling is the de-facto standard in all major(!) linux desktop
systems. It is well established and accepted by many DEs. It also
solves many other problems, including: policy,
authentication/authorization, well-known name registry, efficient
broadcasts/multicasts, peer discovery, bus discovery, metadata
transmission, and more.
It is a shame that we cannot use this well-established protocol for
low-latency applications. We, effectively, have to duplicate all this
code on custom UDS and other transports just because DBus is too slow.
kdbus tries to unify those efforts, so that we don't need multiple
policy implementations, name registries and peer discovery mechanisms.
Furthermore, kdbus implements comprehensive, yet optional, metadata
transmission that allows to identify and authenticate peers in a
race-free manner (which is *not* possible with UDS).
Also, kdbus provides a single transport bus with sequential message
numbering. If you use multiple channels, you cannot give any ordering
guarantees across peers (for instance, regarding parallel
name-registry changes).
Given these theoretical advantages, here're some examples:
*) The Tizen developers have been complaining about the high latency
of DBus for polkit'ish policy queries. That's why their authentication
framework uses custom UDS sockets (called 'Cynara'). If a
UI-interaction needs multiple authentication-queries, you don't want
it to take multiple milliseconds, given that you usually want to
render the result in the same frame.
*) PulseAudio doesn't use DBus for data transmission. They had to
implement their own marshaling code, transport layer and so on, just
because DBus1-latency is horrible. With kdbus, we can basically drop
this code-duplication and unify the IPC layer. Same is true for
Wayland, btw.
*) By moving broadcast-transmission into the kernel, we can use the
time-slices of the sender to perform heavy operations. This is also
true for policy decisions, etc. With a userspace daemon, we cannot
perform operations in a time-slice of the caller. This makes DoS
attacks much harder.
*) With priority-inheritance, we can do synchronous calls into trusted
peers and let them optionally use our time-slice to perform the
action. This allows syscall-like/binder-like method-calls into other
processes. Without priority-inheritance, this is not possible in a
secure manner (see 'priority-inheritance').
*) Logging-daemons often want to attach metadata to log-messages so
debugging/filtering gets easier. If short-lived programs send
log-messages, the destination peer might not be able to read such
metadata from /proc, as the process might no longer be available at
that time. Same is true for policy-decisions like polkit does. You
cannot send off method-calls and exit. You have to wait for a reply,
even though you might not even care for it. If you don't wait, the
other side might not be able to verify your identity and as such
reject the request.
*) Even though the dbus traffic on idle-systems might be low, this
doesn't mean it's not significant at boot-times or under high-load. If
you run a dbus-monitor of your choice, you will see there is an
significant number of messages exchanged during VT-switches, startup,
shutdown, suspend, wakeup, hotplugging and similar situations where
lots of control-messages are exchanged. We don't want to spend
hundreds of ms just to transmit those messages.
*) dbus-daemon is not available during early-boot or shutdown.
These are just examples off the top of my head, but I think they're
already pretty convincing.
David
From: Johannes Stezenbach <hidden> Date: 2015-01-20 22:01:06
Hi David,
On Tue, Jan 20, 2015 at 06:00:28PM +0100, David Herrmann wrote:
[big snip]
These are just examples off the top of my head, but I think they're
already pretty convincing.
Thank you for writing this up. This is the information I was
looking for which puts kdbus into context and explains
the motivation for its development. Naturally I don't agree
with all of it, but I'm content with what I learned so far.
Daniel informed me off-list that he (and probably others) does
not understand what my questions were aiming at. I'm sorry
about that, I thought it was clear I was just lacking
the background information to understand what kdbus is and
what it is not, and why it exists -- information I couldn't find
in some hours of googling.
Thanks,
Johannes
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-21 10:28:26
Hi David,
On 01/20/2015 03:31 PM, David Herrmann wrote:
Hi Michael
On Tue, Jan 20, 2015 at 2:53 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
On 01/16/2015 08:16 PM, Greg Kroah-Hartman wrote:
quoted
From: Daniel Mack <redacted>
kdbus is a system for low-latency, low-overhead, easy to use
interprocess communication (IPC).
The interface to all functions in this driver is implemented via ioctls
on files exposed through a filesystem called 'kdbusfs'. The default
mount point of kdbusfs is /sys/fs/kdbus. This patch adds detailed
documentation about the kernel level API design.
I have some details feedback on the contents of this file, and some
bigger questions. I'll split them out into separate mails.
So here, the bigger, general questions to start with. I've arrived late
to this, so sorry if they've already been discussed, but the answers to
some of the questions should actually be in this file, I would have
expected.
This is an enormous and complex API. Why is the API ioctl() based,
rather than system-call-based? Have we learned nothing from the hydra
that the futex() multiplexing syscall became? (And kdbus is an order
of magnitude more complex, by the look of things.) At the very least,
a *good* justification of why the API is ioctl()-based should be part
of this documentation file.
An observation: The documentation below is substantial, but this API is
enormous, so the documentation still feels rather thin. What would
really help would be some example code in the doc.
And on the subject of code examples... Are there any (prototype)
working user-space applications that exercise the current kdbus
implementation? That is, can I install these kdbus patches, and
then find a simple example application somewhere that does
something to exercise kdbus?
If you run a 3.18 kernel, you can install kdbus.ko from our repository
and boot a full Fedora system running Gnome3 with kdbus, given that
you compiled systemd with --enable-kdbus (which is still
experimental). No legacy dbus1 daemon is running. Instead, we have a
bus-proxy that converts classic dbus1 to kdbus, so all
bus-communication runs on kdbus.
Good to hear. I think that some info like this should go out in
the "00/" covering mails for future patch revisions, so that people
can get some sense of the testing that has been done.
quoted
And then: is there any substantial real-world application (e.g., a
full D-Bus port) that is being developed in tandem with this kernel
side patch? (I don't mean a user-space library; I mean a seriously
large application.) This is an incredibly complex API whose
failings are only going to become evident through real-world use.
Solidifying an API in the kernel and then discovering the API
problems later when writing real-world applications would make for
a sad story. A story something like that of inotify, an API which
is an order of magnitude less complex than kdbus. (I can't help but
feel that many of inotify problems that I discuss at
https://lwn.net/Articles/605128/ might have been fixed or mitigated
if a few real-world applications had been implemented before the
API was set in stone.)
I think running a whole Gnome3 stack counts as "substantial real-world
application", right?
Yes, I'll give you that ;-).
Sure, it's a dbus1-to-kdbus layer, but all the
systemd tools use kdbus natively and it works just fine. In fact, we
all run kdbus on our main-systems every day.
We've spent over a year fixing races and API misdesigns, we've talked
to other toolkit developers (glib, qt, ..) and made sure we're
backwards compatible to dbus1. I don't think the API is perfect,
everyone makes mistakes. But with bus-proxy and systemd we have two
huge users of kdbus that put a lot of pressure on API design.
I'll say more about that in another mail in a moment. I'm not enthusiastic
about the API.
Is this library intended just for systemd? More generally, is there an
intention to provide a general purpose library API for kdbus? Or is the
intention that each application will roll a library suitable to its
needs? I think an answer to that question would be useful in this
Documentation file.
kdbus is in no way bound to systemd. There are ongoing efforts to port
glib and qt to kdbus natively. The API is pretty simple
^^^^^^^^^^^^^^^^^^^^^^^^
I think you and I must have quite different definitions of "simple"...
(For more on this point, see my reply to Daniel in a moment.)
and I don't
see how a libkdbus would simplify things. In fact, even our tests only
have slim wrappers around the ioctls to simplify error-handling in
test-scenarios.
Again, the above info would be useful in the Documentation file.
Note that most of the toolkit work is on the marshaling level, which
is independent of kdbus. kdbus just provides the transport level. DBus
is just one, yet significant, application-layer on top of kdbus. Our
test-cases use kdbus exclusively to transport raw byte streams.
From: Daniel Mack <daniel@zonque.org> Date: 2015-01-20 18:23:09
On 01/20/2015 02:53 PM, Michael Kerrisk (man-pages) wrote:
This is an enormous and complex API. Why is the API ioctl() based,
rather than system-call-based? Have we learned nothing from the hydra
that the futex() multiplexing syscall became? (And kdbus is an order
of magnitude more complex, by the look of things.) At the very least,
a *good* justification of why the API is ioctl()-based should be part
of this documentation file.
I think the simplest reason is because we want to be able to build kdbus
as a module. It's rather an optional driver than a core kernel feature.
IMO, kernel primitives should be syscalls, but kdbus is not a primitive
but an elaborate subsystem.
Also, the context the kdbus commands operate on originate from a
mountable special-purpose file system. Hence, we decided not to use a
global kernel interface but specific ioctls on the nodes exposed by kdbusfs.
Thanks,
Daniel
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-21 10:33:23
Hello Daniel,
On 01/20/2015 07:23 PM, Daniel Mack wrote:
On 01/20/2015 02:53 PM, Michael Kerrisk (man-pages) wrote:
quoted
This is an enormous and complex API. Why is the API ioctl() based,
rather than system-call-based? Have we learned nothing from the hydra
that the futex() multiplexing syscall became? (And kdbus is an order
of magnitude more complex, by the look of things.) At the very least,
a *good* justification of why the API is ioctl()-based should be part
of this documentation file.
I think the simplest reason is because we want to be able to build kdbus
as a module.
This isn't any _good_ justification...
It's rather an optional driver than a core kernel feature.
Given the various things that I've seen said about kdbus, the
preceding sentence makes little sense to me:
* kdbus will be the framework supporting user-space D-Bus in the
future, and also used by systemd, and so on pretty much every
desktop system.
* kdbus solves much of the bandwidth problem of D-Bus1. That,
along with a host of other features mean that there will be
a lot of user-space developers interested in using this API.
* Various parties in user space are already expressing strong
interest in kdbus.
My guess from the above? This will NOT be an "optional driver".
It *will be* a core kernel feature.
IMO, kernel primitives should be syscalls, but kdbus is not a primitive
but an elaborate subsystem.
Agreed. It's an elaborate subsystem. But that fact doesn't in itself
dictate any particular API design choice.
Also, the context the kdbus commands operate on originate from a
mountable special-purpose file system.
It's not clear to me how this point implies any particular API design
choice.
Hence, we decided not to use a
global kernel interface but specific ioctls on the nodes exposed by kdbusfs.
I don't follow the reasoning here at all. Here's what we have, if I
have grasped it roughly correctly:
* 16 ioctls exposed to user space.
* some 20 different structures exchanged between kernel and user space
* about 14k lines of kernel code implement the above
* some rather thin documentation of the whole lot
Sorry if that last point seems rather harsh. I know that you personally
have done a lot of work on the kdbus.txt file. David Herrmann asserts
that this is a simple API. It is not. He also suggests that there is
no need for a libkdbus. I don't know whether that's right or not, but the
point is then that there's an expectation that the raw kernel API is what
user space will need to work with.
Notwithstanding the fact that there's a lot of (good) information in
kdbus.txt, there's not nearly enough for someone to create useful,
robust applications that use that API (and not enough that I as a
reviewer feel comfortable about reviewing the API). As things stand,
user-space developers will be forced to decipher large amounts of kernel
code and existing applications in order to actually build things. And
when they do, they'll be using one of the worst APIs known to man: ioctl(),
an API that provides no type safety at all.
ioctl() is a get-out-of-jail free card when it comes to API design. Rather
than thinking carefully and long about a set of coherent, stable APIs that
provide some degree of type-safety to user-space, one just adds/changes/removes
an ioctl. And that process seems to be frequent and ongoing even now. (And
it's to your great credit that the API/ABI breaks are clearly and honestly
marked in the kdbus.h changelog.) All of this lightens the burden of API
design for kernel developers, but I'm concerned that the long-term pain
for user-space developers who use an API which (in my estimation) may
come to be widely used will be enormous.
Concretely, I'd like to see the following in kdbus.txt:
* A lot more detail, adding the various pieces that are currently missing.
I've mentioned already the absence of detail on the item blob structures,
but there's probably several other pieces as well. My problem is that the
API is so big and hard to grok that it's hard to even begin to work out
what's missing from the documentation.
* Fleshing out the API summaries with code snippets that illustrate the
use of the APIs.
* At least one simple working example application, complete with a walk
through of how it's built and run.
Yes, all of this is a big demand. But this is a big API that is being added
to the kernel, and one that may become widely used and for a long time.
It's imperative that the API is well documented, and as well designed as
possible. Furthermore, with such improved documentation I feel we'd be in
a better position to evaluate the merits of an ioctl()-based API versus
some other approach.
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
On Wed, Jan 21, 2015 at 11:32:59AM +0100, Michael Kerrisk (man-pages) wrote:
It's rather an optional driver than a core kernel feature.
Given the various things that I've seen said about kdbus, the
preceding sentence makes little sense to me:
* kdbus will be the framework supporting user-space D-Bus in the
future, and also used by systemd, and so on pretty much every
desktop system.
I have to agree with Michael here; it's really, **really**
disengenuous to say that that if you don't want kdbus, you can just
#ifconfig it out. The fact that it systemd will be using it means
that it will very shortly become a core kernel feature which is
absolutely mandatory. Sure, maybe it can be configured out for "tiny
kernels", just as in theory we can configure out the VM system for
really tiny embedded systems. But we should be treating this as
something that is not optional, because the reality is that's the way
it's going to be in very short order. So if that means to use proper
system calls instead of ioctls, we should do that.
- Ted
From: Daniel Mack <daniel@zonque.org> Date: 2015-01-21 16:58:33
Hi Michael,
On 01/21/2015 11:32 AM, Michael Kerrisk (man-pages) wrote:
On 01/20/2015 07:23 PM, Daniel Mack wrote:
quoted
It's rather an optional driver than a core kernel feature.
Given the various things that I've seen said about kdbus, the
preceding sentence makes little sense to me:
* kdbus will be the framework supporting user-space D-Bus in the
future, and also used by systemd, and so on pretty much every
desktop system.
* kdbus solves much of the bandwidth problem of D-Bus1. That,
along with a host of other features mean that there will be
a lot of user-space developers interested in using this API.
* Various parties in user space are already expressing strong
interest in kdbus.
My guess from the above? This will NOT be an "optional driver".
It *will be* a core kernel feature.
There will be userlands that will depend on kdbus, but that will still
not turn the "driver" into a core Linux kernel feature. We really want
it to be losely coupled from the rest of the kernel and optional after
all. The kernel people are working toward making more and more things
optional these days, and there will still be lots of systems that won't
be using kdbus.
quoted
Also, the context the kdbus commands operate on originate from a
mountable special-purpose file system.
It's not clear to me how this point implies any particular API design
choice.
It emphasizes the fact that our ioctl API can only be used with the
nodes exposed by kdbusfs, and vice versa. I think operations on
driver-specific files do not justify a new 'generic' syscall API.
Notwithstanding the fact that there's a lot of (good) information in
kdbus.txt, there's not nearly enough for someone to create useful,
robust applications that use that API (and not enough that I as a
reviewer feel comfortable about reviewing the API). As things stand,
user-space developers will be forced to decipher large amounts of kernel
code and existing applications in order to actually build things. And
when they do, they'll be using one of the worst APIs known to man: ioctl(),
an API that provides no type safety at all.
I don't see how ioctls are any worse than syscalls with pointers to
structures. One can screw up compatibility either way. How is an ioctl
wrapper/prototype any less type-safe than a syscall wrapper?
ioctl() is a get-out-of-jail free card when it comes to API design.
And how are syscalls different in that regard when they would both
transport the same data structures? Also note that all kdbus ioctls
necessarily operate on a file descriptor context, which an ioctl passes
along by default.
Rather
than thinking carefully and long about a set of coherent, stable APIs that
provide some degree of type-safety to user-space, one just adds/changes/removes
an ioctl.
Adding another ioctl in the future for cases we didn't think about
earlier would of course be considered a workaround; and even though such
things happen all the time, it's certainly something we'd like to avoid.
However, we would also need to add another syscall in such cases, which
is even worse IMO. So that's really not an argument against ioctls after
all. What fundamental difference between a raw syscall and a ioctl,
especially with regards to type safety, is there which would help us here?
And that process seems to be frequent and ongoing even now. (And
it's to your great credit that the API/ABI breaks are clearly and honestly
marked in the kdbus.h changelog.) All of this lightens the burden of API
design for kernel developers, but I'm concerned that the long-term pain
for user-space developers who use an API which (in my estimation) may
come to be widely used will be enormous.
Yes, we've jointly reviewed the API details again until just recently to
unify structs and enums etc, and added fields to make the ioctls structs
more versatile for possible future additions. By that, we effectively
broke the ABI, but we did that because we know we can't do such things
again in the future.
But again - I don't see how this would be different when using syscalls
rather than ioctls to transport information between the driver and
userspace. Could you elaborate?
Concretely, I'd like to see the following in kdbus.txt:
* A lot more detail, adding the various pieces that are currently missing.
I've mentioned already the absence of detail on the item blob structures,
but there's probably several other pieces as well. My problem is that the
API is so big and hard to grok that it's hard to even begin to work out
what's missing from the documentation.
* Fleshing out the API summaries with code snippets that illustrate the
use of the APIs.
* At least one simple working example application, complete with a walk
through of how it's built and run.
Yes, all of this is a big demand. But this is a big API that is being added
to the kernel, and one that may become widely used and for a long time.
Fair enough. Everything that helps people understand and use the API in
a sane way is a good thing to have, and so is getting an assessment from
people how are exposed to this API for the first time.
We'll be working on restructuring the documentation.
Thanks,
Daniel
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-22 10:19:27
Hi Daniel,
On 01/21/2015 05:58 PM, Daniel Mack wrote:
Hi Michael,
On 01/21/2015 11:32 AM, Michael Kerrisk (man-pages) wrote:
quoted
On 01/20/2015 07:23 PM, Daniel Mack wrote:
quoted
quoted
It's rather an optional driver than a core kernel feature.
Given the various things that I've seen said about kdbus, the
preceding sentence makes little sense to me:
* kdbus will be the framework supporting user-space D-Bus in the
future, and also used by systemd, and so on pretty much every
desktop system.
* kdbus solves much of the bandwidth problem of D-Bus1. That,
along with a host of other features mean that there will be
a lot of user-space developers interested in using this API.
* Various parties in user space are already expressing strong
interest in kdbus.
My guess from the above? This will NOT be an "optional driver".
It *will be* a core kernel feature.
There will be userlands that will depend on kdbus, but that will still
not turn the "driver" into a core Linux kernel feature. We really want
it to be losely coupled from the rest of the kernel and optional after
all. The kernel people are working toward making more and more things
optional these days, and there will still be lots of systems that won't
be using kdbus.
Make it optional and configurable if you want. But that misses my
point. kdbus is very likely to become an essential, widely used
piece of *general-purpose* piece of ABI infrastructure that will
be configured into virtually every type of system. As such, the same
standards should apply as for a "core kernel feature", whether or
not you want to cal it that.
quoted
quoted
Also, the context the kdbus commands operate on originate from a
mountable special-purpose file system.
It's not clear to me how this point implies any particular API design
choice.
It emphasizes the fact that our ioctl API can only be used with the
nodes exposed by kdbusfs, and vice versa. I think operations on
driver-specific files do not justify a new 'generic' syscall API.
I see your (repeated) use of words like "driver" as just a distraction,
I'm sorry. "Driver-specific files" is an implementation detail. The
point is that kdbus provides a complex, general-purpose feature for all
of userland. It is core infrastructure that will be used by key pieces
of the plumbing layer, and likely by many other applications as well.
*Please* stop suggesting that it is not core infrastructure: kdbus
has the potential to be a great IPC that will be very useful to many,
many user-space developers.
(By the way, we have precedents for device/filesystem-specific system
calls. Even a recent one, in memfd_create().)
quoted
Notwithstanding the fact that there's a lot of (good) information in
kdbus.txt, there's not nearly enough for someone to create useful,
robust applications that use that API (and not enough that I as a
reviewer feel comfortable about reviewing the API). As things stand,
user-space developers will be forced to decipher large amounts of kernel
code and existing applications in order to actually build things. And
when they do, they'll be using one of the worst APIs known to man: ioctl(),
an API that provides no type safety at all.
I don't see how ioctls are any worse than syscalls with pointers to
structures. One can screw up compatibility either way. How is an ioctl
wrapper/prototype any less type-safe than a syscall wrapper?
Taking that argument to the extreme, we would have no system calls
at all, just one gigantic ioctl() ;-).
quoted
ioctl() is a get-out-of-jail free card when it comes to API design.
And how are syscalls different in that regard when they would both
transport the same data structures?
My general impression is that when it comes to system calls,
there's usually a lot more up front thought about API design.
Also note that all kdbus ioctls
necessarily operate on a file descriptor context, which an ioctl passes
along by default.
I fail to see your point here. The same statement applies to multiple
special-purpose system calls (epoll, inotify, various shared memory APIs,
and so on).
quoted
Rather
than thinking carefully and long about a set of coherent, stable APIs that
provide some degree of type-safety to user-space, one just adds/changes/removes
an ioctl.
Adding another ioctl in the future for cases we didn't think about
earlier would of course be considered a workaround; and even though such
things happen all the time, it's certainly something we'd like to avoid.
However, we would also need to add another syscall in such cases, which
is even worse IMO. So that's really not an argument against ioctls after
all. What fundamental difference between a raw syscall and a ioctl,
especially with regards to type safety, is there which would help us here?
Type safety helps user-space application developers. System calls have
it, ioctl() does not.
quoted
And that process seems to be frequent and ongoing even now. (And
it's to your great credit that the API/ABI breaks are clearly and honestly
marked in the kdbus.h changelog.) All of this lightens the burden of API
design for kernel developers, but I'm concerned that the long-term pain
for user-space developers who use an API which (in my estimation) may
come to be widely used will be enormous.
Yes, we've jointly reviewed the API details again until just recently to
unify structs and enums etc, and added fields to make the ioctls structs
more versatile for possible future additions. By that, we effectively
broke the ABI, but we did that because we know we can't do such things
again in the future.
But again - I don't see how this would be different when using syscalls
rather than ioctls to transport information between the driver and
userspace. Could you elaborate?
My suspicion is that not nearly enough thinking has yet been done about
the design of the API. That's based on these observations:
* Documentation that is, considering the size of the API, *way* too thin.
* Some parts of the API not documented at all (various kdbus_item blobs)
* ABI changes happening even quite recently
* API oddities such as the 'kernel_flags' fields. Why do I need to
be told what flags the kernel supports on *every* operation?
The above is just after a day of looking hard at kdbus.txt. I strongly
suspect I'd find a lot of other issues if I spent more time on kdbus.
quoted
Concretely, I'd like to see the following in kdbus.txt:
* A lot more detail, adding the various pieces that are currently missing.
I've mentioned already the absence of detail on the item blob structures,
but there's probably several other pieces as well. My problem is that the
API is so big and hard to grok that it's hard to even begin to work out
what's missing from the documentation.
* Fleshing out the API summaries with code snippets that illustrate the
use of the APIs.
* At least one simple working example application, complete with a walk
through of how it's built and run.
Yes, all of this is a big demand. But this is a big API that is being added
to the kernel, and one that may become widely used and for a long time.
Fair enough. Everything that helps people understand and use the API in
a sane way is a good thing to have, and so is getting an assessment from
people how are exposed to this API for the first time.
We'll be working on restructuring the documentation.
From: David Herrmann <hidden> Date: 2015-01-22 13:46:48
Hi Michael
On Thu, Jan 22, 2015 at 11:18 AM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
On 01/21/2015 05:58 PM, Daniel Mack wrote:
quoted
quoted
quoted
Also, the context the kdbus commands operate on originate from a
mountable special-purpose file system.
It's not clear to me how this point implies any particular API design
choice.
It emphasizes the fact that our ioctl API can only be used with the
nodes exposed by kdbusfs, and vice versa. I think operations on
driver-specific files do not justify a new 'generic' syscall API.
I see your (repeated) use of words like "driver" as just a distraction,
I'm sorry. "Driver-specific files" is an implementation detail. The
point is that kdbus provides a complex, general-purpose feature for all
of userland. It is core infrastructure that will be used by key pieces
of the plumbing layer, and likely by many other applications as well.
*Please* stop suggesting that it is not core infrastructure: kdbus
has the potential to be a great IPC that will be very useful to many,
many user-space developers.
We called it an 'ipc driver' so far. It is in no way meant as
distraction. Feel free to call it 'core infrastructure'. I think we
can agree that we want it to be generically useful, like other ipc
mechanisms, including UDS and netlink.
(By the way, we have precedents for device/filesystem-specific system
calls. Even a recent one, in memfd_create().)
memfd_create() is in no way file-system specific. Internally, it uses
shmem, but that's an implementation detail. The API does not expose
this in any way. If you were referring to sealing, it's implemented as
fcntl(), not as a separate syscall. Furthermore, sealing is only
limited to shmem as it's the only volatile storage. It's not an API
restriction. Other volatile file-systems are free to implement
sealing.
quoted
quoted
ioctl() is a get-out-of-jail free card when it comes to API design.
And how are syscalls different in that regard when they would both
transport the same data structures?
My general impression is that when it comes to system calls,
there's usually a lot more up front thought about API design.
This is no technical reason why to use syscalls over ioctls. Imho,
it's rather a reason to improve the kernel's ioctl-review process.
quoted
Also note that all kdbus ioctls
necessarily operate on a file descriptor context, which an ioctl passes
along by default.
I fail to see your point here. The same statement applies to multiple
special-purpose system calls (epoll, inotify, various shared memory APIs,
and so on).
epoll and inotify don't have a 'parent' object living in the
file-system. They *need* an entry-point. We can use open() for that.
You're right, from a technical viewpoint, there's no restriction.
There're examples for both (eg., see Solaris /dev/poll, as
ioctl()-based 'epoll').
quoted
quoted
Rather
than thinking carefully and long about a set of coherent, stable APIs that
provide some degree of type-safety to user-space, one just adds/changes/removes
an ioctl.
Adding another ioctl in the future for cases we didn't think about
earlier would of course be considered a workaround; and even though such
things happen all the time, it's certainly something we'd like to avoid.
However, we would also need to add another syscall in such cases, which
is even worse IMO. So that's really not an argument against ioctls after
all. What fundamental difference between a raw syscall and a ioctl,
especially with regards to type safety, is there which would help us here?
Type safety helps user-space application developers. System calls have
it, ioctl() does not.
This is simply not true. There is no type-safety in:
syscall(__NR_xyz, some, random, arguments)
The only way a syscall gets 'type-safe', is to provide a wrapper
function. Same applies to ioctls. But people tend to not do that for
ioctls, which is, again, not a technical argument against ioctls. It's
a matter of psychology, though.
I still don't see a technical reason to use syscalls. API proposals welcome!
We're now working on a small kdbus helper library, which provides
type-safe ioctl wrappers, item-iterators and documented examples. But,
like syscalls, nobody is forced to use the wrappers. The API design is
not affected by this.
quoted
quoted
And that process seems to be frequent and ongoing even now. (And
it's to your great credit that the API/ABI breaks are clearly and honestly
marked in the kdbus.h changelog.) All of this lightens the burden of API
design for kernel developers, but I'm concerned that the long-term pain
for user-space developers who use an API which (in my estimation) may
come to be widely used will be enormous.
Yes, we've jointly reviewed the API details again until just recently to
unify structs and enums etc, and added fields to make the ioctls structs
more versatile for possible future additions. By that, we effectively
broke the ABI, but we did that because we know we can't do such things
again in the future.
But again - I don't see how this would be different when using syscalls
rather than ioctls to transport information between the driver and
userspace. Could you elaborate?
My suspicion is that not nearly enough thinking has yet been done about
the design of the API. That's based on these observations:
* Documentation that is, considering the size of the API, *way* too thin.
Ok, working on that.
* Some parts of the API not documented at all (various kdbus_item blobs)
All public structures have documentation in kdbus.h. It may need
improvements, though.
* ABI changes happening even quite recently
Please elaborate why 'recent ABI-changes' are a sign of a premature API.
I seriously doubt any API can be called 'perfect'. On the contrary, I
believe that all APIs could be improved continuously. The fact that
we, at one point, settle on an API is an admission of
backwards-compatibility. I in no way think it's a sign of
'perfection'.
With kdbus our plan is to give API-guarantees starting with upstream
inclusion. We know, that our API will not be perfect, none is. But we
will try hard to fix anything that comes up, as long as we can. And
this effort will manifest in ABI-breaks.
* API oddities such as the 'kernel_flags' fields. Why do I need to
be told what flags the kernel supports on *every* operation?
If we only returned EINVAL on invalid arguments, user-space had to
probe for each flag to see whether it's supported. By returning the
set of supported flags, user-space can cache those and _reliably_ know
which flags are supported.
We decided the overhead of a single u64 copy on each ioctl is
preferred over a separate syscall/ioctl to query kernel flags. If you
disagree, please elaborate (preferably with a suggestion how to do it
better).
The above is just after a day of looking hard at kdbus.txt. I strongly
suspect I'd find a lot of other issues if I spent more time on kdbus.
If you find the time, please do! Any hints how a specific part of the
API could be done better, is highly appreciated. A lot of the more or
less recent changes were done due to reviews from glib developers.
More help is always welcome!
Thanks
David
From: Austin S Hemmelgarn <hidden> Date: 2015-01-22 14:49:18
On 2015-01-22 08:46, David Herrmann wrote:
Hi Michael
On Thu, Jan 22, 2015 at 11:18 AM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
* API oddities such as the 'kernel_flags' fields. Why do I need to
be told what flags the kernel supports on *every* operation?
If we only returned EINVAL on invalid arguments, user-space had to
probe for each flag to see whether it's supported. By returning the
set of supported flags, user-space can cache those and _reliably_ know
which flags are supported.
We decided the overhead of a single u64 copy on each ioctl is
preferred over a separate syscall/ioctl to query kernel flags. If you
disagree, please elaborate (preferably with a suggestion how to do it
better).
While I agree that there should be a way for userspace to get the list
of supported operations, userspace apps will only actually care about
that once, when they begin talking to kdbus, because (ignoring the live
kernel patching that people have been working on recently) the list of
supported operations isn't going to change while the system is running.
While a u64 copy has relatively low overhead, it does have overhead,
and that is very significant when you consider part of the reason some
people want kdbus is for the performance gain. Especially for those
automotive applications that have been mentioned which fire off
thousands of messages during start-up, every little bit of performance
is significant.
On Thu, Jan 22, 2015 at 09:49:00AM -0500, Austin S Hemmelgarn wrote:
While I agree that there should be a way for userspace to get the list of
supported operations, userspace apps will only actually care about that
once, when they begin talking to kdbus, because (ignoring the live kernel
patching that people have been working on recently) the list of supported
operations isn't going to change while the system is running. While a u64
copy has relatively low overhead, it does have overhead, and that is very
significant when you consider part of the reason some people want kdbus is
for the performance gain. Especially for those automotive applications that
have been mentioned which fire off thousands of messages during start-up,
every little bit of performance is significant.
A single u64 in a structure is not going to be measurable at all,
processors just copy memory too fast these days for 4 extra bytes to be
noticable. So let's make this as easy as possible for userspace, making
it simpler logic there, which is much more important than saving
theoretical time in the kernel.
thanks,
greg k-h
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-26 14:46:52
Hello Greg,
On 01/23/2015 05:08 PM, Greg Kroah-Hartman wrote:
On Thu, Jan 22, 2015 at 09:49:00AM -0500, Austin S Hemmelgarn wrote:
quoted
While I agree that there should be a way for userspace to get the list of
supported operations, userspace apps will only actually care about that
once, when they begin talking to kdbus, because (ignoring the live kernel
patching that people have been working on recently) the list of supported
operations isn't going to change while the system is running. While a u64
copy has relatively low overhead, it does have overhead, and that is very
significant when you consider part of the reason some people want kdbus is
for the performance gain. Especially for those automotive applications that
have been mentioned which fire off thousands of messages during start-up,
every little bit of performance is significant.
A single u64 in a structure is not going to be measurable at all,
processors just copy memory too fast these days for 4 extra bytes to be
noticable.
It depends on the definition of measurable, I suppose, but this statement
appears incorrect to me. In some cases (e.g., kdbus_msg_info) we're talking
about *two* u64 fields (kernel_gs, kernel_msg_flags) being used to pass back
sets of valid flags. That's 16 bytes, and it definitely makes a difference.
Simply running a loop that does a naive memcpy() in a tight user-space
loop (code below), I see the following for the execution of 1e9 loops:
Including the two extra u64 fields: 3.2 sec
Without the two extra u64 fields: 2.6 sec
On the same box, doing 1e9 calls to getppid() (i.e., pretty much the
simplest syscall, giving us a rough measure of the context switch) takes
68 seconds. In other words, the cost of copying those 16 bytes is about 1%
of the base context switch/syscall cost. I assume the costs of copying
those 16 bytes across the kernel-user-space boundary would not be cheaper,
but have not tested that. If my assumption is correct, then 1% seems a
significant figure to me in an API whose raison d'être is speed.
So let's make this as easy as possible for userspace, making
it simpler logic there, which is much more important than saving
theoretical time in the kernel.
But this also missed the other part of the point. Copying these fields on
every operation, when in fact they are only needed once, clutters the API,
in my opinion. Good APIs are as simple as they can be to do their job.
Redundancy is an enemy of simplicity. Simplest would have been a one time
API that returns a structure containing all of the supported flags across
the API. Alternatively, the traditional EINVAL approach is well understood,
and suffices.
Thanks,
Michael
=========
#include <stdint.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
struct kdbus_msg_info {
uint64_t offset;
uint64_t msg_size;
uint64_t return_flags;
};
struct kdbus_cmd_send {
uint64_t size;
uint64_t flags;
#if FIELDS >= 1
uint64_t kernel_flags;
#endif
#if FIELDS >= 2
uint64_t kernel_msg_flags;
#endif
uint64_t return_flags;
uint64_t msg_address;
struct kdbus_msg_info reply;
//struct kdbus_item items[0];
} __attribute__((aligned(8)));
int
main(int argc, char *argv[])
{
long nloops, j;
struct kdbus_cmd_send src, dst;
memset(&dst, 0, sizeof(struct kdbus_cmd_send));
printf("struct size: %zd\n", sizeof(struct kdbus_cmd_send));
nloops = (argc > 1) ? atol(argv[1]) : 1000000000;
for (j = 0; j < nloops; j++) {
memcpy(&dst, &src, sizeof(struct kdbus_cmd_send));
}
exit(EXIT_SUCCESS);
}
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
From: David Herrmann <hidden> Date: 2015-01-27 15:05:24
Hi
On Mon, Jan 26, 2015 at 3:46 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
Hello Greg,
On 01/23/2015 05:08 PM, Greg Kroah-Hartman wrote:
quoted
On Thu, Jan 22, 2015 at 09:49:00AM -0500, Austin S Hemmelgarn wrote:
quoted
While I agree that there should be a way for userspace to get the list of
supported operations, userspace apps will only actually care about that
once, when they begin talking to kdbus, because (ignoring the live kernel
patching that people have been working on recently) the list of supported
operations isn't going to change while the system is running. While a u64
copy has relatively low overhead, it does have overhead, and that is very
significant when you consider part of the reason some people want kdbus is
for the performance gain. Especially for those automotive applications that
have been mentioned which fire off thousands of messages during start-up,
every little bit of performance is significant.
A single u64 in a structure is not going to be measurable at all,
processors just copy memory too fast these days for 4 extra bytes to be
noticable.
It depends on the definition of measurable, I suppose, but this statement
appears incorrect to me. In some cases (e.g., kdbus_msg_info) we're talking
about *two* u64 fields (kernel_gs, kernel_msg_flags) being used to pass back
sets of valid flags. That's 16 bytes, and it definitely makes a difference.
Simply running a loop that does a naive memcpy() in a tight user-space
loop (code below), I see the following for the execution of 1e9 loops:
Including the two extra u64 fields: 3.2 sec
Without the two extra u64 fields: 2.6 sec
On the same box, doing 1e9 calls to getppid() (i.e., pretty much the
simplest syscall, giving us a rough measure of the context switch) takes
68 seconds. In other words, the cost of copying those 16 bytes is about 1%
of the base context switch/syscall cost. I assume the costs of copying
those 16 bytes across the kernel-user-space boundary would not be cheaper,
but have not tested that. If my assumption is correct, then 1% seems a
significant figure to me in an API whose raison d'être is speed.
I have no idea how this is related to any kdbus ioctl?
A 16byte copy does not affect the performance of kdbus message
transactions in any way that matters.
quoted
So let's make this as easy as possible for userspace, making
it simpler logic there, which is much more important than saving
theoretical time in the kernel.
But this also missed the other part of the point. Copying these fields on
every operation, when in fact they are only needed once, clutters the API,
in my opinion. Good APIs are as simple as they can be to do their job.
Redundancy is an enemy of simplicity. Simplest would have been a one time
API that returns a structure containing all of the supported flags across
the API. Alternatively, the traditional EINVAL approach is well understood,
and suffices.
We're going to drop "kernel_flags" in favor of a new
KDBUS_FLAG_NEGOTIATE flag which asks the kernel to do feature
negotiation for this ioctl and return the supported flags/items inline
(overwriting the passed data). The ioctl will not be executed and will
not affect the state of the FD.
I hope this keeps the API simple.
Thanks
David
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-01-27 16:03:50
On Tue, Jan 27, 2015 at 7:05 AM, David Herrmann [off-list ref] wrote:
Hi
On Mon, Jan 26, 2015 at 3:46 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
Hello Greg,
On 01/23/2015 05:08 PM, Greg Kroah-Hartman wrote:
quoted
On Thu, Jan 22, 2015 at 09:49:00AM -0500, Austin S Hemmelgarn wrote:
quoted
While I agree that there should be a way for userspace to get the list of
supported operations, userspace apps will only actually care about that
once, when they begin talking to kdbus, because (ignoring the live kernel
patching that people have been working on recently) the list of supported
operations isn't going to change while the system is running. While a u64
copy has relatively low overhead, it does have overhead, and that is very
significant when you consider part of the reason some people want kdbus is
for the performance gain. Especially for those automotive applications that
have been mentioned which fire off thousands of messages during start-up,
every little bit of performance is significant.
A single u64 in a structure is not going to be measurable at all,
processors just copy memory too fast these days for 4 extra bytes to be
noticable.
It depends on the definition of measurable, I suppose, but this statement
appears incorrect to me. In some cases (e.g., kdbus_msg_info) we're talking
about *two* u64 fields (kernel_gs, kernel_msg_flags) being used to pass back
sets of valid flags. That's 16 bytes, and it definitely makes a difference.
Simply running a loop that does a naive memcpy() in a tight user-space
loop (code below), I see the following for the execution of 1e9 loops:
Including the two extra u64 fields: 3.2 sec
Without the two extra u64 fields: 2.6 sec
On the same box, doing 1e9 calls to getppid() (i.e., pretty much the
simplest syscall, giving us a rough measure of the context switch) takes
68 seconds. In other words, the cost of copying those 16 bytes is about 1%
of the base context switch/syscall cost. I assume the costs of copying
those 16 bytes across the kernel-user-space boundary would not be cheaper,
but have not tested that. If my assumption is correct, then 1% seems a
significant figure to me in an API whose raison d'être is speed.
I have no idea how this is related to any kdbus ioctl?
A 16byte copy does not affect the performance of kdbus message
transactions in any way that matters.
Sorry for jumping in so late. Since this version of kdbus was sent,
I've been on vacation for part of the time and I had the flu for the
rest of the time.
What are the performance goals of kdbus? How fast is it ever intended
to be? The reason I ask is that, in the current design, kdbus
collects "metadata" (credentials and other identifying information,
collected in kdbus_meta_proc_collect) from the sender of every message
*at send time*. [1] This is slow, and it will always be slow. The
slowness of this operation will, in my personal system performance
crystal ball, overshadow the cost of a 16 byte copy by several orders
of magnitude.
[1] After much discussion last time around, I'm at least convinced
that the kdbus people have reasons to like the idea of capturing
metadata for each message. I still think the design is wrong even
from a security standpoint, but right now I'm talking about
performance. If you want the data plane to be fast, it should be
separated from the control plane as much as possible, and this design
is the opposite.
--Andy
From: Daniel Mack <daniel@zonque.org> Date: 2015-01-29 08:54:03
Hi Andy,
On 01/27/2015 05:03 PM, Andy Lutomirski wrote:
On Tue, Jan 27, 2015 at 7:05 AM, David Herrmann [off-list ref] wrote:
quoted
A 16byte copy does not affect the performance of kdbus message
transactions in any way that matters.
What are the performance goals of kdbus? How fast is it ever intended
to be?
One of the design goals of kdbus is to speed up a typical D-Bus message
turnaround. That is, to minimize the number of context switches it
currently takes to get a message across the bus, and to avoid
unnecessary extra payload copies.
Even though I'm sure there's still room for future improvement, the
benchmark test we provided in the kernel self-tests shows that basic
data transmission performance is roughly comparable to that of UDS for
smaller payloads. For payloads of bigger sizes (>128kb), kdbus is
actually faster due to its zero-copy mechanism.
The reason I ask is that, in the current design, kdbus
collects "metadata" (credentials and other identifying information,
collected in kdbus_meta_proc_collect) from the sender of every message
*at send time*. [1] This is slow, and it will always be slow. The
slowness of this operation will, in my personal system performance
crystal ball, overshadow the cost of a 16 byte copy by several orders
of magnitude.
That's certainly true, but that's not a contradiction to the performance
argument. Please keep in mind that if a receiving peer does not request
any metadata, the kernel doesn't collect and attach any. We know that
gathering of some of the metadata comes at a price, which is why we
split up the information is such fine-grained pieces.
Also note that if a receiving peer opts in for a certain piece of
metadata, it should do that that for a good reason, because it needs
that data to process a request. Letting kdbus do the work of providing
such information is still a lot faster than having the receiving peer
gather it itself, as that would involve more syscalls and more context
switches (let alone the fact that doing so is inherently racy, as
explained in earlier threads).
So, yes, collecting metadata can slow down message exchange, but after
all, that's an optional feature that has to be used with sense. I'll add
some words on that to the man-pages.
HTH,
Daniel
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-01-29 11:26:06
On Jan 29, 2015 3:53 AM, "Daniel Mack" [off-list ref] wrote:
Hi Andy,
On 01/27/2015 05:03 PM, Andy Lutomirski wrote:
quoted
On Tue, Jan 27, 2015 at 7:05 AM, David Herrmann [off-list ref] wrote:
quoted
quoted
A 16byte copy does not affect the performance of kdbus message
transactions in any way that matters.
quoted
What are the performance goals of kdbus? How fast is it ever intended
to be?
One of the design goals of kdbus is to speed up a typical D-Bus message
turnaround. That is, to minimize the number of context switches it
currently takes to get a message across the bus, and to avoid
unnecessary extra payload copies.
Even though I'm sure there's still room for future improvement, the
benchmark test we provided in the kernel self-tests shows that basic
data transmission performance is roughly comparable to that of UDS for
smaller payloads. For payloads of bigger sizes (>128kb), kdbus is
actually faster due to its zero-copy mechanism.
quoted
The reason I ask is that, in the current design, kdbus
collects "metadata" (credentials and other identifying information,
collected in kdbus_meta_proc_collect) from the sender of every message
*at send time*. [1] This is slow, and it will always be slow. The
slowness of this operation will, in my personal system performance
crystal ball, overshadow the cost of a 16 byte copy by several orders
of magnitude.
That's certainly true, but that's not a contradiction to the performance
argument. Please keep in mind that if a receiving peer does not request
any metadata, the kernel doesn't collect and attach any. We know that
gathering of some of the metadata comes at a price, which is why we
split up the information is such fine-grained pieces.
Also note that if a receiving peer opts in for a certain piece of
metadata, it should do that that for a good reason, because it needs
that data to process a request. Letting kdbus do the work of providing
such information is still a lot faster than having the receiving peer
gather it itself, as that would involve more syscalls and more context
switches (let alone the fact that doing so is inherently racy, as
explained in earlier threads).
All this is true, but if you used connect-time metadata, this would be
a non-issue.
Given that I see almost no advantage to send-time metadata, and I see
three disadvantages (slower, inconsistent with the basic POSIX model,
and inconsistent with existing user-space dbus), I still don't see why
you designed it this way.
There's an added disadvantage of the current design: if a kdbus user
is communicating with a traditional d-bus user using the proxy, then
IIUC the credentials at the time of connection get used.
In summary, the current design is (a) unlike almost everything else
that uses file descriptors, (b) much slower, (c) different from
traditional d-bus, and (d) gives inconsistent behavior to new clients
depending on what server they're connecting to.
--Andy
So, yes, collecting metadata can slow down message exchange, but after
all, that's an optional feature that has to be used with sense. I'll add
some words on that to the man-pages.
HTH,
Daniel
From: Daniel Mack <daniel@zonque.org> Date: 2015-01-29 11:42:33
On 01/29/2015 12:25 PM, Andy Lutomirski wrote:
On Jan 29, 2015 3:53 AM, "Daniel Mack" [off-list ref] wrote:
quoted
Also note that if a receiving peer opts in for a certain piece of
metadata, it should do that that for a good reason, because it needs
that data to process a request. Letting kdbus do the work of providing
such information is still a lot faster than having the receiving peer
gather it itself, as that would involve more syscalls and more context
switches (let alone the fact that doing so is inherently racy, as
explained in earlier threads).
Given that I see almost no advantage to send-time metadata, and I see
three disadvantages (slower, inconsistent with the basic POSIX model,
and inconsistent with existing user-space dbus), I still don't see why
you designed it this way.
Because effective information about tasks may change over time, and
D-Bus is a connection-less protocol that has no notion of peer-to-peer
connections.
As we explained before, currently, D-Bus peers do collect the same
information already if they need to have them, but they have to do deal
with the inherit races in such cases. kdbus is closing the gap by
optionally providing the same information along with each message, if
requested.
There's an added disadvantage of the current design: if a kdbus user
is communicating with a traditional d-bus user using the proxy, then
IIUC the credentials at the time of connection get used.
That's not quite true any more. After our discussion in v2, we agreed on
dropping this detail. If you're using the proxy, no metadata is attached
to messages any more. Userspace has to gather this information in the
traditional, racy way in such cases. You are right - metadata about the
proxy task is of no interest here, and hence dropping the information
altogether is the most consistent thing we can do.
But again - that metadata thing just an optional feature. People
developing with the bare kernel-level API are free to ignore all that
and just just kdbus as low-level protocol for reliable multicast. Note
that in such cases, you would still be able to retrieve the connect-time
metadata if that's needed.
Thanks,
Daniel
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-01-29 12:10:21
On Jan 29, 2015 6:42 AM, "Daniel Mack" [off-list ref] wrote:
On 01/29/2015 12:25 PM, Andy Lutomirski wrote:
quoted
On Jan 29, 2015 3:53 AM, "Daniel Mack" [off-list ref] wrote:
quoted
quoted
Also note that if a receiving peer opts in for a certain piece of
metadata, it should do that that for a good reason, because it needs
that data to process a request. Letting kdbus do the work of providing
such information is still a lot faster than having the receiving peer
gather it itself, as that would involve more syscalls and more context
switches (let alone the fact that doing so is inherently racy, as
explained in earlier threads).
quoted
Given that I see almost no advantage to send-time metadata, and I see
three disadvantages (slower, inconsistent with the basic POSIX model,
and inconsistent with existing user-space dbus), I still don't see why
you designed it this way.
Because effective information about tasks may change over time, and
D-Bus is a connection-less protocol that has no notion of peer-to-peer
connections.
As we explained before, currently, D-Bus peers do collect the same
information already if they need to have them, but they have to do deal
with the inherit races in such cases. kdbus is closing the gap by
optionally providing the same information along with each message, if
requested.
In all these discussions, no one ever gave a decent example use case.
If a process drops some privilege, it must close all fds it has that
captured its old privilege. This has nothing to do with kdbus. With
kdbus, you still need to close and reopen your kdbus fd, unless you've
disabled that bit of metadata, so using send-time metadata hasn't
bought you benefit that I can see.
I agree that the design seems to have improved to a state of being at
least decent, but that doesn't mean that using send-time metadata is a
good idea for systemd or for anything else.
quoted
There's an added disadvantage of the current design: if a kdbus user
is communicating with a traditional d-bus user using the proxy, then
IIUC the credentials at the time of connection get used.
That's not quite true any more. After our discussion in v2, we agreed on
dropping this detail. If you're using the proxy, no metadata is attached
to messages any more. Userspace has to gather this information in the
traditional, racy way in such cases. You are right - metadata about the
proxy task is of no interest here, and hence dropping the information
altogether is the most consistent thing we can do.
But again - that metadata thing just an optional feature. People
developing with the bare kernel-level API are free to ignore all that
and just just kdbus as low-level protocol for reliable multicast. Note
that in such cases, you would still be able to retrieve the connect-time
metadata if that's needed.
It's an optional feature that will get used, non-optionally, thousands
of times on each boot, apparently. Keep in mind that it's also a
scalability problem because it takes locks. If it ever gets used
thousands of times per CPU on a big thousand-core machine, it's going
to suck, and you'll have backed yourself into a corner.
--Andy
From: Daniel Mack <daniel@zonque.org> Date: 2015-02-02 09:35:00
Hi Andy,
On 01/29/2015 01:09 PM, Andy Lutomirski wrote:
On Jan 29, 2015 6:42 AM, "Daniel Mack" [off-list ref] wrote:
quoted
As we explained before, currently, D-Bus peers do collect the same
information already if they need to have them, but they have to do deal
with the inherit races in such cases. kdbus is closing the gap by
optionally providing the same information along with each message, if
requested.
In all these discussions, no one ever gave a decent example use case.
If a process drops some privilege, it must close all fds it has that
captured its old privilege. This has nothing to do with kdbus.
kdbus does not implement any new concept here but sticks to what
SCM_CREDENTIALS does on SOL_SEQPACKET. An application can get a
file-descriptor from socket() or socketpair() and freely pass it around
between different tasks or threads, but messages will always have the
credentials attached that are valid at *send* time. SO_PEERCREDS,
however, still reports the connect-time credentials, and kdbus provides
exactly the same semantics and both ways of retrieving information.
I agree that the design seems to have improved to a state of being at
least decent,
One reason for that is your feedback. Thanks for that again!
It's an optional feature that will get used, non-optionally, thousands
of times on each boot, apparently. Keep in mind that it's also a
scalability problem because it takes locks. If it ever gets used
thousands of times per CPU on a big thousand-core machine, it's going
to suck, and you'll have backed yourself into a corner.
That's right, but again - if an application wants to gather this kind of
information about tasks it interacts with, it can do so today by looking
at /proc or similar sources. Desktop machines do exactly that already,
and the kernel code executed in such cases very much resembles that in
metadata.c, and is certainly not cheaper. kdbus just makes such
information more accessible when requested. Which information is
collected is defined by bit-masks on both the sender and the receiver
connection, and most applications will effectively only use a very
limited set by default if they go through one of the more high-level
libraries.
Also, when metadata is collected, the code mostly takes temporary
references on objects like PIDs, namespaces etc. Which operation would
you consider particularly expensive?
Thanks again,
Daniel
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-02-02 20:12:41
On Feb 2, 2015 1:34 AM, "Daniel Mack" [off-list ref] wrote:
Hi Andy,
On 01/29/2015 01:09 PM, Andy Lutomirski wrote:
quoted
On Jan 29, 2015 6:42 AM, "Daniel Mack" [off-list ref] wrote:
quoted
quoted
As we explained before, currently, D-Bus peers do collect the same
information already if they need to have them, but they have to do deal
with the inherit races in such cases. kdbus is closing the gap by
optionally providing the same information along with each message, if
requested.
In all these discussions, no one ever gave a decent example use case.
If a process drops some privilege, it must close all fds it has that
captured its old privilege. This has nothing to do with kdbus.
kdbus does not implement any new concept here but sticks to what
SCM_CREDENTIALS does on SOL_SEQPACKET. An application can get a
file-descriptor from socket() or socketpair() and freely pass it around
between different tasks or threads, but messages will always have the
credentials attached that are valid at *send* time. SO_PEERCREDS,
however, still reports the connect-time credentials, and kdbus provides
exactly the same semantics and both ways of retrieving information.
quoted
I agree that the design seems to have improved to a state of being at
least decent,
One reason for that is your feedback. Thanks for that again!
quoted
It's an optional feature that will get used, non-optionally, thousands
of times on each boot, apparently. Keep in mind that it's also a
scalability problem because it takes locks. If it ever gets used
thousands of times per CPU on a big thousand-core machine, it's going
to suck, and you'll have backed yourself into a corner.
That's right, but again - if an application wants to gather this kind of
information about tasks it interacts with, it can do so today by looking
at /proc or similar sources. Desktop machines do exactly that already,
and the kernel code executed in such cases very much resembles that in
metadata.c, and is certainly not cheaper. kdbus just makes such
information more accessible when requested. Which information is
collected is defined by bit-masks on both the sender and the receiver
connection, and most applications will effectively only use a very
limited set by default if they go through one of the more high-level
libraries.
I should rephrase a bit. Kdbus doesn't require use of send-time
metadata. It does, however, strongly encourage it, and it sounds like
systemd and other major users will use send-time metadata. Once that
happens, it's ABI (even if it's purely in userspace), and changing it
is asking for security holes to pop up. So you'll be mostly stuck
with it.
Also, when metadata is collected, the code mostly takes temporary
references on objects like PIDs, namespaces etc. Which operation would
you consider particularly expensive?
The refcounting, copies of some of the data, and counting bytes and
allocating space. The refcounting is the part that will scale
particularly badly on many CPUs.
Do you have some simple benchmark code you can share? I'd like to
play with it a bit.
--Andy
From: Daniel Mack <daniel@zonque.org> Date: 2015-02-03 10:09:54
Hi Andy,
On 02/02/2015 09:12 PM, Andy Lutomirski wrote:
On Feb 2, 2015 1:34 AM, "Daniel Mack" [off-list ref] wrote:
quoted
That's right, but again - if an application wants to gather this kind of
information about tasks it interacts with, it can do so today by looking
at /proc or similar sources. Desktop machines do exactly that already,
and the kernel code executed in such cases very much resembles that in
metadata.c, and is certainly not cheaper. kdbus just makes such
information more accessible when requested. Which information is
collected is defined by bit-masks on both the sender and the receiver
connection, and most applications will effectively only use a very
limited set by default if they go through one of the more high-level
libraries.
I should rephrase a bit. Kdbus doesn't require use of send-time
metadata. It does, however, strongly encourage it, and it sounds like
On the kernel level, kdbus just *offers* that, just like sockets offer
SO_PASSCRED. On the userland level, kdbus helps applications get that
information race-free, easier and faster than they would otherwise.
systemd and other major users will use send-time metadata. Once that
happens, it's ABI (even if it's purely in userspace), and changing it
is asking for security holes to pop up. So you'll be mostly stuck
with it.
We know we can't break the ABI. At most, we could deprecate item types
and introduce new ones, but we want to avoid that by all means of
course. However, I fail to see how that is related to send time
metadata, or even to kdbus in general, as all ABIs have to be kept stable.
Do you have some simple benchmark code you can share? I'd like to
play with it a bit.
Sure, it's part of the self-test suite. Call it with "-t benchmark" to
run the benchmark as isolated test with verbose output. The code for
that lives in test-benchmark.c.
Thanks,
Daniel
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-02-04 00:41:29
On Tue, Feb 3, 2015 at 2:09 AM, Daniel Mack [off-list ref] wrote:
Hi Andy,
On 02/02/2015 09:12 PM, Andy Lutomirski wrote:
quoted
On Feb 2, 2015 1:34 AM, "Daniel Mack" [off-list ref] wrote:
quoted
quoted
That's right, but again - if an application wants to gather this kind of
information about tasks it interacts with, it can do so today by looking
at /proc or similar sources. Desktop machines do exactly that already,
and the kernel code executed in such cases very much resembles that in
metadata.c, and is certainly not cheaper. kdbus just makes such
information more accessible when requested. Which information is
collected is defined by bit-masks on both the sender and the receiver
connection, and most applications will effectively only use a very
limited set by default if they go through one of the more high-level
libraries.
I should rephrase a bit. Kdbus doesn't require use of send-time
metadata. It does, however, strongly encourage it, and it sounds like
On the kernel level, kdbus just *offers* that, just like sockets offer
SO_PASSCRED. On the userland level, kdbus helps applications get that
information race-free, easier and faster than they would otherwise.
quoted
systemd and other major users will use send-time metadata. Once that
happens, it's ABI (even if it's purely in userspace), and changing it
is asking for security holes to pop up. So you'll be mostly stuck
with it.
We know we can't break the ABI. At most, we could deprecate item types
and introduce new ones, but we want to avoid that by all means of
course. However, I fail to see how that is related to send time
metadata, or even to kdbus in general, as all ABIs have to be kept stable.
I should have said it differently. ABI is the wrong term -- it's more
of a protocol issue.
It looks like, with the current code, the kernel will provide
(optional) send-time metadata, and the sd-bus library will use it.
The result will be that the communication protocol between clients and
udev, systemd, systemd-logind, g-s-d, etc, will likely involve
send-time metadata. This may end up being a bottleneck.
Once this happens, changing the protocol will be very hard without
introducing security bugs. If people start switching to
connection-time metadata to gain performance, then they'll break both
the communication protocol and the expectations of client code. (In
fact, it'll break twice, sort of, since I think that the current
protocols are connect-time.)
To me, this seems like a down-side of using send-time metadata, albeit
possibly not a huge downside at least in the near term. I don't see a
corresponding benefit, though.
quoted
Do you have some simple benchmark code you can share? I'd like to
play with it a bit.
Sure, it's part of the self-test suite. Call it with "-t benchmark" to
run the benchmark as isolated test with verbose output. The code for
that lives in test-benchmark.c.
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-02-04 23:03:33
On Tue, Feb 3, 2015 at 2:09 AM, Daniel Mack [off-list ref] wrote:
Hi Andy,
On 02/02/2015 09:12 PM, Andy Lutomirski wrote:
quoted
On Feb 2, 2015 1:34 AM, "Daniel Mack" [off-list ref] wrote:
quoted
quoted
That's right, but again - if an application wants to gather this kind of
information about tasks it interacts with, it can do so today by looking
at /proc or similar sources. Desktop machines do exactly that already,
and the kernel code executed in such cases very much resembles that in
metadata.c, and is certainly not cheaper. kdbus just makes such
information more accessible when requested. Which information is
collected is defined by bit-masks on both the sender and the receiver
connection, and most applications will effectively only use a very
limited set by default if they go through one of the more high-level
libraries.
I should rephrase a bit. Kdbus doesn't require use of send-time
metadata. It does, however, strongly encourage it, and it sounds like
On the kernel level, kdbus just *offers* that, just like sockets offer
SO_PASSCRED. On the userland level, kdbus helps applications get that
information race-free, easier and faster than they would otherwise.
quoted
systemd and other major users will use send-time metadata. Once that
happens, it's ABI (even if it's purely in userspace), and changing it
is asking for security holes to pop up. So you'll be mostly stuck
with it.
We know we can't break the ABI. At most, we could deprecate item types
and introduce new ones, but we want to avoid that by all means of
course. However, I fail to see how that is related to send time
metadata, or even to kdbus in general, as all ABIs have to be kept stable.
quoted
Do you have some simple benchmark code you can share? I'd like to
play with it a bit.
Sure, it's part of the self-test suite. Call it with "-t benchmark" to
run the benchmark as isolated test with verbose output. The code for
that lives in test-benchmark.c.
I see "latencies" of around 20 microseconds with lockdep and context
tracking off. For example:
stats (UNIX): 226730 packets processed, latency (nsecs) min/max/avg
3845 // 34828 // 4069
stats (KDBUS): 37103 packets processed, latency (nsecs) min/max/avg
19123 // 99660 // 20696
This is IMO not very good. With memfds off:
stats (UNIX): 226061 packets processed, latency (nsecs) min/max/avg
3885 // 32019 // 4079
stats (KDBUS): 83284 packets processed, latency (nsecs) min/max/avg
10525 // 42578 // 10932
With memfds off and the payload set to 8 bytes:
stats (KDBUS): 77669 packets processed, latency (nsecs) min/max/avg
9963 // 64325 // 11645
stats (UNIX): 253695 packets processed, latency (nsecs) min/max/avg
2986 // 56094 // 3565
Am I missing something here? This is slow enough that a lightweight
userspace dbus daemon should be able to outperform kdbus, or at least
come very close.
It would be kind of nice to know how long just the send call takes, too.
--Andy
From: David Herrmann <hidden> Date: 2015-02-05 00:16:40
Hi
On Thu, Feb 5, 2015 at 12:03 AM, Andy Lutomirski [off-list ref] wrote:
I see "latencies" of around 20 microseconds with lockdep and context
tracking off. For example:
Without metadata nor memfd transmission, I get 2.5us for kdbus, 1.5us
for UDS (8k payload). With 8-byte payloads, I get 2.2us and 1.2us. I
suspect you enabled metadata transmission, which I think is not a fair
comparison.
A few notes on that:
* kdbus is a bus layer. We don't intend to replace UDS, but improve
dbus. Comparing roundtrip times with UDS is tempting, but in no way
fair. To the very least, a bus layer has to perform peer-lookup, which
UDS does not have to do. Imo, 2.5us vs. 1.5us is already pretty nice.
Compare this to ~77us for dbus1 without marshaling.
* We have not optimized kdbus code-paths for speed, yet. Our main
concerns are algorithmic challenges, and we believe they've been
improved considerably with kdbus. I have constantly measured kdbus
performance with 'perf' and flame-graphs, and there're a lot of
possible optimizations (especially on locking). However, I think this
can be done afterwards just fine. Neither API nor ioctl overhead has
shown up in my measurements. If anyone has counter evidence, please
let us know. But I'm a bit reluctant to change our API solely based on
performance guesses.
* We're about 50% slower than UDS on 1-byte transmissions. With 32k
we're on-par. How can a lightweight user-space daemon even get close
to that?
* Broadcast performance is a completely different story. SEND gets
around 30% faster compared to kdbus unicasts (as most of the
control-paths are only taken once per message, instead of once per
destination).
* test-benchmark.c does performance tests in a single process. If the
bus-layer is implemented in user-space, you need to account for
context-switches and task wakeups. My UDS and pipe round-trip latency
tests got around 3x slower if done cross processes (3.7us instead of
1.2us). With a user-space daemon, those slow-downs are taken two times
more often for each roundtrip.
* Process time is accounted on the sender, instead of a shared process
(dbus-daemon). Broadcasts will thus no longer consume time-slices of
dbus-daemon, but only the sender's.
With kdbus, we implement a bus-layer. This is our only target! If your
target environment does not require a bus, then don't use kdbus. We
don't intend to replace UDS. On a bus-layer, we need peer-discovery,
policy-handling, destination-lookups, broadcast-management and more.
Pipes/UDS do not provide any of this.
I cannot see how any other existing bus-implementation comes even
close to kdbus, performance-wise. If someone does, please let us know!
Thanks
David
From: Andy Lutomirski <luto@amacapital.net> Date: 2015-02-08 16:55:07
On Feb 4, 2015 4:16 PM, "David Herrmann" [off-list ref] wrote:
Hi
On Thu, Feb 5, 2015 at 12:03 AM, Andy Lutomirski [off-list ref] wrote:
quoted
I see "latencies" of around 20 microseconds with lockdep and context
tracking off. For example:
Without metadata nor memfd transmission, I get 2.5us for kdbus, 1.5us
for UDS (8k payload). With 8-byte payloads, I get 2.2us and 1.2us. I
suspect you enabled metadata transmission, which I think is not a fair
comparison.
I tried to disable metadata. I may have failed.
Regardless, if metadata is very slow, then that's more reason not to
use it on send. And if you shouldn't use it, then maybe the kernel
shouldn't provide it.
I assumed there was a context switch in there. I can try to test
differently. If UDS is twice as fast *with* a contest switch, then a
userspace solution should be faster.
Also, UDS can use memfds, too.
A few notes on that:
* kdbus is a bus layer. We don't intend to replace UDS, but improve
dbus. Comparing roundtrip times with UDS is tempting, but in no way
fair. To the very least, a bus layer has to perform peer-lookup, which
UDS does not have to do. Imo, 2.5us vs. 1.5us is already pretty nice.
Compare this to ~77us for dbus1 without marshaling.
This makes me wonder what dbus1 is doing wrong.
* We have not optimized kdbus code-paths for speed, yet. Our main
concerns are algorithmic challenges, and we believe they've been
improved considerably with kdbus. I have constantly measured kdbus
performance with 'perf' and flame-graphs, and there're a lot of
possible optimizations (especially on locking). However, I think this
can be done afterwards just fine. Neither API nor ioctl overhead has
shown up in my measurements. If anyone has counter evidence, please
let us know. But I'm a bit reluctant to change our API solely based on
performance guesses.
But removal of send-time metadata can't be done after the fact.
--Andy
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-27 18:03:57
Hi David,
On 01/27/2015 04:05 PM, David Herrmann wrote:
Hi
On Mon, Jan 26, 2015 at 3:46 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
Hello Greg,
On 01/23/2015 05:08 PM, Greg Kroah-Hartman wrote:
quoted
On Thu, Jan 22, 2015 at 09:49:00AM -0500, Austin S Hemmelgarn wrote:
quoted
While I agree that there should be a way for userspace to get the list of
supported operations, userspace apps will only actually care about that
once, when they begin talking to kdbus, because (ignoring the live kernel
patching that people have been working on recently) the list of supported
operations isn't going to change while the system is running. While a u64
copy has relatively low overhead, it does have overhead, and that is very
significant when you consider part of the reason some people want kdbus is
for the performance gain. Especially for those automotive applications that
have been mentioned which fire off thousands of messages during start-up,
every little bit of performance is significant.
A single u64 in a structure is not going to be measurable at all,
processors just copy memory too fast these days for 4 extra bytes to be
noticable.
It depends on the definition of measurable, I suppose, but this statement
appears incorrect to me. In some cases (e.g., kdbus_msg_info) we're talking
about *two* u64 fields (kernel_gs, kernel_msg_flags) being used to pass back
sets of valid flags. That's 16 bytes, and it definitely makes a difference.
Simply running a loop that does a naive memcpy() in a tight user-space
loop (code below), I see the following for the execution of 1e9 loops:
Including the two extra u64 fields: 3.2 sec
Without the two extra u64 fields: 2.6 sec
On the same box, doing 1e9 calls to getppid() (i.e., pretty much the
simplest syscall, giving us a rough measure of the context switch) takes
68 seconds. In other words, the cost of copying those 16 bytes is about 1%
of the base context switch/syscall cost. I assume the costs of copying
those 16 bytes across the kernel-user-space boundary would not be cheaper,
but have not tested that. If my assumption is correct, then 1% seems a
significant figure to me in an API whose raison d'être is speed.
I have no idea how this is related to any kdbus ioctl?
A 16byte copy does not affect the performance of kdbus message
transactions in any way that matters.
I'm not sure if it's related/significant or not, since I'm ignorant
of the performance figures for kdbus. I just got curious when Greg
stated that the cost of copying would not be noticeable. (I got curious
also about my assumption, and did an experiment with a dummy system call
that throws bytes across the fence into user space. The cost of an
extra 16 bytes (56 to 72 bytes) is about 3% of the base syscall/context
switch cost.)
quoted
quoted
So let's make this as easy as possible for userspace, making
it simpler logic there, which is much more important than saving
theoretical time in the kernel.
But this also missed the other part of the point. Copying these fields on
every operation, when in fact they are only needed once, clutters the API,
in my opinion. Good APIs are as simple as they can be to do their job.
Redundancy is an enemy of simplicity. Simplest would have been a one time
API that returns a structure containing all of the supported flags across
the API. Alternatively, the traditional EINVAL approach is well understood,
and suffices.
We're going to drop "kernel_flags" in favor of a new
KDBUS_FLAG_NEGOTIATE flag which asks the kernel to do feature
negotiation for this ioctl and return the supported flags/items inline
(overwriting the passed data). The ioctl will not be executed and will
not affect the state of the FD.
I hope this keeps the API simple.
Not sure I quite understand the details from your description, but I assume
the it'll end up in the doc, and I'll try to take a look later.
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-23 11:47:17
Hi David,
On 01/22/2015 02:46 PM, David Herrmann wrote:
Hi Michael
On Thu, Jan 22, 2015 at 11:18 AM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
On 01/21/2015 05:58 PM, Daniel Mack wrote:
quoted
quoted
quoted
Also, the context the kdbus commands operate on originate from a
mountable special-purpose file system.
It's not clear to me how this point implies any particular API design
choice.
It emphasizes the fact that our ioctl API can only be used with the
nodes exposed by kdbusfs, and vice versa. I think operations on
driver-specific files do not justify a new 'generic' syscall API.
I see your (repeated) use of words like "driver" as just a distraction,
I'm sorry. "Driver-specific files" is an implementation detail. The
point is that kdbus provides a complex, general-purpose feature for all
of userland. It is core infrastructure that will be used by key pieces
of the plumbing layer, and likely by many other applications as well.
*Please* stop suggesting that it is not core infrastructure: kdbus
has the potential to be a great IPC that will be very useful to many,
many user-space developers.
We called it an 'ipc driver' so far. It is in no way meant as
distraction. Feel free to call it 'core infrastructure'. I think we
can agree that we want it to be generically useful, like other ipc
mechanisms, including UDS and netlink.
Yes.
quoted
(By the way, we have precedents for device/filesystem-specific system
calls. Even a recent one, in memfd_create().)
memfd_create() is in no way file-system specific. Internally, it uses
shmem, but that's an implementation detail. The API does not expose
this in any way. If you were referring to sealing, it's implemented as
fcntl(), not as a separate syscall. Furthermore, sealing is only
limited to shmem as it's the only volatile storage. It's not an API
restriction. Other volatile file-systems are free to implement
sealing.
My bad. I mispoke there.
quoted
quoted
quoted
ioctl() is a get-out-of-jail free card when it comes to API design.
And how are syscalls different in that regard when they would both
transport the same data structures?
My general impression is that when it comes to system calls,
there's usually a lot more up front thought about API design.
This is no technical reason why to use syscalls over ioctls. Imho,
it's rather a reason to improve the kernel's ioctl-review process.
Agreed it's not a technical reason. But the distinction I make
does capture how things usually work.
[...]
quoted
quoted
quoted
Rather
than thinking carefully and long about a set of coherent, stable APIs that
provide some degree of type-safety to user-space, one just adds/changes/removes
an ioctl.
Adding another ioctl in the future for cases we didn't think about
earlier would of course be considered a workaround; and even though such
things happen all the time, it's certainly something we'd like to avoid.
However, we would also need to add another syscall in such cases, which
is even worse IMO. So that's really not an argument against ioctls after
all. What fundamental difference between a raw syscall and a ioctl,
especially with regards to type safety, is there which would help us here?
Type safety helps user-space application developers. System calls have
it, ioctl() does not.
This is simply not true. There is no type-safety in:
syscall(__NR_xyz, some, random, arguments)
The only way a syscall gets 'type-safe', is to provide a wrapper
function. Same applies to ioctls. But people tend to not do that for
ioctls, which is, again, not a technical argument against ioctls. It's
a matter of psychology, though.
Yes, I see I wasn't quite clear enough. I should have said:
Type safety helps user-space application developers.
*As typically provided to user-space via libc wrappers*,
system calls have it, ioctl() does not.
And system call wrappers are generally provided pretty much automatically
by libcs (at least, mostly, there's some to-ing and fro-ing about this in
glibc-land these days as you are aware from the memfd_create() story;
http://thread.gmane.org/gmane.comp.lib.glibc.alpha/45884/focus=46937).
So, in practice user-space programmers typically automatically get type
safety with system calls, but not with ioctl().
I still don't see a technical reason to use syscalls. API proposals welcome!
We're now working on a small kdbus helper library, which provides
type-safe ioctl wrappers, item-iterators and documented examples. But,
like syscalls, nobody is forced to use the wrappers. The API design is
not affected by this.
quoted
quoted
quoted
And that process seems to be frequent and ongoing even now. (And
it's to your great credit that the API/ABI breaks are clearly and honestly
marked in the kdbus.h changelog.) All of this lightens the burden of API
design for kernel developers, but I'm concerned that the long-term pain
for user-space developers who use an API which (in my estimation) may
come to be widely used will be enormous.
Yes, we've jointly reviewed the API details again until just recently to
unify structs and enums etc, and added fields to make the ioctls structs
more versatile for possible future additions. By that, we effectively
broke the ABI, but we did that because we know we can't do such things
again in the future.
But again - I don't see how this would be different when using syscalls
rather than ioctls to transport information between the driver and
userspace. Could you elaborate?
My suspicion is that not nearly enough thinking has yet been done about
the design of the API. That's based on these observations:
* Documentation that is, considering the size of the API, *way* too thin.
Ok, working on that.
quoted
* Some parts of the API not documented at all (various kdbus_item blobs)
All public structures have documentation in kdbus.h. It may need
improvements, though.
Again -- that's very thin--one liners aand sentence fragments for the most
part. (Not that I think that needs to be fixed, just that that doesn't
fit with my definition of "documented", which should be something like
what I've requested for kdbus.txt.)
quoted
* ABI changes happening even quite recently
Please elaborate why 'recent ABI-changes' are a sign of a premature API.
I seriously doubt any API can be called 'perfect'. On the contrary, I
believe that all APIs could be improved continuously. The fact that
we, at one point, settle on an API is an admission of
backwards-compatibility. I in no way think it's a sign of
'perfection'.
I agree that no API is perfect. But some emerge from the starting gate
in much better state than others. kdbus will likely be an important API,
and it's important that it's well designed one at the outset.
With kdbus our plan is to give API-guarantees starting with upstream
inclusion. We know, that our API will not be perfect, none is. But we
will try hard to fix anything that comes up, as long as we can. And
this effort will manifest in ABI-breaks.
Fair enough. My concern is that upstream inclusion is being rushed before
the API design can be well assessed.
quoted
* API oddities such as the 'kernel_flags' fields. Why do I need to
be told what flags the kernel supports on *every* operation?
If we only returned EINVAL on invalid arguments, user-space had to
probe for each flag to see whether it's supported. By returning the
set of supported flags, user-space can cache those and _reliably_ know
(Not sure why you emphasize "reliably" here...)
which flags are supported.
We decided the overhead of a single u64 copy on each ioctl is
preferred over a separate syscall/ioctl to query kernel flags. If you
disagree, please elaborate (preferably with a suggestion how to do it
better).
Well that's a quite unconventional design choice. Determining the set
of supported flags in an API is a one-time operation. The natural--and
I would say, *obviously* better--approach to this would be either the
traditional EINVAL approach or an API that is called once to retrieve
the set of supported flags. Instead, kdbus clutters the APIs with a
mostly unneeded extra piece of information on *every* call, and
fractionally increases the run time, for no good reason.
Now, you might say that my suggested alternatives are not obviously
better. My response is that, at the very least, in the documentation
I'd expect to see this unconventional approach clearly highlighted
and accompanied with a sound reason for choosing it. (Note: so far I
still haven't actually seen a sound reason...)
quoted
The above is just after a day of looking hard at kdbus.txt. I strongly
suspect I'd find a lot of other issues if I spent more time on kdbus.
If you find the time, please do! Any hints how a specific part of the
API could be done better, is highly appreciated. A lot of the more or
less recent changes were done due to reviews from glib developers.
Could you point me at those reviews (mailing list archives?)?
So, I'm thinking about things such as the following:
* The odd choice of ioctl() as the API mechanism for what should become
a key user-space API. (BTW, which other widely used IPC API ever
took the approach of ioctl() as the mechanism?)
* Weak justifications for unconventional API design choices such
as the 'kernel_flags' above.
* Thin documentation that doesn't provide nearly enough detail,
has no worked examples of the use of the APIs (when it should
contain a multitude of such examples), and has no rationale
for the API design choices [1].
* An API design that consists of 16 ioctl() requests and 20+
structures exchanged between user and kernel space being called
"simple". (Clearly it is not.)
Given a list of points like that, I worry that not nearly enough
thought has been put into design of the API, and certainly would be
very concerned to think that it might be merged into mainline
in the near future.
At this point, I think the onus is on the kdbus developers to
provide strong evidence that they have a good API design, one that
will well serve the needs of thousands of user-space programmers for
the next few decades. Such evidence would include at least:
* Detailed documentation that fully described all facets of the API
* A number of working, well documented example programs that start
(very) simple, and ramp up to demonstrate more complex pieces
of the API.
* Documented rationale for API design choices.
To date, much of that sort of evidence is lacking, and I worry that
the job of proper API design will be left to someone else, someone
who devises a user-space library that provides a suitable
abstraction on top of the current ioctl() API (but may be forced to
make design compromises because of design failings in the underlying
kernel API).
More help is always welcome!
That's what I'm trying to do at the moment...
Cheers,
Michael
[1] Elsewhere in this thread, you've said that "with bus-proxy and
systemd we have two huge users of kdbus that put a lot of
pressure on API design." I would have expected that the results
of that pressure should be captured in documentation of the
rationale for the API design. I haven't found any of that,
so far.
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
On Thu, Jan 22, 2015 at 11:18:50AM +0100, Michael Kerrisk (man-pages) wrote:
quoted
quoted
And that process seems to be frequent and ongoing even now. (And
it's to your great credit that the API/ABI breaks are clearly and honestly
marked in the kdbus.h changelog.) All of this lightens the burden of API
design for kernel developers, but I'm concerned that the long-term pain
for user-space developers who use an API which (in my estimation) may
come to be widely used will be enormous.
Yes, we've jointly reviewed the API details again until just recently to
unify structs and enums etc, and added fields to make the ioctls structs
more versatile for possible future additions. By that, we effectively
broke the ABI, but we did that because we know we can't do such things
again in the future.
But again - I don't see how this would be different when using syscalls
rather than ioctls to transport information between the driver and
userspace. Could you elaborate?
My suspicion is that not nearly enough thinking has yet been done about
the design of the API. That's based on these observations:
* Documentation that is, considering the size of the API, *way* too thin.
* Some parts of the API not documented at all (various kdbus_item blobs)
* ABI changes happening even quite recently
* API oddities such as the 'kernel_flags' fields. Why do I need to
be told what flags the kernel supports on *every* operation?
The above is just after a day of looking hard at kdbus.txt. I strongly
suspect I'd find a lot of other issues if I spent more time on kdbus.
"not enough thinking"?
We started working on kdbus 2 years ago this FOSDEM (in a few weeks.)
Before that we have been thinking about this for many years, learning
from the previous attempts to get this type of feature merged into the
kernel, talking with users about what they need for this, and soliciting
kernel developer's opinions on what type of API would be best for this
type of feature.
Since then we have done nothing but constantly revise the API. My first
mock ups were way too simple, and in discussing things with people much
more knowledgeable about D-Bus, they pointed out the problems, and we
iterated. And iterated. And iterated some more. We have worked with
just about every userspace libdbus developer group, including QtDbus
developers as well as glib developers. Now not all of them agreed with
some of our decisions in the implementation, which is fair enough, you
can't please everyone, but they _all_ agree that what we have now is the
proper way to implement this type of functionality and have reviewed the
features as being correct and compatible with their needs and users.
Those discussions have happened in emails, presentations, meetings, and
hackfests pretty much continuously for the past 2 years all around the
world.
We have stress-tested the api with both unit tests (which are included
here in the patch set) as well as a real-world implementation (sd-bus in
the systemd source repo.) That real-world implementation successfully
has been booting many of our daily machines for many months now.
Yes, the documentation can always be better, but please don't confuse
the lack of understanding how D-Bus works and its model with the lack of
understanding this kdbus implementation, the two are not comparable.
For some good primers on what D-Bus is, and the terminology it expects
see:
http://dbus.freedesktop.org/doc/dbus-tutorial.html
and also:
http://dbus.freedesktop.org/doc/dbus-faq.html#other-ipc
We are not going to put a basic "here is what D-Bus is and how to use
it" into the kernel tree, that is totally outside the scope here.
I suggest reading the tutorial above, and then going back and reading
the kdbus documentation provided. If you think we are lacking stuff on
the kdbus side, we will be glad to flush out any needed areas.
Also, Daniel has said he will work on a basic userspace "example"
library to show how to use this api, which might make the api a bit
easier to understand.
However, I personally don't think this "example code" is necessary at
all. We don't ask for this type of "simple examples" from other new
kernel apis we create and add to the kernel all the time. We require
there to be a user of the api, but not one that is so well documented
that others can write a from-scratch raw userspace replacement.
Specific examples of this are my previously mentioned ioctl users
(btrfs, mei, mic, openvlan, etc.), and the grand-daddy of all horrid
apis, DRM.
Users aren't going to be writing their own "raw kdbus" libraries. Or if
they are, they are going to start with one of the existing
implementations we have (the test examples and sd-bus, and I think there
is a native Go implementation somewhere as well.) Users are going to be
using those libraries to write their code, and to be honest, the user
api for sd-bus is a delight to use compared to the "old style" libdbus
interface as we have the benefit of 10 years of experience working with
D-Bus apis in the wild now to learn from past mistakes.
Back to the API. We have taken review comments on the previous postings
of the code and reworked the API, moving it from a character device to
be a filesystem, which ended up making things a lot easier in the end, a
good example of a review process that is working. Those changes are
a sign that our development review process works. People pointed out
problems in our character api that we hadn't thought about from the
kernel implementation side. And so we changed them and the code is
better and more robust because of it, a success story for our review
process.
Personally, I was the one that started down the character device node
path, so blame that design decision on me, not the other developers
here. And I was wrong with that, but moving from character to a
filesystem wasn't a huge change, the structures and interactions all
remained almost identical, as the logic behind the API is, in my
opinion, correct for the problem it is addressing.
The 37 different developers who have contributed to this code base are
quite talented and skilled and experienced in user and kernel apis,
having implemented many apis of their own that users rely on every day.
Yes, we all make design mistakes, and you might not agree with some of
them, that's fine. But it is flat out rude to say that we have not been
thinking about this, when I would guess that this is one of the largest
(in time and contributions) kernel development feature to be worked on
in the past few years.
And yes, I'm being very defensive, as I take this very seriously, so
please, don't so lightly dismiss us as not knowing what we are doing, as
frankly, we do.
Thanks for making it this far, I'll go back to technical discussions of
the API now, as that's what we should be doing, not casting aspirations
as to the aptitude of the people involved.
thanks,
greg k-h
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-26 14:43:03
Hi Greg,
First of all, I seem to have offended you. That was not my intention.
It's certainly not my intent to disparage you or your work (or for
that matter, the other kdbus developers). Insofar as any of the wordings
I've used suggested otherwise, I do apologize.
I'll comment on various points below, keeping it as technical as I can.
Then I have a couple of general questions at the end with the goal
of ensuring that my comments are not coming from a broken world view.
On 01/23/2015 04:54 PM, Greg Kroah-Hartman wrote:
On Thu, Jan 22, 2015 at 11:18:50AM +0100, Michael Kerrisk (man-pages) wrote:
quoted
quoted
quoted
And that process seems to be frequent and ongoing even now. (And
it's to your great credit that the API/ABI breaks are clearly and honestly
marked in the kdbus.h changelog.) All of this lightens the burden of API
design for kernel developers, but I'm concerned that the long-term pain
for user-space developers who use an API which (in my estimation) may
come to be widely used will be enormous.
Yes, we've jointly reviewed the API details again until just recently to
unify structs and enums etc, and added fields to make the ioctls structs
more versatile for possible future additions. By that, we effectively
broke the ABI, but we did that because we know we can't do such things
again in the future.
But again - I don't see how this would be different when using syscalls
rather than ioctls to transport information between the driver and
userspace. Could you elaborate?
My suspicion is that not nearly enough thinking has yet been done about
the design of the API. That's based on these observations:
* Documentation that is, considering the size of the API, *way* too thin.
* Some parts of the API not documented at all (various kdbus_item blobs)
* ABI changes happening even quite recently
* API oddities such as the 'kernel_flags' fields. Why do I need to
be told what flags the kernel supports on *every* operation?
The above is just after a day of looking hard at kdbus.txt. I strongly
suspect I'd find a lot of other issues if I spent more time on kdbus.
"not enough thinking"?
We started working on kdbus 2 years ago this FOSDEM (in a few weeks.)
Before that we have been thinking about this for many years, learning
from the previous attempts to get this type of feature merged into the
kernel, talking with users about what they need for this, and soliciting
kernel developer's opinions on what type of API would be best for this
type of feature.
Since then we have done nothing but constantly revise the API. My first
mock ups were way too simple, and in discussing things with people much
more knowledgeable about D-Bus, they pointed out the problems, and we
iterated. And iterated. And iterated some more. We have worked with
just about every userspace libdbus developer group, including QtDbus
developers as well as glib developers. Now not all of them agreed with
some of our decisions in the implementation, which is fair enough, you
can't please everyone, but they _all_ agree that what we have now is the
proper way to implement this type of functionality and have reviewed the
features as being correct and compatible with their needs and users.
Those discussions have happened in emails, presentations, meetings, and
hackfests pretty much continuously for the past 2 years all around the
world.
We have stress-tested the api with both unit tests (which are included
here in the patch set) as well as a real-world implementation (sd-bus in
the systemd source repo.) That real-world implementation successfully
has been booting many of our daily machines for many months now.
Notwithstanding that I don't see how a unit test stress tests an API
*design*, I've no reason to doubt that kdbus works. But that's not the
point of my concern. I worry how usable this API is going to be for the
world at large.
Yes, the documentation can always be better, but please don't confuse
the lack of understanding how D-Bus works and its model with the lack of
understanding this kdbus implementation, the two are not comparable.
For some good primers on what D-Bus is, and the terminology it expects
see:
http://dbus.freedesktop.org/doc/dbus-tutorial.html
and also:
http://dbus.freedesktop.org/doc/dbus-faq.html#other-ipc
We are not going to put a basic "here is what D-Bus is and how to use
it" into the kernel tree, that is totally outside the scope here.
I didn't expect that you should do that. But it does touch on a general
question that I'll leave to the end of this mail.
I suggest reading the tutorial above, and then going back and reading
the kdbus documentation provided. If you think we are lacking stuff on
the kdbus side, we will be glad to flush out any needed areas.
Also, Daniel has said he will work on a basic userspace "example"
library to show how to use this api, which might make the api a bit
easier to understand.
However, I personally don't think this "example code" is necessary at
all. We don't ask for this type of "simple examples" from other new
kernel apis we create and add to the kernel all the time. We require
there to be a user of the api, but not one that is so well documented
that others can write a from-scratch raw userspace replacement.
What you've just summarized there is how low a bar we've historically
set in API design. Thus, the API is littered with (for example) dozens
of system calls that were insufficiently well thought out in their
design, and have subsequently been superseded by replacements that
fixed the design mistakes. One of the cause of that problems is the
targeting of "*a* user of the API"--general purpose APIs need to be
considered from the point of view of multiple potentially different
use cases. And I'm certainly not talking about being able to reimplement
the API, But, the API is a contract, and it needs to be well understood
by its creators and consumers in order that they can assess and use that
API. Extensive Documentation generally is the best way to do that.
And, anyway, I had understood that there was a rough consensus that we do
want to see more tests/examples and documentation happening in the future.
Certainly, the number kernel developers who are taking a shot at
writing man pages these days is refreshing. (We are 7/7 for man-page
documented system calls in the 3.17-3.19 frame. That's a trend I've done
my best to encourage, and hope to see continue in the future.) And now
we have kselftest, in part thanks to your good efforts.
Specific examples of this are my previously mentioned ioctl users
(btrfs, mei, mic, openvlan, etc.), and the grand-daddy of all horrid
apis, DRM.
You've made this comparison a number of times, but I think it misses
a crucial point. Those examples are all(?) domain-specific APIs with
relatively few users in terms of user-space applications, whereas,
IIUC, kdbus is intended to be an IPC mechanism that can be employed
by user-space applications in a general-purpose fashion, and upon
which potentially a multitude of different applications might be built.
That's why I think the decision to use an ioctl()-based interface
needs to be considered from a (very) skeptical point of view: no other
general-purpose IPC mechanism employs such an approach. (Again, see
one of my general questions at the end of this mail.)
Users aren't going to be writing their own "raw kdbus" libraries. Or if
they are, they are going to start with one of the existing
implementations we have (the test examples and sd-bus, and I think there
is a native Go implementation somewhere as well.) Users are going to be
using those libraries to write their code, and to be honest, the user
api for sd-bus is a delight to use compared to the "old style" libdbus
interface as we have the benefit of 10 years of experience working with
D-Bus apis in the wild now to learn from past mistakes.
Back to the API. We have taken review comments on the previous postings
of the code and reworked the API, moving it from a character device to
be a filesystem, which ended up making things a lot easier in the end, a
good example of a review process that is working. Those changes are
a sign that our development review process works. People pointed out
problems in our character api that we hadn't thought about from the
kernel implementation side. And so we changed them and the code is
better and more robust because of it, a success story for our review
process.
Personally, I was the one that started down the character device node
path, so blame that design decision on me, not the other developers
here. And I was wrong with that, but moving from character to a
filesystem wasn't a huge change, the structures and interactions all
remained almost identical, as the logic behind the API is, in my
opinion, correct for the problem it is addressing.
The 37 different developers who have contributed to this code base are
quite talented and skilled and experienced in user and kernel apis,
having implemented many apis of their own that users rely on every day.
Yes, but I am not sure that the 15 developers who made each made 1 commit
(out of 2816 to date) would have done much work on the API. And probably
the same is true for the 9 more who made just 2 or 3 commits. As one
would expect, the great deal of the good work has been done by a small
core: just shy of 95% commits by the top 5 committers.
Yes, we all make design mistakes, and you might not agree with some of
them, that's fine. But it is flat out rude to say that we have not been
thinking about this, when I would guess that this is one of the largest
(in time and contributions) kernel development feature to be worked on
in the past few years.
And yes, I'm being very defensive, as I take this very seriously, so
please, don't so lightly dismiss us as not knowing what we are doing, as
frankly, we do.
Greg, I did not say you hadn't been thinking about this [API design].
(But I acknowledge my words could have been better chosen.) However,
API design is hard to get right, and causes endless pain when it's wrong.
And by now I've been watching long enough to know that the mistakes
are frequent. Even Davide Libenzi, who once upon a time was one of our
more prolific and talented creators of APIs made mistakes. Thus, for
example, as we speak, the third iteration of epoll_wait() is in
development (epoll_wait() ==> epoll_pwait() ==> epoll_pwait1()).
And epoll is an API that is significantly simpler than kdbus.
In my observation, a good API design is a well documented API design.
Otherwise, it's virtually impossible to think thoroughly about the API,
and that is especially true as the API gets larger. (And AFAICS, the
kdbus API is bigger than, for example, the epoll API by an order of
magnitude, or so.) Part of that documentation also should include some
concrete examples of the use of the API. Again because it helps people
to think about and assess the API. (This is especially the case for
the fresh minds that explore the new API for the first time without
having the preconceptions that are almost inevitable for the creators
of the API.)
(BTW, I'm not ignoring your contents about the D-Bus spec above. But
kdbus is a free-standing API, IIUC, and as such, it should be assessed
on its own.)
I would summarize your statement above, as "trust us, we know what
we're doing". With respect, my default position is not to trust. It's
nothing personal: API design is hard, and mistakes are too often made.
What I want to say in return is "trust us", where "us" is used very
inclusively to mean: the kernel maintainers, and for that matter
user-space programmers, who need enough information that we can make a
well-informed assessment of the merits of an API that will need to be
supported forever and may have a multitude of different users. In
my assessment, the current information is far from sufficient, and
it's a considerable risk to merge the API lacking such information.
Thanks for making it this far, I'll go back to technical discussions of
the API now, as that's what we should be doing, not casting aspirations
as to the aptitude of the people involved.
Greg, that comment, with its implication that I am not concerned about
technical matters, but rather with something more malicious was quite
uncalled for.
But, I am happy to return to technical matters. And I think it best to
start with a couple of fundamental questions, since some of the comments
I've seen to date from different kdbus developers seem to conflict:
1. Is this intended to be a general purpose API that might see a
multitude of different users, or is it thought of as an API designed
to support a few specific users such as D-Bus and maybe a handful of
others? I had thought the former, but when you point me in the
direction of the D-Bus spec, I start to have doubts.
2. Is the API to be invoked directly by applications or is intended to
be used only behind specific libraries? You seem to be saying that
the latter is the case (here, I'm referring to your comment above
about sd-bus). However, when I asked David Herrmann a similar
question I got this responser:
"kdbus is in no way bound to systemd. There are ongoing efforts
to port glib and qt to kdbus natively. The API is pretty simple
and I don't see how a libkdbus would simplify things. In fact,
even our tests only have slim wrappers around the ioctls to
simplify error-handling in test-scenarios."
To me, that implies that users will employ the raw kernel API.
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
From: Tom Gundersen <hidden> Date: 2015-01-26 15:27:16
Hi Michael,
On Mon, Jan 26, 2015 at 3:42 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
2. Is the API to be invoked directly by applications or is intended to
be used only behind specific libraries? You seem to be saying that
the latter is the case (here, I'm referring to your comment above
about sd-bus). However, when I asked David Herrmann a similar
question I got this responser:
"kdbus is in no way bound to systemd. There are ongoing efforts
to port glib and qt to kdbus natively. The API is pretty simple
and I don't see how a libkdbus would simplify things. In fact,
even our tests only have slim wrappers around the ioctls to
simplify error-handling in test-scenarios."
To me, that implies that users will employ the raw kernel API.
The way I read this is that there will (probably) be a handful of
users, namely the existing dbus libraries: libdus, sd-bus, glib, Qt,
ell, and maybe a few others. However, third-party developers will not
know/care about the details of kdbus, they'll just be coding against
the dbus libraries as before (might be minor changes, but they
certainly won't need to know anything about the kernel API). Similarly
to how userspace developers now code against their libc of choice,
rather than use kernel syscalls directly.
HTH,
Tom
From: christoph Hellwig <hch@infradead.org> Date: 2015-01-26 16:45:37
On Mon, Jan 26, 2015 at 04:26:53PM +0100, Tom Gundersen wrote:
The way I read this is that there will (probably) be a handful of
users, namely the existing dbus libraries: libdus, sd-bus, glib, Qt,
ell, and maybe a few others. However, third-party developers will not
know/care about the details of kdbus, they'll just be coding against
the dbus libraries as before (might be minor changes, but they
certainly won't need to know anything about the kernel API). Similarly
to how userspace developers now code against their libc of choice,
rather than use kernel syscalls directly.
Which means we do need proper man pages and detailed documentation for
it, just like syscalls for syscalls which just happened to be used by
a few libcs. I suspect it really should be implemented as
syscalls anyway, but we can leave that argument aside from now. Good
documentation certainly helps with making that decision in an educated
way.
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-26 16:45:47
On 01/26/2015 04:26 PM, Tom Gundersen wrote:
Hi Michael,
On Mon, Jan 26, 2015 at 3:42 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
2. Is the API to be invoked directly by applications or is intended to
be used only behind specific libraries? You seem to be saying that
the latter is the case (here, I'm referring to your comment above
about sd-bus). However, when I asked David Herrmann a similar
question I got this responser:
"kdbus is in no way bound to systemd. There are ongoing efforts
to port glib and qt to kdbus natively. The API is pretty simple
and I don't see how a libkdbus would simplify things. In fact,
even our tests only have slim wrappers around the ioctls to
simplify error-handling in test-scenarios."
To me, that implies that users will employ the raw kernel API.
The way I read this is that there will (probably) be a handful of
users, namely the existing dbus libraries: libdus, sd-bus, glib, Qt,
ell, and maybe a few others. However, third-party developers will not
know/care about the details of kdbus, they'll just be coding against
the dbus libraries as before (might be minor changes, but they
certainly won't need to know anything about the kernel API). Similarly
to how userspace developers now code against their libc of choice,
rather than use kernel syscalls directly.
Thanks, Tom, for the input. I'm still confused though, since elsewhere
in this thread David Herrmann said in response to a question of mine:
I think we can agree that we want it to be generically useful,
like other ipc mechanisms, including UDS and netlink.
Again, that sounds to me like the vision is not "a handful of users".
Hopefully Greg and David can clarify.
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
From: David Herrmann <hidden> Date: 2015-01-27 15:23:34
Hi
On Mon, Jan 26, 2015 at 5:45 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
On 01/26/2015 04:26 PM, Tom Gundersen wrote:
quoted
On Mon, Jan 26, 2015 at 3:42 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
2. Is the API to be invoked directly by applications or is intended to
be used only behind specific libraries? You seem to be saying that
the latter is the case (here, I'm referring to your comment above
about sd-bus). However, when I asked David Herrmann a similar
question I got this responser:
"kdbus is in no way bound to systemd. There are ongoing efforts
to port glib and qt to kdbus natively. The API is pretty simple
and I don't see how a libkdbus would simplify things. In fact,
even our tests only have slim wrappers around the ioctls to
simplify error-handling in test-scenarios."
To me, that implies that users will employ the raw kernel API.
The way I read this is that there will (probably) be a handful of
users, namely the existing dbus libraries: libdus, sd-bus, glib, Qt,
ell, and maybe a few others. However, third-party developers will not
know/care about the details of kdbus, they'll just be coding against
the dbus libraries as before (might be minor changes, but they
certainly won't need to know anything about the kernel API). Similarly
to how userspace developers now code against their libc of choice,
rather than use kernel syscalls directly.
Thanks, Tom, for the input. I'm still confused though, since elsewhere
in this thread David Herrmann said in response to a question of mine:
I think we can agree that we want it to be generically useful,
like other ipc mechanisms, including UDS and netlink.
Again, that sounds to me like the vision is not "a handful of users".
Hopefully Greg and David can clarify.
I only expect a handful of users to call the ioctls directly. The
libraries that implement the payload-marshaling, in particular. It's a
similar situation with netlink.
Thanks
David
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-27 17:53:52
On 01/27/2015 04:23 PM, David Herrmann wrote:
Hi
On Mon, Jan 26, 2015 at 5:45 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
On 01/26/2015 04:26 PM, Tom Gundersen wrote:
quoted
On Mon, Jan 26, 2015 at 3:42 PM, Michael Kerrisk (man-pages)
[off-list ref] wrote:
quoted
2. Is the API to be invoked directly by applications or is intended to
be used only behind specific libraries? You seem to be saying that
the latter is the case (here, I'm referring to your comment above
about sd-bus). However, when I asked David Herrmann a similar
question I got this responser:
"kdbus is in no way bound to systemd. There are ongoing efforts
to port glib and qt to kdbus natively. The API is pretty simple
and I don't see how a libkdbus would simplify things. In fact,
even our tests only have slim wrappers around the ioctls to
simplify error-handling in test-scenarios."
To me, that implies that users will employ the raw kernel API.
The way I read this is that there will (probably) be a handful of
users, namely the existing dbus libraries: libdus, sd-bus, glib, Qt,
ell, and maybe a few others. However, third-party developers will not
know/care about the details of kdbus, they'll just be coding against
the dbus libraries as before (might be minor changes, but they
certainly won't need to know anything about the kernel API). Similarly
to how userspace developers now code against their libc of choice,
rather than use kernel syscalls directly.
Thanks, Tom, for the input. I'm still confused though, since elsewhere
in this thread David Herrmann said in response to a question of mine:
I think we can agree that we want it to be generically useful,
like other ipc mechanisms, including UDS and netlink.
Again, that sounds to me like the vision is not "a handful of users".
Hopefully Greg and David can clarify.
I only expect a handful of users to call the ioctls directly. The
libraries that implement the payload-marshaling, in particular. It's a
similar situation with netlink.
Thanks, David, for the clarification. I think it would have been helpful
to have that more clearly stated up front, especially as some comments
in this thread, such as the above, could be interpreted to mean quite
the opposite. Can I suggest that some text on this point be added to
kdbus.txt?
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
From: Daniel Mack <daniel@zonque.org> Date: 2015-01-27 18:14:17
Hi Michael,
On 01/27/2015 06:53 PM, Michael Kerrisk (man-pages) wrote:
On 01/27/2015 04:23 PM, David Herrmann wrote:
quoted
I only expect a handful of users to call the ioctls directly. The
libraries that implement the payload-marshaling, in particular. It's a
similar situation with netlink.
Thanks, David, for the clarification. I think it would have been helpful
to have that more clearly stated up front, especially as some comments
in this thread, such as the above, could be interpreted to mean quite
the opposite. Can I suggest that some text on this point be added to
kdbus.txt?
We're currently working on an a set of comprehensive man pages to
document all the commands in the API, along with every struct, enum etc.
We do that so that developers are able to actually understand every
detail of the API, even though most people - as David explained - will
not use that interface directly in the first place but let one of the
high-level libraries help them integrate D-Bus functionality into their
applications.
If you want, have a look at the upstream repository for a preliminary
version of the new docs.
Thanks,
Daniel
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-29 02:57:31
Hello Daniel,
On 01/27/2015 07:14 PM, Daniel Mack wrote:
Hi Michael,
On 01/27/2015 06:53 PM, Michael Kerrisk (man-pages) wrote:
quoted
On 01/27/2015 04:23 PM, David Herrmann wrote:
quoted
quoted
I only expect a handful of users to call the ioctls directly. The
libraries that implement the payload-marshaling, in particular. It's a
similar situation with netlink.
Thanks, David, for the clarification. I think it would have been helpful
to have that more clearly stated up front, especially as some comments
in this thread, such as the above, could be interpreted to mean quite
the opposite. Can I suggest that some text on this point be added to
kdbus.txt?
We're currently working on an a set of comprehensive man pages to
document all the commands in the API, along with every struct, enum etc.
We do that so that developers are able to actually understand every
detail of the API, even though most people - as David explained - will
not use that interface directly in the first place but let one of the
high-level libraries help them integrate D-Bus functionality into their
applications.
(I suggest that some text about this text go into the kdbus(7) page.)
If you want, have a look at the upstream repository for a preliminary
version of the new docs.
That's at https://code.google.com/p/d-bus/ , right? This looks like a
good direction to go in. Thanks for tackling that.
I hope to take a longer look sometime soon, but a few general conventions
for man-pages that you might want to consider following:
* When listing errors, I think you should change your
language/formatting somewhat. Examples here from kdbus.endpoint.7:
(1) The man page says
RETURN VALUE
On success, all mentioned ioctl commands return 0.
Better to write this from a user-space point of view:
RETURN VALUE
On success, all mentioned ioctl commands return 0; on
error, -1 is returned, and errno is set to indicate
the error.
(2) I would change the wording in the ERRORS sections from
"may return the following errors"
to
"may fail with the following errors"
(3) When listing the errors, drop the minus signs; that's not
what user-space sees. They see a positive value in errno.
(4) The usual formatting convention for constants, including
error constants in man pages is boldface, rather than
underline/emphasis.
(5) Insofar as it's possible, it would be good to make all
pages format nicely within 80 columns. Some of the literal
text and ASCII art could, I think, be narrowed.
Thanks,
Michael
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-20 13:59:10
On 01/16/2015 08:16 PM, Greg Kroah-Hartman wrote:
From: Daniel Mack <redacted>
kdbus is a system for low-latency, low-overhead, easy to use
interprocess communication (IPC).
The interface to all functions in this driver is implemented via ioctls
on files exposed through a filesystem called 'kdbusfs'. The default
mount point of kdbusfs is /sys/fs/kdbus. This patch adds detailed
documentation about the kernel level API design.
And now the details feedback.
Please note that for the various points I raise below, even in
cases where I don't suggest/request a fix, the fact that I've
needed to answer a question probably suggests a deficiency in
the documentation that probably needs to be remedied.
Many of my comments below are wording and typo fixes. While
these may sem trivial, the existence of various wording problems
and typos is a significant distraction, especially while trying
to grok a document of this size.
I've marked one or two notable questions about the API with "###".
@@ -0,0 +1,2107 @@+D-Bus is a system for powerful, easy to use interprocess communication (IPC).++The focus of this document is an overview of the low-level, native kernel D-Bus+transport called kdbus. Kdbus exposes its functionality via files in a+filesystem called 'kdbusfs'. All communication between processes takes place+via ioctls on files exposed through the mount point of a kdbusfs. The default+mount point of kdbusfs is /sys/fs/kdbus.++Please note that kdbus was designed as transport layer for D-Bus, but is in no+way limited, nor controlled by the D-Bus protocol specification. The D-Bus+protocol is one possible application layer on top of kdbus.++For the general D-Bus protocol specification, the payload format, the+marshaling, and the communication semantics, please refer to:+ http://dbus.freedesktop.org/doc/dbus-specification.html++For a kdbus specific userspace library implementation please refer to:+ http://cgit.freedesktop.org/systemd/systemd/tree/src/systemd/sd-bus.h++Articles about D-Bus and kdbus:+ http://lwn.net/Articles/580194/+++1. Terminology+===============================================================================++ Domain:+ A domain is created each time a kdbusfs is mounted. Each process that is+ capable to mount a new instance of a kdbusfs will have its own kdbus
s/is capable to mount/mounts/
+ hierarchy. Each domain (ie, each mount point) offers its own "control"
+ file to create new buses. Domains have no connection to each other and
+ cannot see nor talk to each other. See section 5 for more details.
Smoother would be:
s/cannot see nor talk/can neither see nor talk/
+
+ Bus:
+ A bus is a named object inside a domain. Clients exchange messages
+ over a bus. Multiple buses themselves have no connection to each other;
+ messages can only be exchanged on the same bus. The default endpoint of
+ a bus, where clients establish the connection to, is the "bus" file
Maybe:
where clients establish the connection to
==>
to which clients establish connections
?
+ /sys/fs/kdbus/<bus name>/bus.
+ Common operating system setups create one "system bus" per system, and one
+ "user bus" for every logged-in user. Applications or services may create
At the kdbus level is there any difference between such system and user buses?
If not, it would perhaps be good to insert a parenthetical aside to say
that.
+ their own private buses. See section 5 for more details.
+
+ Endpoint:
+ An endpoint provides a file to talk to a bus. Opening an endpoint
+ creates a new connection to the bus to which the endpoint belongs. All
+ endpoints have unique names and are accessible as files underneath the
+ directory of a bus, e.g., /sys/fs/kdbus/<bus>/<endpoint>
+ Every bus has a default endpoint called "bus". A bus can optionally offer
+ additional endpoints with custom names to provide restricted access to the
+ bus. Custom endpoints carry additional policy which can be used to create
+ sandboxes with locked-down, limited, filtered access to a bus. See
+ section 5 for more details.
+
+ Connection:
+ A connection to a bus is created by opening an endpoint file of a bus and
+ becoming an active client with the HELLO exchange. Every ordinary client
+ connection has a unique identifier on the bus and can address messages to
+ every other connection on the same bus by using the peer's connection id
+ as the destination. See section 6 for more details.
+
+ Pool:
+ Each connection allocates a piece of shmem-backed memory that is used
+ to receive messages and answers to ioctl commands from the kernel. It is
+ never used to send anything to the kernel. In order to access that memory,
+ userspace must mmap() it into its address space.
s/userspace/a userspace application/
+ See section 12 for more details.
+
+ Well-known Name:
+ A connection can, in addition to its implicit unique connection id, request
+ the ownership of a textual well-known name. Well-known names are noted in
+ reverse-domain notation, such as com.example.service1. Connections offering
+ a service on a bus are usually reached by its well-known name. The analogy
Noun/pronoun number disagreement at "Connections... its".
==>
A connection that offers a service on a bus is usually reached by its
well-known name.
+ of connection id and well-known name is an IP address and a DNS name
Doing s/id/ID/ throughout the doc would help readability. (The doc
already uses "ID" in some places, so consistency is a further argument
for this change.)
+ associated with that address.
+
+ Message:
+ Connections can exchange messages with other connections by addressing
+ the peers with their connection id or well-known name. A message consists
+ of a message header with kernel-specific information on how to route the
What does "kernel-specific" mean here? Something needs explaining (or removing).
+ message, and the message payload, which is a logical byte stream of
+ arbitrary size. Messages can carry additional file descriptors to be passed
So, this is FD passing like UNIX domain sockets? If yes, it would be helpful
here to mention that analogy.
+ from one connection to another. Every connection can specify which set of
+ metadata the kernel should attach to the message when it is delivered
+ to the receiving connection. Metadata contains information like: system
+ timestamps, uid, gid, tid, proc-starttime, well-known-names, process comm,
s/well-known-names/well-known names/
See the note on "ID" above. I think UID, GID, TID throughout would help
readability.
+ process exe, process argv, cgroup, capabilities, seclabel, audit session,
+ loginuid and the connection's human-readable name.
+ See section 7 and 13 for more details.
+
+ Item:
+ The API of kdbus implements a notion of items, submitted through and
s/a notion/the notion/
+ returned by most ioctls, and stored inside data structures in the
+ connection's pool. See section 4 for more details.
+
+ Broadcast and Match:
+ Broadcast messages are potentially sent to all connections of a bus. By
+ default, the connections will not actually receive any of the sent
+ broadcast messages; only after installing a match for specific message
+ properties, a broadcast message passes this filter.
"Filter" suddenly gets mentioned here without previously being defined. I suspect
the last piece should read more like:
a connection will receive a broadcast message only after it installs a filter
that matches the specific message properties of the broadcast message.
+ See section 10 for more details.
+
+ Policy:
+ A policy is a set of rules that define which connections can see, talk to,
+ or register a well-know name on the bus. A policy is attached to buses and
s/know/know/
+ custom endpoints, and modified by policy holder connections or owners of
+ custom endpoints. See section 11 for more details.
+ See section 11 for more details.
Repeated last sentence.
+ Privileged bus users:
+ A user connecting to the bus is considered privileged if it is either the
+ creator of the bus, or if it has the CAP_IPC_OWNER capability flag set.
+
+
+2. Control Files Layout
+===============================================================================
+
+The kdbus interface is exposed through files in its kdbusfs mount point
+(defaults to /sys/fs/kdbus):
+
+ /sys/fs/kdbus (mount point of kdbusfs)
+ |-- control (domain control-file)
+ |-- 0-system (bus of user uid=0)
+ | |-- bus (default endpoint of bus '0-system')
+ | `-- ep.apache (custom endpoint of bus '0-system')
+ |-- 1000-user (bus of user uid=1000)
+ | `-- bus (default endpoint of bus '1000-user')
+ `-- 2702-user (bus of user uid=2702)
+ |-- bus (default endpoint of bus '2702-user')
+ `-- ep.app (custom endpoint of bus '2702-user')
+
+
+3. Data Structures and flags
+===============================================================================
+
+3.1 Data structures and interconnections
+----------------------------------------
+
+ +--------------------------------------------------------------------------+
+ | Domain (Mount Point) |
+ | /sys/fs/kdbus/control |
+ | +----------------------------------------------------------------------+ |
+ | | Bus (System Bus) | |
+ | | /sys/fs/kdbus/0-system/ | |
+ | | +-------------------------------+ +--------------------------------+ | |
+ | | | Endpoint | | Endpoint | | |
+ | | | /sys/fs/kdbus/0-system/bus | | /sys/fs/kdbus/0-system/ep.app | | |
+ | | +-------------------------------+ +--------------------------------+ | |
+ | | +--------------+ +--------------+ +--------------+ +---------------+ | |
+ | | | Connection | | Connection | | Connection | | Connection | | |
+ | | | :1.22 | | :1.25 | | :1.55 | | :1.81 | | |
+ | | +--------------+ +--------------+ +--------------+ +---------------+ | |
+ | +----------------------------------------------------------------------+ |
+ | |
+ | +----------------------------------------------------------------------+ |
+ | | Bus (User Bus for UID 2702) | |
+ | | /sys/fs/kdbus/2702-user/ | |
+ | | +-------------------------------+ +--------------------------------+ | |
+ | | | Endpoint | | Endpoint | | |
+ | | | /sys/fs/kdbus/2702-user/bus | | /sys/fs/kdbus/2702-user/ep.app | | |
+ | | +-------------------------------+ +--------------------------------+ | |
+ | | +--------------+ +--------------+ +--------------+ +---------------+ | |
+ | | | Connection | | Connection | | Connection | | Connection | | |
+ | | | :1.22 | | :1.25 | | :1.55 | | :1.81 | | |
+ | | +--------------+ +--------------+ +--------------------------------+ | |
+ | +----------------------------------------------------------------------+ |
+ +--------------------------------------------------------------------------+
+
+The above description uses the D-Bus notation of unique connection names that
+adds a ":1." prefix to the connection's unique ID. kdbus itself doesn't
+use that notation, neither internally nor externally. However, libraries and
+other userspace code that aims for compatibility to D-Bus might.
s/compatibility to/compatibility with/
+
+3.2 Flags
+---------
+
+All ioctls used in the communication with the driver contain three 64-bit
+fields: 'flags', 'kernel_flags' and 'return_flags'. All of them are specific
+to the ioctl used.
+
+In 'flags', the behavior of the command can be tweaked. All bits that are not
+recognized by the kernel in this field are rejected, and the ioctl fails with
+-EINVAL.
+
+In 'kernel_flags', the kernel driver writes back the mask of supported bits
+upon each call, and sets the KDBUS_FLAGS_KERNEL bit. This is a way to probe
+possible kernel features and make userspace code forward and backward
+compatible.
So, is "kernel_flags' a bounding superset of what the caller may specify
in 'flags'? If yes, please make that clearer.
+
+In 'return_flags', the kernel can return results of the command, in addition
+to the actual return value. This is mostly to inform userspace about non-fatal
+conditions that occurred during the execution of the command.
+
+
+4. Items
+===============================================================================
+
+To flexibly augment transport structures, data blobs of type struct kdbus_item
+can be attached to the structs passed into the ioctls. Some ioctls make items
+of certain types mandatory, others are optional. Unsupported items will cause
+the ioctl to fail -EINVAL.
+
+The total size of an item is variable and is in some cases defined by the item
+type. In other cases, they can be of arbitrary length (for instance, a string).
+
+Items are also used for information stored in a connection's pool, such as
+received messages, name lists or requested connection or bus owner information.
+
+Whenever items are used as part of the kdbus kernel API, they are embedded in
+structs that have an overall size of their own, so there can be multiple items
"have an overall size of their own" as hard to grok. I think you mean
... are embedded insides structs that themselves include a size field
containing the overall size of the structure. This allows multiple
items per ioctl.
+per ioctl.
+
+The kernel expects all items to be aligned to 8-byte boundaries. Unaligned
+items or such that are unsupported by the ioctl are rejected.
s/such/items/?
(Or otherwise replace "such" with whatever it actually means.)
+A simple iterator in userspace would iterate over the items until the items
+have reached the embedding structure's overall size. An example implementation
+of such an iterator can be found in tools/testing/selftests/kdbus/kdbus-util.h.
+
+
+5. Creation of new domains, buses and endpoints
+===============================================================================
+
+
+5.1 Domains
+-----------
+
+A domain is a container of buses. Domains themselves do not provide any IPC
+functionality. Their sole purpose is to manage buses allocated in their
+domain. Each time kdbusfs is mounted, a new kdbus domain is created, with its
+own 'control' file. The lifetime of the domain ends once the user has unmounted
+the kdbusfs. If you mount kdbusfs multiple times, each will have its own kdbus
+domain internally.
What does that last sentence mean? Somehow it needs to be reworded to better
convey whatever sense it is trying to convey.
Operations performed on one domain do not affect any
+other domain.
+
+The full kdbusfs hierarchy, any sub-directory, or file can be bind-mounted to
+an external mount point and will remain fully functional. The kdbus domain and
+any linked resources stay available until the original mount and all subsequent
+bind-mounts have been unmounted.
+
+During creation, domains pin the user-namespace of the creator and use
+it as controlling user-namespace for this domain. Any user accounting is done
s/as/as the/
+relative to that user-namespace.
+
+Newly created kdbus domains do not have any bus pre-created. The only resource
+available is a 'control' file, which is used to manage kdbus domains.
+Currently, 'control' files are exclusively used to create buses via
s/exclusively used/used exclusively/
+KDBUS_CMD_BUS_MAKE, but further ioctls might be added in the future.
+
+
+5.2 Buses
+---------
+
+A bus is a shared resource between connections to transmit messages. Each bus
==> A bus is a resource that is shared between connections in order to
transmit messages
+is independent and operations on the bus will not have any effect on other
+buses. A bus is a management entity, that controls the addresses of its
s/,//
+connections, their policies and message transactions performed via this bus.
+
+Each bus is bound to the domain it was created on. It has a custom name that is
+unique across all buses of a domain. In kdbusfs, a bus is presented as a
+directory. No operations can be performed on the bus itself, instead you need
s/,/;/
+to perform those on an endpoint associated with the bus. Endpoints are
s/those/operations/
+accessible as files underneath the bus directory. A default endpoint called
+"bus" is provided on each bus.
+
+Bus names may be chosen freely except for one restriction: the name
+must be prefixed with the numeric UID of the creator and a dash. This
s/UID/effective UID/
(I assume it's the effective UID...)
+is required to avoid namespace clashes between different users. When
+creating a bus the name must be passed in properly formatted, or the
the name must be passed in properly formatted
==>
that name that is passed in must be properly formatted
+kernel will refuse creation of the bus. Example: "1047-foobar" is an
+OK name for a bus registered by a user with UID 1047. However,
s/OK/acceptable/
+"1024-foobar" is not, and neither is "foobar".
+The UID must be provided in the user-namespace of the parent domain.
+
+To create a new bus, you need to open the control file of a domain and run the
s/run/employ/ (One doesn't "run" a system call.)
+KDBUS_CMD_BUS_MAKE ioctl. The control file descriptor that was used to issue
+KDBUS_CMD_BUS_MAKE must not have been used for any other control-ioctl before
Maybe better:
have been used for any other control-ioctl before
==>
previously have been used for any other control-ioctl
+and needs to be kept open for the entire life-time of the created bus. Closing
s/needs/must/ (just reads smoother)
+it will immediately cleanup the entire bus and all its associated resources and
+endpoints. Every control file descriptor can only be used to create a single
+new bus; from that point on, it is not used for any further communication until
+the final close().
+
+Each bus will generate a random, 128-bit UUID upon creation. It will be
/It/This UUID/
+returned to creators of connections through kdbus_cmd_hello.id128 and can
+be used by userspace to uniquely identify buses, even across different machines
+or containers. The UUID will have its variant bits set to 'DCE', and denote
+version 4 (random).
I find that last sentence rather difficult to grasp. I think more detail
needs to be added.
+When creating buses, a variable list of items that must be passed in
+the items array is expected otherwise bus creation will fail.
What does "a variable list of items that must be passed in
the items array" mean? Something needs fixing, I think.
+
+5.3 Endpoints
+-------------
+
+Endpoints are entry points to a bus. By default, each bus has a default
+endpoint called 'bus'. The bus owner has the ability to create custom
+endpoints with specific names, permissions, and policy databases (see below).
+An endpoint is presented as file underneath the directory of the parent bus.
+
+To create a custom endpoint, open the default endpoint ('bus') and use the
+KDBUS_CMD_ENDPOINT_MAKE ioctl with "struct kdbus_cmd_make". Custom endpoints
+always have a policy database that, by default, forbids any operation. You have
+to explicitly install policy entries to allow any operation on this endpoint.
+Once KDBUS_CMD_ENDPOINT_MAKE succeeded, this file descriptor will manage the
+newly created endpoint resource. It cannot be used to manage further resources.
+
+Endpoint names may be chosen freely except for one restriction: the name
+must be prefixed with the numeric UID of the creator and a dash. This
s/UID/effective UID/
+is required to avoid namespace clashes between different users. When
+creating an endpoint the name must be passed in properly formatted, or the
creating an endpoint the name must be passed in properly formatted
==>
creating an endpoint, the name that is passed in must be properly formatted
+kernel will refuse creation of the endpoint. Example: "1047-foobar" is an
+OK name for an endpoint registered by a user with UID 1047. However,
s/OK/acceptable/
+"1024-foobar" is not, and neither is "foobar".
Because this text reads almost exactly as the bus text in 5.2, I did
a double take here. I suggest making the text more distinct in each case.
So, for example:
Example: "1047-my-endpoint" is an OK name for an endpoint registered
by a user with UID 1047. However, "1024-my-endpoint" is not, and
neither is "my-endpoint".
(And you could do similar in the bus text in section 5.2.)
+The UID must be provided in the user-namespace of the parent domain.
+
+To create connections to a bus, you use KDBUS_CMD_HELLO. See section 6 for
+details. Note that once KDBUS_CMD_HELLO succeeded, this file descriptor manages
Note that once KDBUS_CMD_HELLO succeeded,
==>
Note that after a successful KDBUS_CMD_HELLO,
+the newly created connection resource. It cannot be used to manage further
+resources.
+
+
+5.4 Creating buses and endpoints
+--------------------------------
+
+KDBUS_CMD_BUS_MAKE, and KDBUS_CMD_ENDPOINT_MAKE take a
s/,//
+struct kdbus_cmd_make argument.
+
+struct kdbus_cmd_make {
+ __u64 size;
+ The overall size of the struct, including its items.
+
+ __u64 flags;
+ The flags for creation.
+
+ KDBUS_MAKE_ACCESS_GROUP
+ Make the bus or endpoint file group-accessible
+
+ KDBUS_MAKE_ACCESS_WORLD
+ Make the bus or endpoint file world-accessible
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ struct kdbus_item items[0];
+ A list of items that has specific meanings for KDBUS_CMD_BUS_MAKE
s/has/have/
+ and KDBUS_CMD_ENDPOINT_MAKE (see above).
+
+ Following items are expected for KDBUS_CMD_BUS_MAKE:
+ KDBUS_ITEM_MAKE_NAME
+ Contains a string to identify the bus name.
So, up to here, I've seen no definition of 'kdbus_item', which leaves me
asking questions like: what subfield is KDBUS_ITEM_MAKE_NAME stored in?
which subfield holds the pointer to the string?
Somewhere earlier, kdbus_item needs to be exaplained in more detail,
I think.
+
+ KDBUS_ITEM_BLOOM_PARAMETER
+ Bus-wide bloom parameters passed in a dbus_bloom_parameter struct
+
+ KDBUS_ITEM_ATTACH_FLAGS_RECV
+ An optional item that contains a set of required attach flags
+ that connections must allow. This item is used as a negotiation
+ measure during connection creation. If connections do not satisfy
+ the bus requirements, they are not allowed on the bus.
+ If not set, the bus does not require any metadata to be attached,
s/,/;/
+ in this case connections are free to set their own attach flags.
+
+ KDBUS_ITEM_ATTACH_FLAGS_SEND
+ An optional item that contains a set of attach flags that are
+ returned to connections when they query the bus creator metadata.
+ If not set, no metadata is returned.
+
+ Unrecognized items are rejected, and the ioctl will fail with -EINVAL.
+};
+
+
+6. Connections
+===============================================================================
+
+
+6.1 Connection IDs and well-known connection names
+--------------------------------------------------
+
+Connections are identified by their connection id, internally implemented as a
+uint64_t counter. The IDs of every newly created bus start at 1, and every new
+connection will increment the counter by 1. The ids are not reused.
Again, please change "ids" to IDs" throughout.
+
+In higher level tools, the user visible representation of a connection is
+defined by the D-Bus protocol specification as ":1.<id>".
+
+Messages with a specific uint64_t destination id are directly delivered to
+the connection with the corresponding id. Messages with the special destination
+id KDBUS_DST_ID_BROADCAST are broadcast messages and are potentially delivered
+to all known connections on the bus; clients interested in broadcast messages
+need to subscribe to the specific messages they are interested, though before
+any broadcast message reaches them.
The piece following the semicolon would be better as this separate sentence,
I think:
However, in order to receive any broadcast messages, clients must
to subscribe to the specific messages in which they are interested.
+
+Messages synthesized and sent directly by the kernel will carry the special
+source id KDBUS_SRC_ID_KERNEL (0).
+
+In addition to the unique uint64_t connection id, established connections can
+request the ownership of well-known names, under which they can be found and
+addressed by other bus clients. A well-known name is associated with one and
+only one connection at a time. See section 8 on name acquisition and the
+name registry, and the validity of names.
+
+Messages can specify the special destination id 0 and carry a well-known name
+in the message data. Such a message is delivered to the destination connection
+which owns that well-known name.
+
+ +-------------------------------------------------------------------------+
+ | +---------------+ +---------------------------+ |
+ | | Connection | | Message | -----------------+ |
+ | | :1.22 | --> | src: 22 | | |
+ | | | | dst: 25 | | |
+ | | | | | | |
+ | | | | | | |
+ | | | +---------------------------+ | |
+ | | | | |
+ | | | <--------------------------------------+ | |
+ | +---------------+ | | |
+ | | | |
+ | +---------------+ +---------------------------+ | | |
+ | | Connection | | Message | -----+ | |
+ | | :1.25 | --> | src: 25 | | |
+ | | | | dst: 0xffffffffffffffff | -------------+ | |
+ | | | | (KDBUS_DST_ID_BROADCAST) | | | |
+ | | | | | ---------+ | | |
+ | | | +---------------------------+ | | | |
+ | | | | | | |
+ | | | <--------------------------------------------------+ |
+ | +---------------+ | | |
+ | | | |
+ | +---------------+ +---------------------------+ | | |
+ | | Connection | | Message | --+ | | |
+ | | :1.55 | --> | src: 55 | | | | |
+ | | | | dst: 0 / org.foo.bar | | | | |
+ | | | | | | | | |
+ | | | | | | | | |
+ | | | +---------------------------+ | | | |
+ | | | | | | |
+ | | | <------------------------------------------+ | |
+ | +---------------+ | | |
+ | | | |
+ | +---------------+ | | |
+ | | Connection | | | |
+ | | :1.81 | | | |
+ | | org.foo.bar | | | |
+ | | | | | |
+ | | | | | |
+ | | | <-----------------------------------+ | |
+ | | | | |
+ | | | <----------------------------------------------+ |
+ | +---------------+ |
+ +-------------------------------------------------------------------------+
+
+
+6.2 Creating connections
+------------------------
+
+A connection to a bus is created by opening an endpoint file of a bus and
+becoming an active client with the KDBUS_CMD_HELLO ioctl. Every connected client
+connection has a unique identifier on the bus and can address messages to every
+other connection on the same bus by using the peer's connection id as the
+destination.
+
+The KDBUS_CMD_HELLO ioctl takes the following struct as argument.
+
+struct kdbus_cmd_hello {
+ __u64 size;
+ The overall size of the struct, including all attached items.
+
+ __u64 flags;
+ Flags to apply to this connection:
+
+ KDBUS_HELLO_ACCEPT_FD
+ When this flag is set, the connection can be sent file descriptors
+ as message payload. If it's not set, any attempt of doing so will
s/any attempt of doing so/an attempt to send file descriptors/
+ result in -ECOMM on the sender's side.
+
+ KDBUS_HELLO_ACTIVATOR
+ Make this connection an activator (see below). With this bit set,
+ an item of type KDBUS_ITEM_NAME has to be attached which describes
s/attached which describes/attached. This item describes/
+ the well-known name this connection should be an activator for.
+
+ KDBUS_HELLO_POLICY_HOLDER
+ Make this connection a policy holder (see below). With this bit set,
+ an item of type KDBUS_ITEM_NAME has to be attached which describes
s/attached which describes/attached. This item describes/
+ the well-known name this connection should hold a policy for.
+
+ KDBUS_HELLO_MONITOR
+ Make this connection an eaves-dropping connection. See section 6.8 for
s/eaves-dropping/eavesdropping/
+ more information.
+
+To also receive broadcast messages,
^
Indentation error.
+ the connection has to upload appropriate matches as well.
+ This flag is only valid for privileged bus connections.
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ __u64 attach_flags_send;
+ Set the bits for metadata this connection permits to be sent to the
+ receiving peer. Only metadata items that are both allowed to be sent by
+ the sender and that are requested by the receiver will effectively be
+ attached to the message eventually. Note, however, that the bus may
What does "eventually" mean here?
+ optionally enforce some of those bits to be set. If the match fails,
s/enforce/require/ ?
+ -ECONNREFUSED will be returned. In either case, this field will be set
+ to the mask of metadata items that are enforced by the bus. The
+ KDBUS_FLAGS_KERNEL bit will as well be set.
+
+ __u64 attach_flags_recv;
+ Request the attachment of metadata for each message received by this
+ connection. The metadata actually attached may actually augment the list
Seems like two "actually" in the previous line is one too many.
+ of requested items. See section 13 for more details.
+
+ __u64 bus_flags;
+ Upon successful completion of the ioctl, this member will contain the
+ flags of the bus it connected to.
+
+ __u64 id;
+ Upon successful completion of the ioctl, this member will contain the
+ id of the new connection.
+
+ __u64 pool_size;
+ The size of the communication pool, in bytes. The pool can be accessed
+ by calling mmap() on the file descriptor that was used to issue the
+ KDBUS_CMD_HELLO ioctl.
+
+ __u64 offset;
+ The kernel will return the offset in the pool where returned details
+ will be stored.
+
+ __u8 id128[16];
+ Upon successful completion of the ioctl, this member will contain the
+ 128 bit wide UUID of the connected bus.
s/128 bit wide/128-bit/
+
+ struct kdbus_item items[0];
+ Variable list of items to add optional additional information. The
+ following items are currently expected/valid:
+
+ KDBUS_ITEM_CONN_DESCRIPTION
+ Contains a string to describes this connection's name, so it can be
s/to/that/
+ identified later.
+
+ KDBUS_ITEM_NAME
+ KDBUS_ITEM_POLICY_ACCESS
+ For activators and policy holders only, combinations of these two
+ items describe policy access entries (see section about policy).
+
+ KDBUS_ITEM_CREDS
+ KDBUS_ITEM_PIDS
+ KDBUS_ITEM_SECLABEL
+ Privileged bus users may submit these types in order to create
+ connections with faked credentials. This information will be returned
+ when peer information is queried by KDBUS_CMD_CONN_INFO. See section
+ 13 for more information.
+
+ Items of other types are rejected, and the ioctl will fail with -EINVAL.
+};
+
+At the offset returned in the 'offset' field of struct kdbus_cmd_hello, the
+kernel will store items of the following types:
+
+ KDBUS_ITEM_BLOOM_PARAMETER
+ Bloom filter parameter as defined by the bus creator (see below).
+
+The offset in the pool has to be freed with the KDBUS_CMD_FREE ioctl.
As far as I can tell, KDBUS_CMD_FREE detailed anywhere in this file. It
needs a detailed description soemwhere.
+
+6.3 Activator and policy holder connection
+------------------------------------------
+
+An activator connection is a placeholder for a well-known name. Messages sent
+to such a connection can be used by userspace to start an implementer
+connection, which will then get all the messages from the activator copied
+over. An activator connection cannot be used to send any message.
+
+A policy holder connection only installs a policy for one or more names.
+These policy entries are kept active as long as the connection is alive, and
+are removed once it terminates. Such a policy connection type can be used to
+deploy restrictions for names that are not yet active on the bus. A policy
+holder connection cannot be used to send any message.
+
+The creation of activator, policy holder or monitor connections is an operation
What is a "monitor connection"? That term springs up unannounced. Is it
an "eavesdropping connection" as described above? Either define the term
"monitor connection" or use consistent terminology. (Actually, further down
in the document, it is clarified that "monitor connection" == "eavesdropper".
But that is not clear at THIS point in the document. It needs to be clearer.)
+restricted to privileged users on the bus (see section "Terminology").
+
+
+6.4 Retrieving information on a connection
+------------------------------------------
+
+The KDBUS_CMD_CONN_INFO ioctl can be used to retrieve credentials and
+properties of the initial creator of a connection. This ioctl uses the
+following struct:
+
+struct kdbus_cmd_info {
+ __u64 size;
+ The overall size of the struct, including the name with its 0-byte string
+ terminator.
"0-byte string terminator" reads strangely. I assume you mean "terminating
null byte" / "null-terminated string" Best to use those more standard terms.
So, maybe:
including the name with its terminating null byte
or:
including the null-terminated 'name' string
There are multiple other instances of 0-byte in the doc, and I think they
should also be fixed in a similar fashion.
+
+ __u64 flags;
+ Specify which metadata items should be attached to the answer.
s/Specify/Specifies/
+ See section 13 for more details.
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ __u64 id;
+ The connection's numerical ID to retrieve information for. If set to
Hard to parse sentence.
==> The numerical ID of the connection for which information is to
be retrieved.
+ non-zero value, the 'name' field is ignored.
s/non-zero value/a non-zero value/
+
+ __u64 offset;
+ When the ioctl returns, this value will yield the offset of the connection
s/value will yield/field will contain/
+ information inside the caller's pool.
+
+ __u64 info_size;
+ The kernel will return the size of the returned information, so applications
+ can optionally mmap specific parts of the pool.
+
+ struct kdbus_item items[0];
+ The optional item list, containing the well-known name to look up as
+ a KDBUS_ITEM_OWNED_NAME. Only required if the 'id' field is set to 0.
+ Items of other types are rejected, and the ioctl will fail with -EINVAL.
+};
+
+After the ioctl returns, the following struct will be stored in the caller's
+pool at 'offset'.
+
+struct kdbus_info {
+ __u64 size;
+ The overall size of the struct, including all its items.
+
+ __u64 id;
+ The connection's unique ID.
+
+ __u64 flags;
+ The connection's flags as specified when it was created.
+
+ struct kdbus_item items[0];
+ Depending on the 'flags' field in struct kdbus_cmd_info, items of
+ types KDBUS_ITEM_OWNED_NAME and KDBUS_ITEM_CONN_DESCRIPTION are followed
s/are followed/follow/
+ here.
+};
+
+Once the caller is finished with parsing the return buffer, it needs to call
+KDBUS_CMD_FREE for the offset.
+
+
+6.5 Getting information about a connection's bus creator
+--------------------------------------------------------
+
+The KDBUS_CMD_BUS_CREATOR_INFO ioctl takes the same struct as
+KDBUS_CMD_CONN_INFO but is used to retrieve information about the creator of
+the bus the connection is attached to. The metadata returned by this call is
+collected during the creation of the bus and is never altered afterwards, so
+it provides pristine information on the task that created the bus, at the
+moment when it did so.
+
+In response to this call, a slice in the connection's pool is allocated and
+filled with an object of type struct kdbus_info, pointed to by the ioctl's
+'offset' field.
+
+struct kdbus_info {
+ __u64 size;
+ The overall size of the struct, including all its items.
+
+ __u64 id;
+ The bus ID
+
+ __u64 flags;
+ The bus flags as specified when it was created.
s/it/the bus/
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ struct kdbus_item items[0];
+ Metadata information is stored in items here. The item list contains
+ a KDBUS_ITEM_MAKE_NAME item that indicates the bus name of the
+ calling connection.
+};
+
+Once the caller is finished with parsing the return buffer, it needs to call
+KDBUS_CMD_FREE for the offset.
+
+
+6.6 Updating connection details
+-------------------------------
+
+Some of a connection's details can be updated with the KDBUS_CMD_CONN_UPDATE
+ioctl, using the file descriptor that was used to create the connection.
+The update command uses the following struct.
+
+struct kdbus_cmd_update {
+ __u64 size;
+ The overall size of the struct, including all its items.
+
+ __u64 flags;
+ Currently no flags are supported. Reserved for future use.
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ struct kdbus_item items[0];
+ Items to describe the connection details to be updated. The following item
+ types are supported:
+
+ KDBUS_ITEM_ATTACH_FLAGS_SEND
+ Supply a new set of items that this connection permits to be sent along
+ with messages.
+
+ KDBUS_ITEM_ATTACH_FLAGS_RECV
+ Supply a new set of items to be attached to each message.
+
+ KDBUS_ITEM_NAME
+ KDBUS_ITEM_POLICY_ACCESS
+ Policy holder connections may supply a new set of policy information
+ with these items. For other connection types, -EOPNOTSUPP is returned.
+
+ Items of other types are rejected, and the ioctl will fail with -EINVAL.
+};
+
+
+6.7 Termination
+---------------
+
+A connection can be terminated by simply closing its file descriptor. All
+pending incoming messages will be discarded, and the memory in the pool will
+be freed.
+
+An alternative way of closing down a connection is calling the KDBUS_CMD_BYEBYE
+ioctl on it, which will only succeed if the message queue of the connection is
+empty at the time of closing, otherwise, -EBUSY is returned.
The preceding is a little hard to parse. I suggest:
An alternative way of closing down a connection is via the KDBUS_CMD_BYEBYE
ioctl. This ioctly will succeed only if the message queue of the connection is
empty at the time of closing; otherwise, -EBUSY is returned.
+
+When this ioctl returns successfully, the connection has been terminated and
+won't accept any new messages from remote peers. This way, a connection can
+be terminated race-free, without losing any messages.
+
+
+6.8 Monitor connections ('eavesdropper')
+----------------------------------------
+
+Eavesdropping connections are created by setting the KDBUS_HELLO_MONITOR flag
+in struct kdbus_hello.flags. Such connections have all properties of any other,
+regular connection, except for the following details:
+
+ * They will get every message sent over the bus, both unicasts and broadcasts
+
+ * Installing matches for broadcast messages is neither necessary nor allowed
+
+ * They cannot send messages or be directly addressed as receiver
+
+ * Their creation and destruction will not cause KDBUS_ITEM_ID_{ADD,REMOVE}
+ notifications to be generated, so other connections cannot detect the
+ presence of an eavesdropper.
+
+
+7. Messages
+===============================================================================
+
+Messages consist of a fixed-size header followed directly by a list of
+variable-sized data 'items'. The overall message size is specified in the
+header of the message. The chain of data items can contain well-defined
+message metadata fields, raw data, references to data, or file descriptors.
+
+
+7.1 Sending messages
+--------------------
+
+Messages are passed to the kernel with the KDBUS_CMD_SEND ioctl. Depending
+on the destination address of the message, the kernel delivers the message to
+the specific destination connection or to all connections on the same bus.
+Sending messages across buses is not possible. Messages are always queued in
+the memory pool of the destination connection (see below).
+
+The KDBUS_CMD_SEND ioctl uses struct kdbus_cmd_send to describe the message
+transfer.
+
+struct kdbus_cmd_send {
+ __u64 size;
+ The overall size of the struct, including the attached items.
+
+ __u64 flags;
+ Flags for message delivery:
+
+ KDBUS_SEND_SYNC_REPLY
+ By default, all calls to kdbus are considered asynchronous,
+ non-blocking. However, as there are many use cases that need to wait
+ for a remote peer to answer a method call, there's a way to send a
+ message and wait for a reply in a synchronous fashion. This is what
+ the KDBUS_MSG_SYNC_REPLY controls. The KDBUS_CMD_SEND ioctl will block
+ until the reply has arrived, the timeout limit is reached, in case the
+ remote connection was shut down, or if interrupted by a signal before
+ any reply; see signal(7).
+
+ The offset of the reply message in the sender's pool is stored in in
+ 'offset_reply' when the ioctl has returned without error. Hence, there
+ is no need for another KDBUS_CMD_RECV ioctl or anything else to receive
+ the reply.
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call of
+ KDBUS_CMD_SEND.
+
+ __u64 kernel_msg_flags;
+ Valid bits for message flags, returned by the kernel upon each call of
+ KDBUS_CMD_SEND.
+
+ __u64 return_flags;
+ Kernel-provided flags, returning non-fatal errors that occurred during
+ send. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ __u64 msg_address;
+ Userspace has to provide a pointer to a message (struct kdbus_msg) to send.
+
+ struct kdbus_msg_info reply;
+ Only used for synchronous replies. See description of struct kdbus_cmd_recv
+ for more details.
+
+ struct kdbus_item items[0];
+ The following items are currently recognized:
+
+ KDBUS_ITEM_CANCEL_FD
+ When this optional item is passed in, and the call is executed as SYNC
+ call, the passed in file descriptor can be used as alternative
+ cancellation point. The kernel will call poll() on this file descriptor,
+ and if it reports any incoming bytes, the blocking send operation will
+ be canceled, and the call will return -ECANCELED. Any type of file
+ descriptor that implements poll() can be used as payload to this item.
+ For asynchronous message sending, this item is accepted but ignored.
+
+ All other items are rejected, and the ioctl will fail with -EINVAL.
+};
+
+The message referenced by 'msg_address' above has the following layout.
+
+struct kdbus_msg {
+ __u64 size;
+ The overall size of the struct, including the attached items.
+
+ __u64 flags;
+ KDBUS_MSG_EXPECT_REPLY
+ Expect a reply from the remote peer to this message. With this bit set,
s/from the remote peer to this message/to this message from the remote peer/
+ the timeout_ns field must be set to a non-zero number of nanoseconds in
+ which the receiving peer is expected to reply. If such a reply is not
+ received in time, the sender will be notified with a timeout message
+ (see below). The value must be an absolute value, in nanoseconds and
+ based on CLOCK_MONOTONIC.
Why is the option of a relative timeout not available?
+ For a message to be accepted as reply, it must be a direct message to
+ the original sender (not a broadcast), and its kdbus_msg.reply_cookie
+ must match the previous message's kdbus_msg.cookie.
+
+ Expected replies also temporarily open the policy of the sending
+ connection, so the other peer is allowed to respond within the given
+ time window.
+
+ KDBUS_MSG_NO_AUTO_START
+ By default, when a message is sent to an activator connection, the
+ activator notified and will start an implementer. This flag inhibits
+ that behavior. With this bit set, and the remote being an activator,
+ -EADDRNOTAVAIL is returned from the ioctl.
+
+ __s64 priority;
+ The priority of this message. Receiving messages (see below) may
+ optionally be constrained to messages of a minimal priority. This
+ allows for use cases where timing critical data is interleaved with
+ control data on the same connection. If unused, the priority should be
+ set to zero.
+
+ __u64 dst_id;
+ The numeric ID of the destination connection, or KDBUS_DST_ID_BROADCAST
+ (~0ULL) to address every peer on the bus, or KDBUS_DST_ID_NAME (0) to look
+ it up dynamically from the bus' name registry. In the latter case, an item
+ of type KDBUS_ITEM_DST_NAME is mandatory.
+
+ __u64 src_id;
+ Upon return of the ioctl, this member will contain the sending
+ connection's numerical ID. Should be 0 at send time.
+
+ __u64 payload_type;
+ Type of the payload in the actual data records. Currently, only
+ KDBUS_PAYLOAD_DBUS is accepted as input value of this field. When
+ receiving messages that are generated by the kernel (notifications),
+ this field will yield KDBUS_PAYLOAD_KERNEL.
s/yield/cointain/ ?
+
+ __u64 cookie;
+ Cookie of this message, for later recognition. Also, when replying
+ to a message (see above), the cookie_reply field must match this value.
+
+ __u64 timeout_ns;
+ If the message sent requires a reply from the remote peer (see above),
+ this field contains the timeout in absolute nanoseconds based on
+ CLOCK_MONOTONIC.
+
+ __u64 cookie_reply;
+ If the message sent is a reply to another message, this field must
+ match the cookie of the formerly received message.
+
+ struct kdbus_item items[0];
+ A dynamically sized list of items to contain additional information.
+ The following items are expected/valid:
+
+ KDBUS_ITEM_PAYLOAD_VEC
+ KDBUS_ITEM_PAYLOAD_MEMFD
+ KDBUS_ITEM_FDS
+ Actual data records containing the payload. See section "Passing of
+ Payload Data".
+
+ KDBUS_ITEM_BLOOM_FILTER
+ Bloom filter for matches (see below).
+
+ KDBUS_ITEM_DST_NAME
+ Well-known name to send this message to. Required if dst_id is set
+ to KDBUS_DST_ID_NAME. If a connection holding the given name can't
+ be found, -ESRCH is returned.
+ For messages to a unique name (ID), this item is optional. If present,
+ the kernel will make sure the name owner matches the given unique name.
+ This allows userspace tie the message sending to the condition that a
+ name is currently owned by a certain unique name.
+};
+
+The message will be augmented by the requested metadata items when queued into
+the receiver's pool. See also section 13.2 ("Metadata and namespaces").
+
+
+7.2 Message layout
+------------------
+
+The layout of a message is shown below.
+
+ +-------------------------------------------------------------------------+
+ | Message |
+ | +---------------------------------------------------------------------+ |
+ | | Header | |
+ | | size: overall message size, including the data records | |
+ | | destination: connection id of the receiver | |
+ | | source: connection id of the sender (set by kernel) | |
+ | | payload_type: "DBusDBus" textual identifier stored as uint64_t | |
+ | +---------------------------------------------------------------------+ |
+ | +---------------------------------------------------------------------+ |
+ | | Data Record | |
+ | | size: overall record size (without padding) | |
+ | | type: type of data | |
+ | | data: reference to data (address or file descriptor) | |
+ | +---------------------------------------------------------------------+ |
+ | +---------------------------------------------------------------------+ |
+ | | padding bytes to the next 8 byte alignment | |
+ | +---------------------------------------------------------------------+ |
+ | +---------------------------------------------------------------------+ |
+ | | Data Record | |
+ | | size: overall record size (without padding) | |
+ | | ... | |
+ | +---------------------------------------------------------------------+ |
+ | +---------------------------------------------------------------------+ |
+ | | padding bytes to the next 8 byte alignment | |
+ | +---------------------------------------------------------------------+ |
+ | +---------------------------------------------------------------------+ |
+ | | Data Record | |
+ | | size: overall record size | |
+ | | ... | |
+ | +---------------------------------------------------------------------+ |
+ | ... further data records ... |
+ +-------------------------------------------------------------------------+
+
+
+7.3 Passing of Payload Data
+---------------------------
+
+When connecting to the bus, receivers request a memory pool of a given size,
+large enough to carry all backlog of data enqueued for the connection. The
+pool is internally backed by a shared memory file which can be mmap()ed by
+the receiver.
+
+KDBUS_MSG_PAYLOAD_VEC:
+ Messages are directly copied by the sending process into the receiver's pool,
s/,/./ and then start a new sentence.
+ that way two peers can exchange data by effectively doing a single-copy from
+ one process to another, the kernel will not buffer the data anywhere else.
s/,/;/
+
+KDBUS_MSG_PAYLOAD_MEMFD:
+ Messages can reference memfd files which contain the data.
+ memfd files are tmpfs-backed files that allow sealing of the content of the
+ file, which prevents all writable access to the file content.
+ Only memfds that have (F_SEAL_SHRINK|F_SEAL_GROW|F_SEAL_WRITE|F_SEAL_SEAL) set
+ are accepted as payload data, which enforces reliable passing of data.
+ The receiver can assume that neither the sender nor anyone else can alter the
+ content after the message is sent.
+ Apart from the sender filling-in the content into memfd files, the data will
+ be passed as zero-copy from one process to another, read-only, shared between
+ the peers.
+
+The sender must not make any assumptions on the type how data is received by the
Wording error at "the type how". Some fix is needed.
+remote peer. The kernel is free to re-pack multiple VEC and MEMFD payloads. For
+instance, the kernel may decide to merge multiple VECs into a single VEC, inline
+MEMFD payloads into memory or merge all passed VECs into a single MEMFD.
+However, the kernel preserves the order of passed data. This means, the order of
s/,//
+all VEC and MEMFD items is not changed in respect to each other.
s/in/with/
+
+In other words: All passed VEC and MEMFD data payloads are treated as a single
+stream of data that may be received by the remote peer in a different set of
+hunks than it was sent as.
+
+
+7.4 Receiving messages
+----------------------
+
+Messages are received by the client with the KDBUS_CMD_RECV ioctl. The endpoint
+file of the bus supports poll() to wake up the receiving process when new
s%poll()%poll/select()/epoll% ?
+messages are queued up to be received.
+
+With the KDBUS_CMD_RECV ioctl, a struct kdbus_cmd_recv is used.
+
+struct kdbus_cmd_recv {
+ __u64 size;
+ The overall size of the struct, including the attached items.
+
+ __u64 flags;
+ Flags to control the receive command.
+
+ KDBUS_RECV_PEEK
+ Just return the location of the next message. Do not install file
+ descriptors or anything else. This is usually used to determine the
+ sender of the next queued message.
+
+ KDBUS_RECV_DROP
+ Drop the next message without doing anything else with it, and free the
+ pool slice. This a short-cut for KDBUS_RECV_PEEK and KDBUS_CMD_FREE.
+
+ KDBUS_RECV_USE_PRIORITY
+ Use the priority field (see below).
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Kernel-provided flags, returning non-fatal errors that occurred during
+ send. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ __s64 priority;
+ With KDBUS_RECV_USE_PRIORITY set in flags, receive the next message in
+ the queue with at least the given priority. If no such message is waiting
+ in the queue, -ENOMSG is returned.
###
How do I simply select the highest priority message, without knowing what
its priority is?
+
+ __u64 dropped_msgs;
+ If the CMD_RECV ioctl fails with EOVERFLOW, this field is filled by
+ the kernel with the number of messages that couldn't be transmitted to
+ this connection. In that case, the @offset member must not be accessed.
+
+ struct kdbus_msg_info msg;
+ Embedded struct to be filled when the command succeeded (see below).
+
+ struct kdbus_item items[0];
+ Items to specify further details for the receive command. Currently unused.
+};
+
+Both 'struct kdbus_cmd_recv' and 'struct kdbus_cmd_send' embed 'struct
+kdbus_msg_info'. For the SEND ioctl, it is used to catch synchronous replies,
+if one was requested, and is unused otherwise.
+
+struct kdbus_msg_info {
+ __u64 offset;
+ Upon return of the ioctl, this field contains the offset in the receiver's
+ memory pool. The memory must be freed with KDBUS_CMD_FREE.
+
+ __u64 msg_size;
+ Upon successful return of the ioctl, this field contains the size of the
+ allocated slice at offset @offset. It is the combination of the size of
+ the stored kdbus_msg object plus all appended VECs. You can use it in
+ combination with @offset to map a single message, instead of mapping the
+ whole pool.
+
+ __u64 return_flags;
+ Kernel-provided return flags. Currently, the following flags are defined.
+
+ KDBUS_RECV_RETURN_INCOMPLETE_FDS
+ The message contained file descriptors which couldn't be installed
+ into the receiver's task. Most probably that happened because the
+ maximum number of file descriptors for that task were exceeded.
+ The message is still delivered, so this is not a fatal condition.
+ File descriptors inside the KDBUS_ITEM_FDS item that could not be
+ installed will be set to -1.
+};
+
+Unless KDBUS_RECV_DROP was passed, and given that the ioctl succeeded, the
s/and given that the ioctl succeeded/after a successful KDBUS_CMD_RECV ioctl/
+offset field contains the location of the new message inside the receiver's
+pool. The message is stored as struct kdbus_msg at this offset, and can be
+interpreted with the semantics described above.
+
+Also, if the connection allowed for file descriptor to be passed
+(KDBUS_HELLO_ACCEPT_FD), and if the message contained any, they will be
+installed into the receiving process after the KDBUS_CMD_RECV ioctl returns.
###
"after"??? When exactly?
+The receiving task is obliged to close all of them appropriately. If
+KDBUS_RECV_PEEK is set, no file descriptors are installed. This allows for
+peeking at a message and dropping it via KDBUS_RECV_DROP, without installing
+the passed file descriptors into the receiving process.
+
+The caller is obliged to call KDBUS_CMD_FREE with the returned offset when
+the memory is no longer needed.
+
+
+8. Name registry
+===============================================================================
+
+Each bus instantiates a name registry to resolve well-known names into unique
+connection IDs for message delivery. The registry will be queried when a
+message is sent with kdbus_msg.dst_id set to KDBUS_DST_ID_NAME, or when a
+registry dump is requested.
+
+All of the below is subject to policy rules for SEE and OWN permissions.
+
+
+8.1 Name validity
+-----------------
+
+A name has to comply to the following rules to be considered valid:
s/comply to/comply with/
+
+ - The name has two or more elements separated by a period ('.') character
+ - All elements must contain at least one character
+ - Each element must only contain the ASCII characters "[A-Z][a-z][0-9]_"
+ and must not begin with a digit
+ - The name must contain at least one '.' (period) character
+ (and thus at least two elements)
+ - The name must not begin with a '.' (period) character
+ - The name must not exceed KDBUS_NAME_MAX_LEN (255)
+
+
+8.2 Acquiring a name
+--------------------
+
+To acquire a name, a client uses the KDBUS_CMD_NAME_ACQUIRE ioctl with the
+following data structure.
+
+struct kdbus_cmd_name {
+ __u64 size;
+ The overall size of this struct, including the name with its 0-byte string
+ terminator.
+
+ __u64 flags;
+ Flags to control details in the name acquisition.
+
+ KDBUS_NAME_REPLACE_EXISTING
+ Acquiring a name that is already present usually fails, unless this flag
+ is set in the call, and KDBUS_NAME_ALLOW_REPLACEMENT or (see below) was
+ set when the current owner of the name acquired it, or if the current
+ owner is an activator connection (see below).
+
+ KDBUS_NAME_ALLOW_REPLACEMENT
+ Allow other connections to take over this name. When this happens, the
+ former owner of the connection will be notified of the name loss.
+
+ KDBUS_NAME_QUEUE (acquire)
+ A name that is already acquired by a connection, and which wasn't
+ requested with the KDBUS_NAME_ALLOW_REPLACEMENT flag set can not be
+ acquired again. However, a connection can put itself in a queue of
+ connections waiting for the name to be released. Once that happens, the
+ first connection in that queue becomes the new owner and is notified
+ accordingly.
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ struct kdbus_item items[0];
+ Items to submit the name. Currently, one item of type KDBUS_ITEM_NAME is
+ expected and allowed, and the contained string must be a valid bus name.
+ Items of other types are rejected, and the ioctl will fail with -EINVAL.
+};
+
+
+8.3 Releasing a name
+--------------------
+
+A connection may release a name explicitly with the KDBUS_CMD_NAME_RELEASE
+ioctl. If the connection was an implementer of an activatable name, its
+pending messages are moved back to the activator. If there are any connections
+queued up as waiters for the name, the oldest one of them will become the new
+owner. The same happens implicitly for all names once a connection terminates.
+
+The KDBUS_CMD_NAME_RELEASE ioctl uses the same data structure as the
+acquisition call, but with slightly different field usage.
+
+struct kdbus_cmd_name {
+ __u64 size;
+ The overall size of this struct, including the name with its 0-byte string
+ terminator.
+
+ __u64 flags;
+ Flags to the command. Currently unused.
And, so presumably must be 0? Best to note that.
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+ struct kdbus_item items[0];
+ Items to submit the name. Currently, one item of type KDBUS_ITEM_NAME is
+ expected and allowed, and the contained string must be a valid bus name.
+};
+
+
+8.4 Dumping the name registry
+-----------------------------
+
+A connection may request a complete or filtered dump of currently active bus
+names with the KDBUS_CMD_NAME_LIST ioctl, which takes a struct
+kdbus_cmd_name_list as argument.
+
+struct kdbus_cmd_name_list {
+ __u64 flags;
+ Any combination of flags to specify which names should be dumped.
+
+ KDBUS_NAME_LIST_UNIQUE
+ List the unique (numeric) IDs of the connection, whether it owns a name
+ or not.
+
+ KDBUS_NAME_LIST_NAMES
+ List well-known names stored in the database which are actively owned by
+ a real connection (not an activator).
+
+ KDBUS_NAME_LIST_ACTIVATORS
+ List names that are owned by an activator.
+
+ KDBUS_NAME_LIST_QUEUED
+ List connections that are not yet owning a name but are waiting for it
+ to become available.
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ __u64 offset;
+ When the ioctl returns successfully, the offset to the name registry dump
+ inside the connection's pool will be stored in this field.
+};
+
+The returned list of names is stored in a struct kdbus_name_list that in turn
+contains a dynamic number of struct kdbus_cmd_name that carry the actual
+information. The fields inside that struct kdbus_cmd_name is described next.
+
+struct kdbus_name_info {
+ __u64 size;
+ The overall size of this struct, including the name with its 0-byte string
+ terminator.
+
+ __u64 owner_id;
+ The owning connection's unique ID.
+
+ __u64 conn_flags;
+ The flags of the owning connection.
+
+ struct kdbus_item items[0];
+ Items containing the actual name. Currently, one item of type
+ KDBUS_ITEM_OWNED_NAME will be attached, including the name's flags. In that
+ item, the flags field of the name may carry the following bits:
+
+ KDBUS_NAME_ALLOW_REPLACEMENT
+ Other connections are allowed to take over this name from the
+ connection that owns it.
+
+ KDBUS_NAME_IN_QUEUE (list)
+ When retrieving a list of currently acquired name in the registry, this
+ flag indicates whether the connection actually owns the name or is
+ currently waiting for it to become available.
+
+ KDBUS_NAME_ACTIVATOR (list)
+ An activator connection owns a name as a placeholder for an implementer,
+ which is started on demand as soon as the first message arrives. There's
+ some more information on this topic below. In contrast to
+ KDBUS_NAME_REPLACE_EXISTING, when a name is taken over from an activator
+ connection, all the messages that have been queued in the activator
+ connection will be moved over to the new owner. The activator connection
+ will still be tracked for the name and will take control again if the
+ implementer connection terminates.
+ This flag can not be used when acquiring a name, but is implicitly set
+ through KDBUS_CMD_HELLO with KDBUS_HELLO_ACTIVATOR set in
+ kdbus_cmd_hello.conn_flags.
+};
+
+The returned buffer must be freed with the KDBUS_CMD_FREE ioctl when the user
+is finished with it.
+
+
+9. Notifications
+===============================================================================
+
+The kernel will notify its users of the following events.
+
+ * When connection A is terminated while connection B is waiting for a reply
+ from it, connection B is notified with a message with an item of type
+ KDBUS_ITEM_REPLY_DEAD.
+
+ * When connection A does not receive a reply from connection B within the
+ specified timeout window, connection A will receive a message with an item
+ of type KDBUS_ITEM_REPLY_TIMEOUT.
+
+ * When an ordinary connection (not a monitor) is created on or removed from
+ a bus, messages with an item of type KDBUS_ITEM_ID_ADD or
+ KDBUS_ITEM_ID_REMOVE, respectively, are sent to all bus members that match
+ these messages through their match database. Eavesdroppers (monitor
+ connections) do not cause such notifications to be sent. They are invisible
+ on the bus.
+
+ * When a connection gains or loses ownership of a name, messages with an item
+ of type KDBUS_ITEM_NAME_ADD, KDBUS_ITEM_NAME_REMOVE or
+ KDBUS_ITEM_NAME_CHANGE are sent to all bus members that match these
+ messages through their match database.
+
+A kernel notification is a regular kdbus message with the following details.
+
+ * kdbus_msg.src_id == KDBUS_SRC_ID_KERNEL
+ * kdbus_msg.dst_id == KDBUS_DST_ID_BROADCAST
+ * kdbus_msg.payload_type == KDBUS_PAYLOAD_KERNEL
+ * Has exactly one of the aforementioned items attached
+
+Kernel notifications have an item of type KDBUS_ITEM_TIMESTAMP attached.
+
+
+10. Message Matching, Bloom filters
+===============================================================================
+
+10.1 Matches for broadcast messages from other connections
+----------------------------------------------------------
+
+A message addressed at the connection ID KDBUS_DST_ID_BROADCAST (~0ULL) is a
+broadcast message, delivered to all connected peers which installed a rule to
+match certain properties of the message. Without any rules installed in the
+connection, no broadcast message or kernel-side notifications will be delivered
+to the connection. Broadcast messages are subject to policy rules and TALK
+access checks.
+
+See section 11 for details on policies, and section 11.5 for more
+details on implicit policies.
+
+Matches for messages from other connections (not kernel notifications) are
+implemented as bloom filters. The sender adds certain properties of the message
+as elements to a bloom filter bit field, and sends that along with the
+broadcast message.
+
+The connection adds the message properties it is interested as elements to a
+bloom mask bit field, and uploads the mask to the match rules of the
+connection.
+
+The kernel will match the broadcast message's bloom filter against the
+connections bloom mask (simply by &-ing it), and decide whether the message
+should be delivered to the connection.
+
+The kernel has no notion of any specific properties of the message, all it
+sees are the bit fields of the bloom filter and mask to match against. The
+use of bloom filters allows simple and efficient matching, without exposing
+any message properties or internals to the kernel side. Clients need to deal
+with the fact that they might receive broadcasts which they did not subscribe
+to, as the bloom filter might allow false-positives to pass the filter.
+
+To allow the future extension of the set of elements in the bloom filter, the
+filter specifies a "generation" number. A later generation must always contain
+all elements of the set of the previous generation, but can add new elements
+to the set. The match rules mask can carry an array with all previous
+generations of masks individually stored. When the filter and mask are matched
+by the kernel, the mask with the closest matching "generation" is selected
+as the index into the mask array.
+
+
+10.2 Matches for kernel notifications
+------------------------------------
+
+To receive kernel generated notifications (see section 9), a connection must
+install special match rules that are different from the bloom filter matches
+described in the section above. They can be filtered by a sender connection's
+ID, by one of the name the sender connection owns at the time of sending the
+message, or by type of the notification (id/name add/remove/change).
s/type/the type/
+
+10.3 Adding a match
+-------------------
+
+To add a match, the KDBUS_CMD_MATCH_ADD ioctl is used, which takes a struct
+of the struct described below.
+
+Note that each of the items attached to this command will internally create
+one match 'rule', and the collection of them, which is submitted as one block
+via the ioctl is called a 'match'. To allow a message to pass, all rules of a
s/ioctl/ioctl,/
+match have to be satisfied. Hence, adding more items to the command will only
+narrow the possibility of a match to effectively let the message pass, and will
+cause the connection's user space process to wake up less likely.
Make that last line
decrease the chance that the connection's userspace process wil be woken up
+
+Multiple matches can be installed per connection. As long as one of it has a
s/it/them/ ?
(If that change is not correct, then the sentence is quite confused.)
+set of rules which allows the message to pass, this one will be decisive.
+
+struct kdbus_cmd_match {
+ __u64 size;
+ The overall size of the struct, including its items.
+
+ __u64 cookie;
+ A cookie which identifies the match, so it can be referred to at removal
+ time.
+
+ __u64 flags;
+ Flags to control the behavior of the ioctl.
+
+ KDBUS_MATCH_REPLACE:
+ Remove all entries with the given cookie before installing the new one.
+ This allows for race-free replacement of matches.
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ struct kdbus_item items[0];
+ Items to define the actual rules of the matches. The following item types
+ are expected. Each item will cause one new match rule to be created.
+
+ KDBUS_ITEM_BLOOM_MASK
+ An item that carries the bloom filter mask to match against in its
+ data field. The payload size must match the bloom filter size that
+ was specified when the bus was created.
+ See section 10.4 for more information.
+
+ KDBUS_ITEM_NAME
+ Specify a name that a sending connection must own at a time of sending
s/a time/the time/
+ a broadcast message in order to match this rule.
+
+ KDBUS_ITEM_ID
+ Specify a sender connection's ID that will match this rule.
+
+ KDBUS_ITEM_NAME_ADD
+ KDBUS_ITEM_NAME_REMOVE
+ KDBUS_ITEM_NAME_CHANGE
+ These items request delivery of broadcast messages that describe a name
+ acquisition, loss, or change. The details are stored in the item's
+ kdbus_notify_name_change member. All information specified must be
+ matched in order to make the message pass. Use KDBUS_MATCH_ID_ANY to
+ match against any unique connection ID.
+
+ KDBUS_ITEM_ID_ADD
+ KDBUS_ITEM_ID_REMOVE
+ These items request delivery of broadcast messages that are generated
+ when a connection is created or terminated. struct kdbus_notify_id_change
+ is used to store the actual match information. This item can be used to
+ monitor one particular connection ID, or, when the id field is set to
+ KDBUS_MATCH_ID_ANY, all of them.
+
+ Items of other types are rejected, and the ioctl will fail with -EINVAL.
+};
+
+
+10.4 Bloom filters
+------------------
+
+Bloom filters allow checking whether a given word is present in a dictionary.
+This allows connections to set up a mask for information it is interested in,
+and will be delivered signal messages that have a matching filter.
+
+For general information on bloom filters, see
+
+ https://en.wikipedia.org/wiki/Bloom_filter
+
+The size of the bloom filter is defined per bus when it is created, in
+kdbus_bloom_parameter.size. All bloom filters attached to signals on the bus
+must match this size, and all bloom filter matches uploaded by connections must
+also match the size, or a multiple thereof (see below).
+
+The calculation of the mask has to be done on the userspace side. The kernel
+just checks the bitmasks to decide whether or not to let the message pass. All
+bits in the mask must match the filter in and bit-wise AND logic, but the
Parse error at "in and bit-wise AND logic". I am not sure what you are meaning
there, buth something needs fixing.
+mask may have more bits set than the filter. Consequently, false positive
+matches are expected to happen, and userspace must deal with that fact.
+
+Masks are entities that are always passed to the kernel as part of a match
+(with an item of type KDBUS_ITEM_BLOOM_MASK), and filters can be attached to
+signals, with an item of type KDBUS_ITEM_BLOOM_FILTER.
+
+For a filter to match, all its bits have to be set in the match mask as well.
+For example, consider a bus has a bloom size of 8 bytes, and the following
s/has/that has/
+mask/filter combinations:
+
+ filter 0x0101010101010101
+ mask 0x0101010101010101
+ -> matches
+
+ filter 0x0303030303030303
+ mask 0x0101010101010101
+ -> doesn't match
+
+ filter 0x0101010101010101
+ mask 0x0303030303030303
+ -> matches
+
+Hence, in order to catch all messages, a mask filled with 0xff bytes can be
+installed as a wildcard match rule.
+
+Uploaded matches may contain multiple masks, each of which in the size of the
Parse error at "each of which in the size of"
s/in/is/?
+bloom size defined by the bus. Each block of a mask is called a 'generation',
+starting at index 0.
+
+At match time, when a signal is about to be delivered, a bloom mask generation
+is passed, which denotes which of the bloom masks the filter should be matched
+against. This allows userspace to provide backward compatible masks at upload
+time, while older clients can still match against older versions of filters.
+
+
+10.5 Removing a match
+--------------------
+
+Matches can be removed through the KDBUS_CMD_MATCH_REMOVE ioctl, which again
+takes struct kdbus_cmd_match as argument, but its fields are used slightly
+differently.
+
+struct kdbus_cmd_match {
+ __u64 size;
+ The overall size of the struct. As it has no items in this use case, the
+ value should yield 16.
s/yield/contain/ ?
+
+ __u64 cookie;
+ The cookie of the match, as it was passed when the match was added.
+ All matches that have this cookie will be removed.
+
+ __u64 flags;
+ Unused for this use case,
+
+ __u64 kernel_flags;
+ Valid flags for this command, returned by the kernel upon each call.
+
+ __u64 return_flags;
+ Flags returned by the kernel. Currently unused.
And, so presumably always returned as 0? Best to note that.
+
+ struct kdbus_item items[0];
+ Unused und not allowed for this use case.
+};
+
+
+11. Policy
+===============================================================================
+
+A policy databases restrict the possibilities of connections to own, see and
+talk to well-known names. It can be associated with a bus (through a policy
s/It/A policy/
+holder connection) or a custom endpoint.
+
+See section 8.1 for more details on the validity of well-known names.
+
+Default endpoints of buses always have a policy database. The default
+policy is to deny all operations except for operations that are covered by
+implicit policies. Custom endpoints always have a policy, and by default,
+a policy database is empty. Therefore, unless policy rules are added, all
s/a policy database/the policy database/
+operations will also be denied by default.
+
+See section 11.5 for more details on implicit policies.
+
+A set of policy rules is described by a name and multiple access rules, defined
+by the following struct.
+
+struct kdbus_policy_access {
+ __u64 type; /* USER, GROUP, WORLD */
+ One of the following.
+
+ KDBUS_POLICY_ACCESS_USER
+ Grant access to a user with the uid stored in the 'id' field.
+
+ KDBUS_POLICY_ACCESS_GROUP
+ Grant access to a user with the gid stored in the 'id' field.
+
+ KDBUS_POLICY_ACCESS_WORLD
+ Grant access to everyone. The 'id' field is ignored.
+
+ __u64 access; /* OWN, TALK, SEE */
+ The access to grant.
+
+ KDBUS_POLICY_SEE
+ Allow the name to be seen.
+
+ KDBUS_POLICY_TALK
+ Allow the name to be talked to.
+
+ KDBUS_POLICY_OWN
+ Allow the name to be owned.
+
+ __u64 id;
+ For KDBUS_POLICY_ACCESS_USER, stores the uid.
+ For KDBUS_POLICY_ACCESS_GROUP, stores the gid.
+};
+
+Policies are set through KDBUS_CMD_HELLO (when creating a policy holder
+connection), KDBUS_CMD_CONN_UPDATE (when updating a policy holder connection),
+KDBUS_CMD_ENDPOINT_MAKE (creating a custom endpoint) or
+KDBUS_CMD_ENDPOINT_UPDATE (updating a custom endpoint). In all cases, the name
+and policy access information is stored in items of type KDBUS_ITEM_NAME and
+KDBUS_ITEM_POLICY_ACCESS. For this transport, the following rules apply.
+
+ * An item of type KDBUS_ITEM_NAME must be followed by at least one
+ KDBUS_ITEM_POLICY_ACCESS item
+ * An item of type KDBUS_ITEM_NAME can be followed by an arbitrary number of
+ KDBUS_ITEM_POLICY_ACCESS items
+ * An arbitrary number of groups of names and access levels can be passed
+
+uids and gids are internally always stored in the kernel's view of global ids,
+and are translated back and forth on the ioctl level accordingly.
+
+
+11.2 Wildcard names
+-------------------
+
+Policy holder connections may upload names that contain the wild card suffix
s/wild card/wildcard/
+(".*"). That way, a policy can be uploaded that is effective for every
+well-known name that extends the provided name by exactly one more level.
+
+For example, if an item of a set up uploaded policy rules contains the name
I find that last line very difficult to parse. Someehting is broken.
What are you trying to say?
+"foo.bar.*", both "foo.bar.baz" and "foo.bar.bazbaz" are valid, but
s/both/then both/ to help parsing.
+"foo.bar.baz.baz" is not.
+
+This allows connections to take control over multiple names that the policy
+holder doesn't need to know about when uploading the policy.
+
+Such wildcard entries are not allowed for custom endpoints.
+
+
+11.3 Policy example
+-------------------
+
+For example, a set of policy rules may look like this:
+
+ KDBUS_ITEM_NAME: str='org.foo.bar'
+ KDBUS_ITEM_POLICY_ACCESS: type=USER, access=OWN, id=1000
+ KDBUS_ITEM_POLICY_ACCESS: type=USER, access=TALK, id=1001
+ KDBUS_ITEM_POLICY_ACCESS: type=WORLD, access=SEE
+ KDBUS_ITEM_NAME: str='org.blah.baz'
+ KDBUS_ITEM_POLICY_ACCESS: type=USER, access=OWN, id=0
+ KDBUS_ITEM_POLICY_ACCESS: type=WORLD, access=TALK
+
+That means that 'org.foo.bar' may only be owned by uid 1000, but every user on
+the bus is allowed to see the name. However, only uid 1001 may actually send
+a message to the connection and receive a reply from it.
+
+The second rule allows 'org.blah.baz' to be owned by uid 0 only, but every user
+may talk to it.
+
+
+11.4 TALK access and multiple well-known names per connection
+-------------------------------------------------------------
+
+Note that TALK access is checked against all names of a connection.
+For example, if a connection owns both 'org.foo.bar' and 'org.blah.baz', and
+the policy database allows 'org.blah.baz' to be talked to by WORLD, then this
+permission is also granted to 'org.foo.bar'. That might sound illogical, but
+after all, we allow messages to be directed to either the ID or a well-known
+name, and policy is applied to the connection, not the name. In other words,
+the effective TALK policy for a connection is the most permissive of all names
+the connection owns.
+
+For broadcast messages, the receiver needs TALK permissions to the sender to
+receive the broadcast.
+
+If a policy database exists for a bus (because a policy holder created one on
+demand) or for a custom endpoint (which always has one), each one is consulted
By "created one on demand" do you mean "explicitly created one"? If so
please use the latter wording, since its clearer.
+during name registry listing, name owning or message delivery. If either one
What is "name owning". I think this could be worded better.
+fails, the operation is failed with -EPERM.
+
+For best practices, connections that own names with a restricted TALK
+access should not install matches. This avoids cases where the sent
+message may pass the bloom filter due to false-positives and may also
+satisfy the policy rules.
+
+
+11.5 Implicit policies
+----------------------
+
+Depending on the type of the endpoint, a set of implicit rules that
+override installed policies might be enforced.
+
+On default endpoints, the following set is enforced and checked before
+any user-supplied policy is checked.
+
+ * Privileged connections always override any installed policy. Those
+ connections could easily install their own policies, so there is no
+ reason to enforce installed policies.
+ * Connections can always talk to connections of the same user. This
+ includes broadcast messages.
+
+Custom endpoints have stricter policies. The following rules apply:
+
+ * Policy rules are always enforced, even if the connection is a privileged
+ connection.
+ * Policy rules are always enforced for TALK access, even if both ends are
+ running under the same user. This includes broadcast messages.
+ * To restrict the set of names that can be seen, endpoint policies can
+ install "SEE" policies.
+
+
+12. Pool
+===============================================================================
+
+A pool for data received from the kernel is installed for every connection of
+the bus, and is sized according to the information stored in the
+KDBUS_ITEM_BLOOM_PARAMETER item that is returned by KDBUS_CMD_HELLO.
+
+The pool is written to by the kernel when one of the following ioctls is issued:
+
+ * KDBUS_CMD_HELLO, to receive details about the bus the connection was made to
+ * KDBUS_CMD_RECV, to receive a message
+ * KDBUS_CMD_NAME_LIST, to dump the name registry
+ * KDBUS_CMD_CONN_INFO, to retrieve information on a connection
+
+The offsets returned by either one of the aforementioned ioctls describe offsets
+inside the pool. In order to make the slice available for subsequent calls,
+KDBUS_CMD_FREE has to be called on the offset.
+
+To access the memory, the caller is expected to mmap() it to its task, like
s/to its task//
+this:
+
+ /*
+ * POOL_SIZE has to be a multiple of PAGE_SIZE, and it must match the
+ * value that was previously returned through the KDBUS_ITEM_BLOOM_PARAMETER
+ * item field when the KDBUS_CMD_HELLO ioctl returned.
+ */
+
+ buf = mmap(NULL, POOL_SIZE, PROT_READ, MAP_SHARED, conn_fd, 0);
+
+Alternatively, instead of mapping the entire pool buffer, only parts of it can
+be mapped. The length of the response is returned by the kernel along with the
+offset for each of the ioctls listed above.
+
+
+13. Metadata
+===============================================================================
+
+kdbus records data about the system in certain situations. Such metadata can
+refer to the currently active process (creds, PIDs, current user groups, process
+names and its executable path, cgroup membership, capabilities, security label
+and audit information), connection information (description string, currently
+owned names) and the timestamp.
+
+Metadata is collected in the following circumstances:
+
+ * When a bus is created (KDBUS_CMD_MAKE), information about the calling task
+ is collected. This data is returned by the kernel via the
+ KDBUS_CMD_BUS_CREATOR_INFO call-
s/-/./
+
+ * When a connection is created (KDBUS_CMD_HELLO), information about the
+ calling task is collected. Alternatively, a privileged connection may
+ provide 'faked' information about credentials, PIDs and a security labels
s/a security/security/
+ which will be taken instead. This data is returned by the kernel as
+ information on a connection (KDBUS_CMD_CONN_INFO).
+
+ * When a message is sent (KDBUS_CMD_SEND), information about the sending task
+ and the sending connection are collected. This metadata will be attached
+ to the message when it arrives in the receiver's pool. If the connection
+ sending the message installed faked credentials (see above), the message
+ will not be augmented by any information about the currently sending task.
+
+Which metadata items are actually delivered depends on the following sets and
+masks:
+
+ (a) the system-wide kmod creds mask (module parameter 'attach_flags_mask')
+ (b) the per-connection send creds mask, set by the connecting client
+ (c) the per-connection receive creds mask, set by the connecting client
+ (d) the per-bus minimal creds mask, set by the bus creator
+ (e) the per-bus owner creds mask, set by the bus creator
+ (f) the mask specified when querying creds of a bus peer
+ (g) the mask specified when querying creds of a bus owner
+
+With the following rules:
+
+ [1] The creds attached to messages are determined as (a & b & c).
+ [2] When connecting to a bus (KDBUS_CMD_HELLO), and (~b & d) != 0, the call
+ will fail, the connection is refused.
+ [3] When querying creds of a bus peer, the creds returned are (a & b & f)
+ [4] When querying creds of a bus owner, the creds returned are (a & e & g)
+ [5] When creating a new bus, and (d & ~a) != 0, then bus creation will fail
+
+Hence, userspace might not always get all requested metadata items that it
+requested. Code must be written so that it can cope with this fact.
+
+
+13.1 Known item types
+---------------------
+
+The following attach flags are currently supported.
+
+ KDBUS_ATTACH_TIMESTAMP
+ Attaches an item of type KDBUS_ITEM_TIMESTAMP which contains both the
+ monotonic and the realtime timestamp, taken when the message was
+ processed on the kernel side.
+
+ KDBUS_ATTACH_CREDS
+ Attaches an item of type KDBUS_ITEM_CREDS, containing credentials as
+ described in struct kdbus_creds: the user and group IDs in the usual four
+ flavors: real, effective, saved and file-system related.
+
+ KDBUS_ATTACH_PIDS
+ Attaches an item of type KDBUS_ITEM_PIDS, containing information on the
+ process. In particular, the PID (process ID), TID (thread ID), and PPID
+ (PID of the parent process).
+
+ KDBUS_ATTACH_AUXGROUPS
+ Attaches an item of type KDBUS_ITEM_AUXGROUPS, containing a dynamic
+ number of auxiliary groups the sending task was a member of.
+
+ KDBUS_ATTACH_NAMES
+ Attaches items of type KDBUS_ITEM_OWNED_NAME, one for each name the sending
+ connection currently owns. The name and flags are stored in kdbus_item.name
+ for each of them.
+
+ KDBUS_ATTACH_TID_COMM [*]
+ Attaches an items of type KDBUS_ITEM_TID_COMM, transporting the sending
+ task's 'comm', for the tid. The string is stored in kdbus_item.str.
+
+ KDBUS_ATTACH_PID_COMM [*]
+ Attaches an items of type KDBUS_ITEM_PID_COMM, transporting the sending
+ task's 'comm', for the pid. The string is stored in kdbus_item.str.
+
+ KDBUS_ATTACH_EXE [*]
+ Attaches an item of type KDBUS_ITEM_EXE, containing the path to the
+ executable of the sending task, stored in kdbus_item.str.
+
+ KDBUS_ATTACH_CMDLINE [*]
+ Attaches an item of type KDBUS_ITEM_CMDLINE, containing the command line
+ arguments of the sending task, as an array of strings, stored in
+ kdbus_item.str.
+
+ KDBUS_ATTACH_CGROUP
+ Attaches an item of type KDBUS_ITEM_CGROUP with the task's cgroup path.
+
+ KDBUS_ATTACH_CAPS
+ Attaches an item of type KDBUS_ITEM_CAPS, carrying sets of capabilities
+ that should be accessed via kdbus_item.caps.caps. Also, userspace should
+ be written in a way that it takes kdbus_item.caps.last_cap into account,
+ and derive the number of sets and rows from the item size and the reported
+ number of valid capability bits.
+
+ KDBUS_ATTACH_SECLABEL
+ Attaches an item of type KDBUS_ITEM_SECLABEL, which contains the SELinux
+ security label of the sending task. SELinux and other MACs might want to
+ do additional per-service security checks. For example, a service manager
+ might want to check the security label of a service file against the
+ security label of the client process checking the SELinux database before
+ allowing access. The label can be accessed via kdbus_item->str.
+
+ KDBUS_ATTACH_AUDIT
+ Attaches an item of type KDBUS_ITEM_AUDIT, which contains the audit
+ sessionid and loginuid of the sending task. Access via kdbus_item->audit.
+
+ KDBUS_ATTACH_CONN_DESCRIPTION
+ Attaches an item of type KDBUS_ITEM_CONN_DESCRIPTION that contains a
+ descriptive string of the sending peer. That string can be supplied
+ during HELLO by attaching an item of type KDBUS_ITEM_CONN_DESCRIPTION.
+
+
+[*] Note that the content stored in these items can easily be tampered by
+ the sending tasks. Therefore, they should NOT be used for any sort of
+ security relevant assumptions. The only reason why they are transmitted is
+ to let receivers know about details that were set when metadata was
+ collected, even though the task they were collected from is not active any
+ longer when the items are received.
+
+
+13.2 Metadata and namespaces
+----------------------------
+
+Metadata such as PIDs, UIDs or GIDs are automatically translated to the
+namespaces of the task that receives them.
+
+
+14. Error codes
+===============================================================================
+
+Below is a list of error codes that might be returned by the individual
+ioctl commands. The list focuses on the return values from kdbus code itself,
+and might not cover those of all kernel internal functions.
+
+For all ioctls:
+
+ -ENOMEM The kernel memory is exhausted
+ -ENOTTY Illegal ioctl command issued for the file descriptor
Why ENOTTY here, rather than EINVAL? The latter is, I beleive, the usual
ioctl() error for invalid commands, I believe (If you keep ENOTTY, add an
explanation in this document.)
+ -ENOSYS The requested functionality is not available
Maybe add here an explanation or examples of why the functionality is
not available?
+ -EINVAL Unsupported item attached to command
+
+For all ioctls that carry a struct as payload:
+
+ -EFAULT The supplied data pointer was not 64-bit aligned, or was
+ inaccessible from the kernel side.
+ -EINVAL The size inside the supplied struct was smaller than expected
+ -EMSGSIZE The size inside the supplied struct was bigger than expected
Why two different errors for smaller and larger than expected? (If you keep things this
way, pelase explain the reason in this document.)
+ -ENAMETOOLONG A supplied name is larger than the allowed maximum size
+
+For KDBUS_CMD_BUS_MAKE:
+
+ -EINVAL The flags supplied in the kdbus_cmd_make struct are invalid or
+ the supplied name does not start with the current uid and a '-'
+ -EEXIST A bus of that name already exists
+ -ESHUTDOWN The domain for the bus is already shut down
+ -EMFILE The maximum number of buses for the current user is exhausted
+
+For KDBUS_CMD_ENDPOINT_MAKE:
+
+ -EPERM The calling user is not privileged (see Terminology)
+ -EINVAL The flags supplied in the kdbus_cmd_make struct are invalid
+ -EEXIST An endpoint of that name already exists
+
+For KDBUS_CMD_HELLO:
+
+ -EFAULT The supplied pool size was 0 or not a multiple of the page size
+ -EINVAL The flags supplied in the kdbus_cmd_make struct are invalid, or
+ an illegal combination of KDBUS_HELLO_MONITOR,
+ KDBUS_HELLO_ACTIVATOR and KDBUS_HELLO_POLICY_HOLDER was passed
+ in the flags, or an invalid set of items was supplied
+ -ECONNREFUSED The attach_flags_send field did not satisfy the requirements of
+ the bus
+ -EPERM An KDBUS_ITEM_CREDS items was supplied, but the current user is
+ not privileged
+ -ESHUTDOWN The bus has already been shut down
+ -EMFILE The maximum number of connection on the bus has been reached
s/connection/connections/
+ -EOPNOTSUPP The endpoint does not support the connection flags
+ supplied in the kdbus_cmd_hello struct
+
+For KDBUS_CMD_BYEBYE:
+
+ -EALREADY The connection has already been shut down
+ -EBUSY There are still messages queued up in the connection's pool
+
+For KDBUS_CMD_SEND:
+
+ -EOPNOTSUPP The connection is not an ordinary connection, or the passed
+ file descriptors are either kdbus handles or unix domain
+ sockets. Both are currently unsupported
+ -EINVAL The submitted payload type is KDBUS_PAYLOAD_KERNEL,
+ KDBUS_MSG_EXPECT_REPLY was set without timeout or cookie
+ values, KDBUS_MSG_SYNC_REPLY was set without
+ KDBUS_MSG_EXPECT_REPLY, an invalid item was supplied,
+ src_id was != 0 and different from the current connection's ID,
s/src_id was != 0 and different/src_id was nonzero and was different/
+ a supplied memfd had a size of 0, a string was not properly
+ null-terminated
+ -ENOTUNIQ The supplied destination is KDBUS_DST_ID_BROADCAST, a file
+ descriptor was passed, KDBUS_MSG_EXPECT_REPLY was set,
+ or a timeout was given for a broadcast message
+ -E2BIG Too many items
+ -EMSGSIZE The size of the message header and items or the payload vector
+ is too big.
+ -EEXIST Multiple KDBUS_ITEM_FDS, KDBUS_ITEM_BLOOM_FILTER or
+ KDBUS_ITEM_DST_NAME items were supplied
+ -EBADF The supplied KDBUS_ITEM_FDS or KDBUS_MSG_PAYLOAD_MEMFD items
+ contained an illegal file descriptor
+ -EMEDIUMTYPE The supplied memfd is not a sealed kdbus memfd
+ -EMFILE Too many file descriptors inside a KDBUS_ITEM_FDS
+ -EBADMSG An item had illegal size, both a dst_id and a
+ KDBUS_ITEM_DST_NAME was given, or both a name and a bloom
+ filter was given
+ -ETXTBSY The supplied kdbus memfd file cannot be sealed or the seal
+ was removed, because it is shared with other processes or
+ still mmap()ed
+ -ECOMM A peer does not accept the file descriptors addressed to it
+ -EFAULT The supplied bloom filter size was not 64-bit aligned
+ -EDOM The supplied bloom filter size did not match the bloom filter
+ size of the bus
+ -EDESTADDRREQ dst_id was set to KDBUS_DST_ID_NAME, but no KDBUS_ITEM_DST_NAME
+ was attached
+ -ESRCH The name to look up was not found in the name registry
+ -EADDRNOTAVAIL KDBUS_MSG_NO_AUTO_START was given but the destination
+ connection is an activator.
+ -ENXIO The passed numeric destination connection ID couldn't be found,
+ or is not connected
+ -ECONNRESET The destination connection is no longer active
+ -ETIMEDOUT Timeout while synchronously waiting for a reply
+ -EINTR System call interrupted while synchronously waiting for a reply
+ -EPIPE When sending a message, a synchronous reply from the receiving
+ connection was expected but the connection died before
+ answering
+ -ENOBUFS Too many pending messages on the receiver side
+ -EREMCHG Both a well-known name and a unique name (ID) was given, but
+ the name is not currently owned by that connection.
+ -EXFULL The memory pool of the receiver is full
+ -EREMOTEIO While synchronously waiting for a reply, the remote peer
+ failed with an I/O error.
+
+For KDBUS_CMD_RECV:
+
+ -EINVAL Invalid flags or offset
+ -EAGAIN No message found in the queue
+ -ENOMSG No message of the requested priority found
+ -EOVERFLOW Broadcast messages have been lost
+
+For KDBUS_CMD_FREE:
+
+ -ENXIO No pool slice found at given offset
+ -EINVAL Invalid flags provided, the offset is valid, but the user is
+ not allowed to free the slice. This happens, for example, if
+ the offset was retrieved with KDBUS_RECV_PEEK.
It would be easier to read if this was written to clarify that there are
two distinct error cases:
-EINVAL Invalid flags provided.
-EINVAL The offset is valid, but the user is
not allowed to free the slice. This happens, for example, if
the offset was retrieved with KDBUS_RECV_PEEK.
+For KDBUS_CMD_NAME_ACQUIRE:
+
+ -EINVAL Illegal command flags, illegal name provided, or an activator
+ tried to acquire a second name
+ -EPERM Policy prohibited name ownership
+ -EALREADY Connection already owns that name
+ -EEXIST The name already exists and can not be taken over
+ -E2BIG The maximum number of well-known names per connection
+ is exhausted
+ -ECONNRESET The connection was reset during the call
+
+For KDBUS_CMD_NAME_RELEASE:
+
+ -EINVAL Invalid command flags, or invalid name provided
+ -ESRCH Name is not found found in the registry
+ -EADDRINUSE Name is owned by a different connection and can't be released
+
+For KDBUS_CMD_NAME_LIST:
+
+ -EINVAL Invalid flags
+ -ENOBUFS No available memory in the connection's pool.
+
+For KDBUS_CMD_CONN_INFO:
+
+ -EINVAL Invalid flags, or neither an ID nor a name was provided,
+ or the name is invalid.
+ -ESRCH Connection lookup by name failed
+ -ENXIO No connection with the provided connection ID found
+
+For KDBUS_CMD_CONN_UPDATE:
+
+ -EINVAL Illegal flags or items
+ -EOPNOTSUPP Operation not supported by connection.
+ -E2BIG Too many policy items attached
+ -EINVAL Wildcards submitted in policy entries, or illegal sequence
+ of policy items
+
+For KDBUS_CMD_ENDPOINT_UPDATE:
+
+ -E2BIG Too many policy items attached
+ -EINVAL Invalid flags, or wildcards submitted in policy entries,
+ or illegal sequence of policy items
+
+For KDBUS_CMD_MATCH_ADD:
+
+ -EINVAL Illegal flags or items
+ -EDOM Illegal bloom filter size
+ -EMFILE Too many matches for this connection
+
+For KDBUS_CMD_MATCH_REMOVE:
+
+ -EINVAL Illegal flags
+ -ENOENT A match entry with the given cookie could not be found.
+
+
+15. Internal object relations
+===============================================================================
+
+This is a simplified outline of the internal kdbus object relations, for
+those interested in the inner life of the driver implementation.
+
+From the a mount point's (domain's) perspective:
+
+struct kdbus_domain
+ |» struct kdbus_domain_user *user (many, owned)
+ '» struct kdbus_node node (embedded)
+ |» struct kdbus_node children (many, referenced)
+ |» struct kdbus_node *parent (pinned)
+ '» struct kdbus_bus (many, pinned)
+ |» struct kdbus_node node (embedded)
+ '» struct kdbus_ep (many, pinned)
+ |» struct kdbus_node node (embedded)
+ |» struct kdbus_bus *bus (pinned)
+ |» struct kdbus_conn conn_list (many, pinned)
+ | |» struct kdbus_ep *ep (pinned)
+ | |» struct kdbus_name_entry *activator_of (owned)
+ | |» struct kdbus_match_db *match_db (owned)
+ | |» struct kdbus_meta *meta (owned)
+ | |» struct kdbus_match_db *match_db (owned)
+ | | '» struct kdbus_match_entry (many, owned)
+ | |
+ | |» struct kdbus_pool *pool (owned)
+ | | '» struct kdbus_pool_slice *slices (many, owned)
+ | | '» struct kdbus_pool *pool (pinned)
+ | |
+ | |» struct kdbus_domain_user *user (pinned)
+ | `» struct kdbus_queue_entry entries (many, embedded)
+ | |» struct kdbus_pool_slice *slice (pinned)
+ | |» struct kdbus_conn_reply *reply (owned)
+ | '» struct kdbus_domain_user *user (pinned)
+ |
+ '» struct kdbus_domain_user *user (pinned)
+ '» struct kdbus_policy_db policy_db (embedded)
+ |» struct kdbus_policy_db_entry (many, owned)
+ | |» struct kdbus_conn (pinned)
+ | '» struct kdbus_ep (pinned)
+ |
+ '» struct kdbus_policy_db_cache_entry (many, owned)
+ '» struct kdbus_conn (pinned)
+
+
+For the life-time of a file descriptor derived from calling open() on a file
+inside the mount point:
+
+struct kdbus_handle
+ |» struct kdbus_meta *meta (owned)
+ |» struct kdbus_ep *ep (pinned)
+ |» struct kdbus_conn *conn (owned)
+ '» struct kdbus_ep *ep (owned)
From: Daniel Mack <daniel@zonque.org> Date: 2015-01-20 17:50:55
Hi Michael,
Thanks a lot for for intense review of the documentation. Much appreciated.
I've addressed all but the below issues, following your suggestions.
On 01/20/2015 02:58 PM, Michael Kerrisk (man-pages) wrote:
quoted
+ and KDBUS_CMD_ENDPOINT_MAKE (see above).
+
+ Following items are expected for KDBUS_CMD_BUS_MAKE:
+ KDBUS_ITEM_MAKE_NAME
+ Contains a string to identify the bus name.
So, up to here, I've seen no definition of 'kdbus_item', which leaves me
asking questions like: what subfield is KDBUS_ITEM_MAKE_NAME stored in?
which subfield holds the pointer to the string?
Somewhere earlier, kdbus_item needs to be exaplained in more detail,
I think.
Hmm, you're quoting text from section 5, and section 4 actually
describes the concept of items quite well I believe?
quoted
+ __s64 priority;
+ With KDBUS_RECV_USE_PRIORITY set in flags, receive the next message in
+ the queue with at least the given priority. If no such message is waiting
+ in the queue, -ENOMSG is returned.
###
How do I simply select the highest priority message, without knowing what
its priority is?
The wording is indeed unclear here. KDBUS_RECV_USE_PRIORITY causes the
messages to be dequeued by their priority. The 'priority' field is
simply a filter that request a minimum priority. By setting this field
to the highest possible value, you effectively bypass the filter. I've
added a better description now.
quoted
+ -ENOMEM The kernel memory is exhausted
+ -ENOTTY Illegal ioctl command issued for the file descriptor
Why ENOTTY here, rather than EINVAL? The latter is, I beleive, the usual
ioctl() error for invalid commands, I believe (If you keep ENOTTY, add an
explanation in this document.)
Hmm, no. -ENOTTY is commonly used as return code when calling ioctls
that can't be handled by the FDs they're called on. 'man errno(3)' even
states: "ENOTTY Inappropriate I/O control operation (POSIX.1)".
quoted
+ -EINVAL Unsupported item attached to command
+
+For all ioctls that carry a struct as payload:
+
+ -EFAULT The supplied data pointer was not 64-bit aligned, or was
+ inaccessible from the kernel side.
+ -EINVAL The size inside the supplied struct was smaller than expected
+ -EMSGSIZE The size inside the supplied struct was bigger than expected
Why two different errors for smaller and larger than expected? (If you keep things this
way, pelase explain the reason in this document.)
Providing a struct that is smaller than the minimum doesn't give the
ioctl a valid set of information to process the request. Hence, I think
-EINVAL is appropriate. However, -EMSGSIZE is something that users might
hit when they make message payloads too big, and I think it's good to
have a change to distinguish the two cases in error handling. I added
something in the document now.
Again, thanks a lot for reading the documentation so accurately!
Daniel
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-21 08:58:20
Hi Daniel,
On 01/20/2015 06:50 PM, Daniel Mack wrote:
Hi Michael,
Thanks a lot for for intense review of the documentation. Much appreciated.
I've addressed all but the below issues, following your suggestions.
Are your changes already visible somewhere?
On 01/20/2015 02:58 PM, Michael Kerrisk (man-pages) wrote:
quoted
quoted
+ and KDBUS_CMD_ENDPOINT_MAKE (see above).
+
+ Following items are expected for KDBUS_CMD_BUS_MAKE:
+ KDBUS_ITEM_MAKE_NAME
+ Contains a string to identify the bus name.
So, up to here, I've seen no definition of 'kdbus_item', which leaves me
asking questions like: what subfield is KDBUS_ITEM_MAKE_NAME stored in?
which subfield holds the pointer to the string?
Somewhere earlier, kdbus_item needs to be exaplained in more detail,
I think.
Hmm, you're quoting text from section 5, and section 4 actually
describes the concept of items quite well I believe?
Well, Section 4 is pretty short. My point is that most of the various
blob formats (e.g., kdbus_pids, kdbus_caps, kdbus_memfd) are not
documented in kdbus.txt. They all should be, IMO.
quoted
quoted
+ __s64 priority;
+ With KDBUS_RECV_USE_PRIORITY set in flags, receive the next message in
+ the queue with at least the given priority. If no such message is waiting
+ in the queue, -ENOMSG is returned.
###
How do I simply select the highest priority message, without knowing what
its priority is?
The wording is indeed unclear here. KDBUS_RECV_USE_PRIORITY causes the
messages to be dequeued by their priority. The 'priority' field is
simply a filter that request a minimum priority. By setting this field
to the highest possible value, you effectively bypass the filter. I've
added a better description now.
Thanks for the clarification.
quoted
quoted
+ -ENOMEM The kernel memory is exhausted
+ -ENOTTY Illegal ioctl command issued for the file descriptor
Why ENOTTY here, rather than EINVAL? The latter is, I beleive, the usual
ioctl() error for invalid commands, I believe (If you keep ENOTTY, add an
explanation in this document.)
Hmm, no. -ENOTTY is commonly used as return code when calling ioctls
that can't be handled by the FDs they're called on. 'man errno(3)' even
states: "ENOTTY Inappropriate I/O control operation (POSIX.1)".
Okay.
quoted
quoted
+ -EINVAL Unsupported item attached to command
+
+For all ioctls that carry a struct as payload:
+
+ -EFAULT The supplied data pointer was not 64-bit aligned, or was
+ inaccessible from the kernel side.
+ -EINVAL The size inside the supplied struct was smaller than expected
+ -EMSGSIZE The size inside the supplied struct was bigger than expected
Why two different errors for smaller and larger than expected? (If you keep things this
way, pelase explain the reason in this document.)
Providing a struct that is smaller than the minimum doesn't give the
ioctl a valid set of information to process the request. Hence, I think
-EINVAL is appropriate. However, -EMSGSIZE is something that users might
hit when they make message payloads too big, and I think it's good to
have a change to distinguish the two cases in error handling. I added
something in the document now.
Thanks.
Again, thanks a lot for reading the documentation so accurately!
Hmm, you're quoting text from section 5, and section 4 actually
describes the concept of items quite well I believe?
Well, Section 4 is pretty short. My point is that most of the various
blob formats (e.g., kdbus_pids, kdbus_caps, kdbus_memfd) are not
documented in kdbus.txt. They all should be, IMO.
Okay, I'll add some text about them.
Best regards,
Daniel
From: Michael Kerrisk (man-pages) <hidden> Date: 2015-01-21 09:08:22
Daniel,
On 01/20/2015 02:58 PM, Michael Kerrisk (man-pages) wrote:
On 01/16/2015 08:16 PM, Greg Kroah-Hartman wrote:
quoted
From: Daniel Mack <redacted>
[...]
quoted
+offset field contains the location of the new message inside the receiver's
+pool. The message is stored as struct kdbus_msg at this offset, and can be
+interpreted with the semantics described above.
+
+Also, if the connection allowed for file descriptor to be passed
+(KDBUS_HELLO_ACCEPT_FD), and if the message contained any, they will be
+installed into the receiving process after the KDBUS_CMD_RECV ioctl returns.
From: Daniel Mack <daniel@zonque.org> Date: 2015-01-21 09:12:27
On 01/21/2015 10:07 AM, Michael Kerrisk (man-pages) wrote:
On 01/20/2015 02:58 PM, Michael Kerrisk (man-pages) wrote:
quoted
quoted
+Also, if the connection allowed for file descriptor to be passed
+(KDBUS_HELLO_ACCEPT_FD), and if the message contained any, they will be
+installed into the receiving process after the KDBUS_CMD_RECV ioctl returns.
###
"after"??? When exactly?
By the way, what was the answer to this question?
I've corrected the wording on this. The file descriptors are installed
when the RECV ioctl is called, so they are ready to use when the call
returns.
Thanks,
Daniel
From: Ahmed S. Darwish <hidden> Date: 2015-01-23 06:28:29
On Fri, Jan 16, 2015 at 11:16:05AM -0800, Greg Kroah-Hartman wrote:
From: Daniel Mack <redacted>
kdbus is a system for low-latency, low-overhead, easy to use
interprocess communication (IPC).
The interface to all functions in this driver is implemented via ioctls
on files exposed through a filesystem called 'kdbusfs'. The default
mount point of kdbusfs is /sys/fs/kdbus.
Pardon my ignorance, but we've always been told that adding
new ioctl()s to the kernel is a very big no-no. But given
the seniority of the folks stewarding this kdbus effort,
there must be a good rationale ;-)
So, can the rationale behind introducing new ioctl()s be
further explained? It would be even better if it's included
in the documentation patch itself.
Thanks,
Darwish