On Mon, Jul 2, 2012 at 6:49 AM, Chris Friesen [off-list ref] wrote:
On 06/29/2012 05:18 PM, David Miller wrote:
quoted
From: Vincent Sanders<redacted>
Date: Sat, 30 Jun 2012 00:12:37 +0100
quoted
I had hoped you would have at least read the opening list where I
outlined the underlying features which explain why none of the
existing IPC match the requirements.
I had hoped that you had read the part we told you last time where
we explained why multicast and "reliable delivery" are fundamentally
incompatible attributes.
We are not creating a full address family in the kernel which exists
for one, and only one, specific and difficult user.
For what it's worth, the company I work for (and a number of other
companies) currently use an out-of-tree datagram multicast messaging
protocol family based on AF_UNIX.
If AF_BUS were to be accepted, it would be essentially trivial for us to
port our existing userspace messaging library to use it instead of our
current protocol family, and we would almost certainly do so.
I'd love to see AF_BUS go in.
Chris Friesen
Hi Chris,
Thanks a lot for your comments and feedback.
We tried different approaches before developing the AF_BUS socket family and one
of them was extending AF_UNIX to support multicast. We posted our patches [1]
and the feedback was that the AF_UNIX code was already a complex and difficult
code to maintain. So, we decided to implement a new family (AF_BUS) that is
orthogonal to the rest of the networking stack and no added complexity nor
performance penalty would pay a user not using our IPC solution.
Looking at netdev archives I saw that you both raised the question about
multicast on unix sockets and post an implementation on early 2003. So if I
understand correctly you are maintaining an out-of-tree solution for around 9
years now.
We developed AF_BUS to improve the performance of the D-Bus IPC system (and our
results show us a 2X speedup) but design it to be as generic as possible so
other users can take advantage of it.
It would be a great help if you can join the discussion and explain the
arguments of your company (and the others companies you were talking about) in
favor of a simpler multicast socket family.
The fact that your company spent lots of engineering resources to maintain an
out-of-tree patch-set for 9 years should raise some eyebrows and convince more
than one people that a simpler local multicast solution is needed on the Linux
kernel (which was one of the reasons why Google also developed Binder I guess).
[1]: https://lkml.org/lkml/2012/2/20/84
[2]: https://lkml.org/lkml/2003/2/27/150
[3]: http://lwn.net/Articles/27001/
Thanks a lot and best regards,
Javier
On 07/02/2012 09:18 AM, Javier Martinez Canillas wrote:
We tried different approaches before developing the AF_BUS socket family and one
of them was extending AF_UNIX to support multicast. We posted our patches [1]
and the feedback was that the AF_UNIX code was already a complex and difficult
code to maintain. So, we decided to implement a new family (AF_BUS) that is
orthogonal to the rest of the networking stack and no added complexity nor
performance penalty would pay a user not using our IPC solution.
That's what I ended up doing as well. In our case it's basically a
stripped-down AF_UNIX with only datagram support, no security, no fd
passing, etc., but with with the addition of multicast and wildcard (for
debugging).
Looking at netdev archives I saw that you both raised the question about
multicast on unix sockets and post an implementation on early 2003. So if I
understand correctly you are maintaining an out-of-tree solution for around 9
years now.
That's correct.
It would be a great help if you can join the discussion and explain the
arguments of your company (and the others companies you were talking about) in
favor of a simpler multicast socket family.
The fact that your company spent lots of engineering resources to maintain an
out-of-tree patch-set for 9 years should raise some eyebrows and convince more
than one people that a simpler local multicast solution is needed on the Linux
kernel (which was one of the reasons why Google also developed Binder I guess).
To be fair, since it was implemented as a separate protocol family the
maintenance burden actually hasn't been large--it's been fairly simple
to port between versions. Also, we do embedded telecom stuff and don't
jump kernel versions all that often. (It's a big headache, requires
coordinating between multiple vendors, etc.)
In our case we typically send small (100-200 byte) messages to a
smallish (1-10) number of listeners, though there are exceptions of
course. Back before I started the original implementation used a
userspace daemon, but it had a number of issues. Originally I was
focussed on the performance gains but I must admit that since then other
factors have made that less of an issue.
Among other things, this messaging is used on some systems to configure
the IP addressing for the system, so it does simplify things to not use
an IP-based protocol for this purpose.
Also, back when I did my original implementation IP multicast wasn't
supported on the loopback device--David, has that changed since then?
If it has, then we probably could figure out a way to make it work using
IP multicast, but I don't know that it would be worth the effort given
the minimal ongoing maintenance costs for our patch.
Chris
On 07/03/2012 10:52 AM, Chris Friesen wrote:
To be fair, since it was implemented as a separate protocol family the
maintenance burden actually hasn't been large--it's been fairly simple
to port between versions. Also, we do embedded telecom stuff and don't
jump kernel versions all that often. (It's a big headache, requires
coordinating between multiple vendors, etc.)
In our case we typically send small (100-200 byte) messages to a
smallish (1-10) number of listeners, though there are exceptions of
course. Back before I started the original implementation used a
userspace daemon, but it had a number of issues. Originally I was
focussed on the performance gains but I must admit that since then other
factors have made that less of an issue.
I should point out that some of the other factors that have been
discussed for AF_BUS also hold true for our implementation:
--strict ordering
--reliable (in our case, if the sender has space in the tx buffer then
messages get to all recipients with buffer space, there are kernel logs
if recipients don't have space)
Also, the fact that it's in the kernel rather than a userspace daemon
reduces priority inversion type issues. Presumably this would apply to
an IP-multicast based solution as well.
One problem that I ran into back when I was experimenting with this
stuff was trying to isolate host-local IP multicast from the rest of the
network. It would be suboptimal to need to set up filtering and such
before being able to use the communication protocol.
Chris