From: Pablo Neira Ayuso <hidden> Date: 2016-09-13 17:24:08
On Tue, Sep 13, 2016 at 03:31:20PM +0200, Daniel Mack wrote:
Hi,
On 09/13/2016 01:56 PM, Pablo Neira Ayuso wrote:
quoted
On Mon, Sep 12, 2016 at 06:12:09PM +0200, Daniel Mack wrote:
quoted
This is v5 of the patch set to allow eBPF programs for network
filtering and accounting to be attached to cgroups, so that they apply
to all sockets of all tasks placed in that cgroup. The logic also
allows to be extendeded for other cgroup based eBPF logic.
1) This infrastructure can only be useful to systemd, or any similar
orchestration daemon. Look, you can only apply filtering policies
to processes that are launched by systemd, so this only works
for server processes.
Sorry, but both statements aren't true. The eBPF policies apply to every
process that is placed in a cgroup, and my example program in 6/6 shows
how that can be done from the command line.
Then you have to explain me how can anyone else than systemd use this
infrastructure?
Also, systemd is able to control userspace processes just fine, and
it not limited to 'server processes'.
My main point is that those processes *need* to be launched by the
orchestrator, which is was refering as 'server processes'.
quoted
For client processes this infrastructure is
*racy*, you have to add new processes in runtime to the cgroup,
thus there will be time some little time where no filtering policy
will be applied. For quality of service, this may be an acceptable
race, but this is aiming to deploy a filtering policy.
That's a limitation that applies to many more control mechanisms in the
kernel, and it's something that can easily be solved with fork+exec.
As long as you have control to launch the processes yes, but this
will not work in other scenarios. Just like cgroup net_cls and friends
are broken for filtering for things that you have no control to
fork+exec.
To use this infrastructure from a non-launcher process, you'll have to
rely on the proc connection to subscribe to new process events, then
echo that pid to the cgroup, and that interface is asynchronous so
*adding new processes to the cgroup is subject to races*.
quoted
2) This aproach looks uninfrastructured to me. This provides a hook
to push a bpf blob at a place in the stack that deploys a filtering
policy that is not visible to others.
That's just as transparent as SO_ATTACH_FILTER. What kind of
introspection mechanism do you have in mind?
SO_ATTACH_FILTER is called from the process itself, so this is a local
filtering policy that you apply to your own process.
In this case, this filtering policy is *global*, other processes with
similar capabilities can get just a bpf blob at best...
[...]
quoted
quoted
After chatting with Daniel Borkmann and Alexei off-list, we concluded
that __dev_queue_xmit() is the place where the egress hooks should live
when eBPF programs need access to the L2 bits of the skb.
3) This egress hook is coming very late, the only reason I find to
place it at __dev_queue_xmit() is that bpf naturally works with
layer 2 information in place. But this new hook is placed in
_everyone's output ath_ that only works for the very specific
usecase I exposed above.
It's about filtering outgoing network packets of applications, and
providing them with L2 information for filtering purposes. I don't think
that's a very specific use-case.
When the feature is not used at all, the added costs on the output path
are close to zero, due to the use of static branches.
*You're proposing a socket filtering facility that hooks layer 2
output path*!
[...]
quoted
I have nothing against systemd or the needs for more
programmability/flexibility in the stack, but I think this needs to
fulfill some requirements to fit into the infrastructure that we have
in the right way.
Well, as I explained already, this patch set results from endless
discussions that went nowhere, about how such a thing can be achieved
with netfilter.
That is only a rough ~30 lines kernel patchset to support this in
netfilter and only one extra input hook, with potential access to
conntrack and better integration with other existing subsystems.
On Tue, Sep 13, 2016 at 07:24:08PM +0200, Pablo Neira Ayuso wrote:
On Tue, Sep 13, 2016 at 03:31:20PM +0200, Daniel Mack wrote:
quoted
Hi,
On 09/13/2016 01:56 PM, Pablo Neira Ayuso wrote:
quoted
On Mon, Sep 12, 2016 at 06:12:09PM +0200, Daniel Mack wrote:
quoted
This is v5 of the patch set to allow eBPF programs for network
filtering and accounting to be attached to cgroups, so that they apply
to all sockets of all tasks placed in that cgroup. The logic also
allows to be extendeded for other cgroup based eBPF logic.
1) This infrastructure can only be useful to systemd, or any similar
orchestration daemon. Look, you can only apply filtering policies
to processes that are launched by systemd, so this only works
for server processes.
Sorry, but both statements aren't true. The eBPF policies apply to every
process that is placed in a cgroup, and my example program in 6/6 shows
how that can be done from the command line.
Then you have to explain me how can anyone else than systemd use this
infrastructure?
Sounds like systemd and bpf phobia combined :)
Jokes aside. I'm puzzled why systemd is even being mentioned here.
Here we use tupperware (our internal container management system) that
is heavily using cgroups and has nothing to do with systemd.
we're working as part of open container initiative, so hopefully soon
all container management systems will benefit from what we're building.
cgroups and bpf are crucial part of this process.
quoted
Also, systemd is able to control userspace processes just fine, and
it not limited to 'server processes'.
My main point is that those processes *need* to be launched by the
orchestrator, which is was refering as 'server processes'.
No experience in systemd, so cannot comment about it,
but that statement is not true for our stuff.
quoted
quoted
For client processes this infrastructure is
*racy*, you have to add new processes in runtime to the cgroup,
thus there will be time some little time where no filtering policy
will be applied. For quality of service, this may be an acceptable
race, but this is aiming to deploy a filtering policy.
That's a limitation that applies to many more control mechanisms in the
kernel, and it's something that can easily be solved with fork+exec.
As long as you have control to launch the processes yes, but this
will not work in other scenarios. Just like cgroup net_cls and friends
are broken for filtering for things that you have no control to
fork+exec.
not true
To use this infrastructure from a non-launcher process, you'll have to
rely on the proc connection to subscribe to new process events, then
echo that pid to the cgroup, and that interface is asynchronous so
*adding new processes to the cgroup is subject to races*.
in general not true either. have you worked with cgroups or just speculating?
*You're proposing a socket filtering facility that hooks layer 2
output path*!
flashback. Not too long ago you were beating drums about netfilter
ingress hook operating at layer 2... sounds like nobody used it
and that was a bad call? Should we remove that netfilter hook then?
Our use case is different from Daniel's.
For us this cgroup+bpf is _not_ for filterting and _not_ for security.
We run a ton of tasks in cgroups that launch all sorts of
things on their own. We need to monitor what they do from networking
point of view. Therefore bpf programs need to monitor the traffic in
particular part of cgroup hierarchy. Not globally and no pass/drop decisions.
The monitoring itself is complicated. Like we need to group and
aggregate within bpf program based on certain bits of ipv6 address
and so on. bpf is only programmable engine that can do this job.
nft is simply not flexible enough to do that.
I'd really love to have an alternative to bpf for such tasks,
but you seem to spend all the energy arguing against bpf whereas
nft still has a lot to be desired.
From: Thomas Graf <tgraf@suug.ch> Date: 2016-09-14 09:04:15
[Sorry for the repost, gmail decided to start sending HTML crap along
overnight for some reason]
On 09/13/16 at 09:42pm, Alexei Starovoitov wrote:
On Tue, Sep 13, 2016 at 07:24:08PM +0200, Pablo Neira Ayuso wrote:
quoted
Then you have to explain me how can anyone else than systemd use this
infrastructure?
Jokes aside. I'm puzzled why systemd is even being mentioned here.
Here we use tupperware (our internal container management system) that
is heavily using cgroups and has nothing to do with systemd.
Just confirming that we are planning to use this decoupled from
systemd as well. I fail to see how this is at all systemd specific.
For us this cgroup+bpf is _not_ for filterting and _not_ for security.
We run a ton of tasks in cgroups that launch all sorts of
things on their own. We need to monitor what they do from networking
point of view. Therefore bpf programs need to monitor the traffic in
particular part of cgroup hierarchy. Not globally and no pass/drop decisions.
+10, although filtering/drop is a valid use case, the really strong
use case is definitely introspection at networking level. Statistics,
monitoring, verification of application correctness, etc.
I don't see why this is at all an either or discussion. If nft wants
cgroups integration similar to this effort, I see no reason why that
should stop this effort.
From: Daniel Mack <daniel@zonque.org> Date: 2016-09-14 11:13:19
Hi Pablo,
On 09/13/2016 07:24 PM, Pablo Neira Ayuso wrote:
On Tue, Sep 13, 2016 at 03:31:20PM +0200, Daniel Mack wrote:
quoted
On 09/13/2016 01:56 PM, Pablo Neira Ayuso wrote:
quoted
On Mon, Sep 12, 2016 at 06:12:09PM +0200, Daniel Mack wrote:
quoted
This is v5 of the patch set to allow eBPF programs for network
filtering and accounting to be attached to cgroups, so that they apply
to all sockets of all tasks placed in that cgroup. The logic also
allows to be extendeded for other cgroup based eBPF logic.
1) This infrastructure can only be useful to systemd, or any similar
orchestration daemon. Look, you can only apply filtering policies
to processes that are launched by systemd, so this only works
for server processes.
Sorry, but both statements aren't true. The eBPF policies apply to every
process that is placed in a cgroup, and my example program in 6/6 shows
how that can be done from the command line.
Then you have to explain me how can anyone else than systemd use this
infrastructure?
I have no idea what makes you think this is limited to systemd. As I
said, I provided an example for userspace that works from the command
line. The same limitation apply as for all other users of cgroups.
My main point is that those processes *need* to be launched by the
orchestrator, which is was refering as 'server processes'.
Yes, that's right. But as I said, this rule applies to many other kernel
concepts, so I don't see any real issue.
quoted
That's a limitation that applies to many more control mechanisms in the
kernel, and it's something that can easily be solved with fork+exec.
As long as you have control to launch the processes yes, but this
will not work in other scenarios. Just like cgroup net_cls and friends
are broken for filtering for things that you have no control to
fork+exec.
Probably, but that's only solvable with rules that store the full cgroup
path then, and do a string comparison (!) for each packet flying by.
quoted
That's just as transparent as SO_ATTACH_FILTER. What kind of
introspection mechanism do you have in mind?
SO_ATTACH_FILTER is called from the process itself, so this is a local
filtering policy that you apply to your own process.
Not necessarily. You can as well do it the inetd way, and pass the
socket to a process that is launched on demand, but do SO_ATTACH_FILTER
+ SO_LOCK_FILTER in the middle. What happens with payload on the socket
is not transparent to the launched binary at all. The proposed cgroup
eBPF solution implements a very similar behavior in that regard.
quoted
It's about filtering outgoing network packets of applications, and
providing them with L2 information for filtering purposes. I don't think
that's a very specific use-case.
When the feature is not used at all, the added costs on the output path
are close to zero, due to the use of static branches.
*You're proposing a socket filtering facility that hooks layer 2
output path*!
As I said, I'm open to discussing that. In order to make it work for L3,
the LL_OFF issues need to be solved, as Daniel explained. Daniel,
Alexei, any idea how much work that would be?
That is only a rough ~30 lines kernel patchset to support this in
netfilter and only one extra input hook, with potential access to
conntrack and better integration with other existing subsystems.
Care to share the patches for that? I'd really like to have a look.
And FWIW, I agree with Thomas - there is nothing wrong with having
multiple options to use for such use-cases.
Thanks,
Daniel
On Wed, Sep 14, 2016 at 01:13:16PM +0200, Daniel Mack wrote:
Hi Pablo,
On 09/13/2016 07:24 PM, Pablo Neira Ayuso wrote:
quoted
On Tue, Sep 13, 2016 at 03:31:20PM +0200, Daniel Mack wrote:
quoted
On 09/13/2016 01:56 PM, Pablo Neira Ayuso wrote:
quoted
On Mon, Sep 12, 2016 at 06:12:09PM +0200, Daniel Mack wrote:
quoted
This is v5 of the patch set to allow eBPF programs for network
filtering and accounting to be attached to cgroups, so that they apply
to all sockets of all tasks placed in that cgroup. The logic also
allows to be extendeded for other cgroup based eBPF logic.
1) This infrastructure can only be useful to systemd, or any similar
orchestration daemon. Look, you can only apply filtering policies
to processes that are launched by systemd, so this only works
for server processes.
Sorry, but both statements aren't true. The eBPF policies apply to every
process that is placed in a cgroup, and my example program in 6/6 shows
how that can be done from the command line.
Then you have to explain me how can anyone else than systemd use this
infrastructure?
I have no idea what makes you think this is limited to systemd. As I
said, I provided an example for userspace that works from the command
line. The same limitation apply as for all other users of cgroups.
So, at least in my work, we have Mesos, but on nearly every machine that Mesos
runs, people also have systemd. Now, there's recently become a bit of a battle
of ownership of things like cgroups on these machines. We can usually solve it
by nesting under systemd cgroups, and thus so far we've avoided making too many
systemd-specific concessions.
The reason this works (mostly), is because everything we touch has a sense of
nesting, where we can apply policy at a place lower in the hierarchy, and yet
systemd's monitoring and policy still stays in place.
Now, with this patch, we don't have that, but I think we can reasonably add some
flag like "no override" when applying policies, or alternatively something like
"no new privileges", to prevent children from applying policies that override
top-level policy. I realize there is a speed concern as well, but I think for
people who want nested policy, we're willing to make the tradeoff. The cost
of traversing a few extra pointers still outweighs the overhead of network
namespaces, iptables, etc.. for many of us.
What do you think Daniel?
quoted
My main point is that those processes *need* to be launched by the
orchestrator, which is was refering as 'server processes'.
Yes, that's right. But as I said, this rule applies to many other kernel
concepts, so I don't see any real issue.
Also, cgroups have become such a big part of how applications are managed
that many of us have solved this problem.
quoted
quoted
That's a limitation that applies to many more control mechanisms in the
kernel, and it's something that can easily be solved with fork+exec.
As long as you have control to launch the processes yes, but this
will not work in other scenarios. Just like cgroup net_cls and friends
are broken for filtering for things that you have no control to
fork+exec.
Probably, but that's only solvable with rules that store the full cgroup
path then, and do a string comparison (!) for each packet flying by.
quoted
quoted
That's just as transparent as SO_ATTACH_FILTER. What kind of
introspection mechanism do you have in mind?
SO_ATTACH_FILTER is called from the process itself, so this is a local
filtering policy that you apply to your own process.
Not necessarily. You can as well do it the inetd way, and pass the
socket to a process that is launched on demand, but do SO_ATTACH_FILTER
+ SO_LOCK_FILTER in the middle. What happens with payload on the socket
is not transparent to the launched binary at all. The proposed cgroup
eBPF solution implements a very similar behavior in that regard.
It would be nice to be able to see whether or not a filter is attached to a
cgroup, but given this is going through syscalls, at least introspection
is possible as opposed to something like netlink.
quoted
quoted
It's about filtering outgoing network packets of applications, and
providing them with L2 information for filtering purposes. I don't think
that's a very specific use-case.
When the feature is not used at all, the added costs on the output path
are close to zero, due to the use of static branches.
*You're proposing a socket filtering facility that hooks layer 2
output path*!
As I said, I'm open to discussing that. In order to make it work for L3,
the LL_OFF issues need to be solved, as Daniel explained. Daniel,
Alexei, any idea how much work that would be?
quoted
That is only a rough ~30 lines kernel patchset to support this in
netfilter and only one extra input hook, with potential access to
conntrack and better integration with other existing subsystems.
Care to share the patches for that? I'd really like to have a look.
And FWIW, I agree with Thomas - there is nothing wrong with having
multiple options to use for such use-cases.
Right now, for containers, we have netfilter and network namespaces.
There's a lot of performance overhead that comes with this. Not only
that, but iptables doesn't really have a simple way of usage by
automated infrastructure. We (firewalld, systemd, dockerd, mesos)
end up fighting with one another for ownership over firewall rules.
Although, I have problems with this approach, I think that it's
a good baseline where we can have top level owned by systemd,
docker underneath that, and Mesos underneath that. We can add
additional hooks for things like Checmate and Landlock, and
with a little more work, we can do compositition, solving
all of our problems.