Re: [PATCH next 1/3] ipvlan: Defer multicast / broadcast processing to a work-queue

On Fri, Apr 24, 2015 at 3:59 PM, Dan Williams [off-list ref] wrote:

On Fri, 2015-04-24 at 15:40 -0700, Mahesh Bandewar wrote:

quoted

On Fri, Apr 24, 2015 at 1:15 PM, Dan Williams [off-list ref] wrote:

quoted

On Thu, 2015-04-23 at 14:29 -0700, Mahesh Bandewar wrote:

quoted

Processing multicast / broadcast in fast path is performance draining
and having more links means more clonning and bringing performance
down further.

Broadcast; in particular, need to be given to all the virtual links.
Earlier tricks of enabling broadcast bit for IPv4 only interfaces are not
really working since it fails autoconf. Which means enabling braodcast
for all the links if protocol specific hacks do not have to be added into
the driver.

This patch defers all (incoming as well as outgoing) multicast traffic to
a work-queue leaving only the unicast traffic in the fast-path. Now if we
need to apply any additional tricks to further reduce the impact of this
(multicast / broadcast) type of traffic, it can be implemented while
processing this work without affecting the fast-path.

These patches appear to work for me for the L2 + DHCP use-case, however
I experienced some quite odd behavior when pinging the ipvlan interface
from another machine.  I did this:

ip link add link eno1 type ipvlan mode l2
ip netns add ipv
ip link set dev ipvlan0 netns ipv
ip netns exec ipv /sbin/dhclient -B -4 -1 -v
-pf /run/dhclient-ipvlan0.pid -C adafdasdfasf ipvlan0
ip netns exec ping 4.2.2.1 <success>

However, when pinging from another machine, I got very inconsistent ping
replies:

64 bytes from 192.168.1.38: icmp_seq=1 ttl=64 time=11.4 ms
64 bytes from 192.168.1.38: icmp_seq=16 ttl=64 time=64.9 ms
64 bytes from 192.168.1.38: icmp_seq=25 ttl=64 time=87.9 ms
64 bytes from 192.168.1.38: icmp_seq=30 ttl=64 time=242 ms
64 bytes from 192.168.1.38: icmp_seq=35 ttl=64 time=40.1 ms
64 bytes from 192.168.1.38: icmp_seq=36 ttl=64 time=60.9 ms

We know that there is that PAUSE frame leak but that should not cause
this behavior if those are present in your network. The sched_work()
which is wrong (as pointed by Eric) especially when the machine is
busy and that might trigger something like this. Almost every 10th -
15th ping packet seems to be processed correctly.

I did get consistent results as opposed what you have shown here, but
will dig some more to see if something obviously wrong here.

quoted

But I cannot reproduce that in a second run (though I haven't rebooted
to test cleanly again).

And oddly, the dhclient process takes a consistent 5% CPU and wireshark
running on eno1 (not even the ipvlan interface) jumps to 100% CPU along
with the dumpcap process taking another 25%, none of which are normal.
This is a 4-core i4790 box, so something is wrong here; is something
holding onto a spinlock for way too long?

But at least it handles the packets ok, so I say progress!  Happy to
help track down the CPU usage issue if you want to give me patches to
test.

Which patch(es) you are referring to?

None that yet exist; simply that if any of the issues I described
triggered thoughts or patches on your end, I'm happy to test them.  I
will try to characterize the issues I have seen more next week and
report back.

OK found the issue! RX frames  are never drained from the queue and
are processed again and again causing the CPU-usage spike you have
observed. I'll integrate the fix into v2

--mahesh..

Dan