Re: [PATCH 0/3]: net: dsa: mt7530: support MT7530 in the MT7621 SoC

On Sun, Dec 16 2018, Florian Fainelli wrote:

On December 16, 2018 3:19:22 PM PST, NeilBrown [off-list ref] wrote:

quoted

On Sun, Dec 16 2018, David Miller wrote:

quoted

From: NeilBrown <neil@brown.name>
Date: Mon, 17 Dec 2018 09:08:54 +1100

quoted

In my 4.4 kernel, the build_skb() call in (the equivalent of)
mtk_poll_rx() takes about 1.2usec and the call to napi_gro_receive()
takes about 3usec.

In my 4.20 kernel, these calls take about 30 and 24 usec

respectively.

quoted

quoted

This easily explains the slowdown.

That's a huge difference.

Nothing jumps out as a possible cause except perhaps retpoline or
something like that.

I'll keep that in mind - thanks.

My guess was CPU-cache invalidation.
I just checked and the other CPU core (there are two - each
hyper-threaded - "other" meaning not the one that handles ethernet
interrupts) gets several thousand "IPI resched" interrupts while
running a 10 second (226MByte) iperf3 receive test.
About 17KB transferred per IPI.
I cannot see where build_skb() would do cache invalidation though.

It doesn't the driver is responsible for that. How is coherency maintained between cores?

I suspect so - yes.  Coherency only needs explicit management with DMA
is used.  This wasn't the problem.

Further investigation showed that the problem was that I had
CONFIG_SLUB_DEBUG set.  That was probably useful in some earlier
debugging exercise, but it clearly isn't useful when performance-testing
the network.
I removed that and I have much nicer numbers - not quite the consistent
900+ that I saw with 4.4, but a lot closer.

Thanks for the encouragement, and sorry of the noise.

NeilBrown

The IPI could be due to receive packet steering, is the MAC multi queue aware on the RX path?
-- 
Florian