Re: [PATCH net] net/mlx5e: Do not recycle pages from emergency reserve

On Sat, 2017-01-21 at 20:12 +0100, kernel netdev wrote:

Den 21. jan. 2017 7.10 PM skrev "Tom Herbert" [off-list ref]:
        On Thu, Jan 19, 2017 at 11:14 AM, Saeed Mahameed
        [off-list ref] wrote:
        > On Thu, Jan 19, 2017 at 9:03 AM, Eric Dumazet
        [off-list ref] wrote:
        >> From: Eric Dumazet [off-list ref]
        >>
        >> A driver using dev_alloc_page() must not reuse a page
        allocated from
        >> emergency memory reserve.
        >>
        >> Otherwise all packets using this page will be immediately
        dropped,
        >> unless for very specific sockets having SOCK_MEMALLOC bit
        set.
        >>
        >> This issue might be hard to debug, because only a fraction
        of received
        >> packets would be dropped.
        >
        > Hi Eric,
        >
        > When you say reuse, you mean point to the same page from
        several SKBs ?
        >
        > Because in our page cache implementation we don't reuse
        pages that
        > already passed to the stack,
        > we just keep them in the page cache until the ref count drop
        back to
        > one, so we recycle them (i,e they will be re-used only when
        no one
        > else is using them).
        >
        
        Saeed,
        
        Speaking of the mlx page cache can we remove this or a least
        make it
        optional to use. It is another example of complex
        functionality being
        put into drivers that makes things like backports more
        complicated and
        provide at best some marginal value. In the case of the mlx5e
        cache
        code the results from pktgen really weren't very impressive in
        the
        first place. Also, the cache suffers from HOL blocking where
        we can
        block the whole cache due to an outstanding reference on just
        one page
        (something that you wouldn't see in pktgen but is likely to
        happen in
        real applications).


(Send from phone in car)


To Tom, have you measured the effect of this page cache? Before
claiming it is ineffective.


My previous measurements show approx 20℅ speedup on a UDP test with
delivery to remote CPU.

I find this a bit strange. When you have time (ie not while driving your
car or during week end) please give more details, for example on message
size. Was it before skb_condense() was added ?

Removing the cache would of cause be a good usecase for speeding up
the page allocator (PCP). Which Mel Gorman and me are working on.
AFAIK current page order0 cost 240 cycles. Mel have reduced til to
180, and without NUMA 150 cycles. And with bulking this can be
amortized to 80 cycles.


--Jesper