On 10/2/19 1:16 AM, David Miller wrote:

From: Ka-Cheong Poon <redacted>
Date: Mon, 30 Sep 2019 02:08:00 -0700

quoted

Currently, RDS calls ib_dma_alloc_coherent() to allocate a large piece
of contiguous DMA coherent memory to store struct rds_header for
sending/receiving packets.  The memory allocated is then partitioned
into struct rds_header.  This is not necessary and can be costly at
times when memory is fragmented.  Instead, RDS should use the DMA
memory pool interface to handle this.

Suggested-by: Håkon Bugge <redacted>
Signed-off-by: Ka-Cheong Poon <redacted>

This is trading a one-time overhead for extra levels of dereferencing
on every single descriptor access in the fast paths.

I do not agree with this tradeoff, please implement this more
reasonably.

The problem with the existing way of pre-allocation is
that when there are a lot of RDS connections, the call to
ib_dma_alloc_coherent() can fail because there are not
enough contiguous memory pages available.  It is causing
problems in production systems.

And the i_{recv|send|_hdrs_dma array dereferencing is done
at send/receive ring initialization and refill.  It is not
done at every access of the header.

Thanks.


-- 
K. Poon
ka-cheong.poon@oracle.com