On Wed, 8 Nov 2017 09:35:47 +0000
Mel Gorman [off-list ref] wrote:

On Wed, Nov 08, 2017 at 02:42:04PM +0900, Tariq Toukan wrote:

quoted

quoted

quoted

Hi all,

After leaving this task for a while doing other tasks, I got back to it now
and see that the good behavior I observed earlier was not stable.

Recall: I work with a modified driver that allocates a page (4K) per packet
(MTU=1500), in order to simulate the stress on page-allocator in 200Gbps
NICs.
  

There is almost new in the data that hasn't been discussed before. The
suggestion to free on a remote per-cpu list would be expensive as it would
require per-cpu lists to have a lock for safe remote access.  

That's right, but each such lock will be significantly less congested than
the buddy allocator lock.  

That is not necessarily true if all the allocations and frees always happen
on the same CPUs. The contention will be equivalent to the zone lock.
Your point will only hold true if there are also heavy allocation streams
from other CPUs that are unrelated.

quoted

In the flow in subject two cores need to
synchronize (one allocates, one frees).
We also need to evaluate the cost of acquiring and releasing the lock in the
case of no congestion at all.
  

If the per-cpu structures have a lock, there will be a light amount of
overhead. Nothing too severe, but it shouldn't be done lightly either.

quoted

quoted

 However,
I'd be curious if you could test the mm-pagealloc-irqpvec-v1r4 branch
ttps://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git .  It's an
unfinished prototype I worked on a few weeks ago. I was going to revisit
in about a months time when 4.15-rc1 was out. I'd be interested in seeing
if it has a postive gain in normal page allocations without destroying
the performance of interrupt and softirq allocation contexts. The
interrupt/softirq context testing is crucial as that is something that
hurt us before when trying to improve page allocator performance.
  

Yes, I will test that once I get back in office (after netdev conference and
vacation).  

Thanks.

I'll also commit to testing this (when I return home, as Tariq I'm also
in Seoul ATM).

 

quoted

Can you please elaborate in a few words about the idea behind the prototype?
Does it address page-allocator scalability issues, or only the rate of
single core page allocations?  

Short answer -- maybe. All scalability issues or rates of allocation are
context and workload dependant so the question is impossible to answer
for the general case.

Broadly speaking, the patch reintroduces the per-cpu lists being for !irq
context allocations again. The last time we did this, hard and soft IRQ
allocations went through the buddy allocator which couldn't scale and
the patch was reverted. With this patch, it goes through a very large
pagevec-like structure that is protected by a lock but the fast paths
for alloc/free are extremely simple operations so the lock hold times are
very small. Potentially, a development path is that the current per-cpu
allocator is replaced with pagevec-like structures that are dynamically
allocated which would also allow pages to be freed to remote CPU lists

I've had huge success using ptr_ring, as a queue between CPUs, to
minimize cross-CPU cache-line touching.  With the recently accepted BPF
map called "cpumap" used for XDP_REDIRECT.

It's important to handle the two borderline cases in ptr_ring, of the
queue being almost full (default handled in ptr_ring) or almost empty.
Like describe in[1] slide 14:

[1] http://people.netfilter.org/hawk/presentations/NetConf2017_Seoul/XDP_devel_update_NetConf2017_Seoul.pdf

The use of XDP_REDIRECT + cpumap, do expose issues with the page
allocator.  E.g. slide 19 show ixgbe recycle scheme failing, but still
hitting the PCP.  Also notice slide 22 deducing the overhead.  Scale
stressing ptr_ring is showed in extra slides 35-39.

(if we could detect when that is appropriate which is unclear). We could
also drain remote lists without using IPIs. The downside is that the memory
footprint of the allocator would be higher and the size could no longer
be tuned so there would need to be excellent justification for such a move.

I haven't posted the patches properly yet because mmotm is carrying too
many patches as it is and this patch indirectly depends on the contents. I
also didn't write memory hot-remove support which would be a requirement
before merging. I hadn't intended to put further effort into it until I
had some evidence the approach had promise. My own testing indicated it
worked but the drivers I was using for network tests did not allocate
intensely enough to show any major gain/loss.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer