It might be interesting to see that start/end address of the 256K-item (256
* 1024 * 2240 = 560 MB) mbuf memory pool. Maybe it's the first size that
straddles two 1GB pages.

Perhaps you should try a tool that reports cache misses, TLB misses, and
related statistics. I don't know much about this area, yet, but this looks
like a good starting place:

http://software.intel.com/en-us/articles/intel-performance-counter-monitor-a-better-way-to-measure-cpu-utilization

--
Regards,
Robert



On Fri, Sep 20, 2013 at 2:48 AM, Dmitry Vyal [off-list ref] wrote:

On 09/19/2013 11:39 PM, Robert Sanford wrote:

quoted

Hi Dmitry,

The biggest drop-off seems to be from size 128K to 256K. Are you using
1GB huge pages already (rather than 2MB)?

I would think that it would not use over 1GB until you ask for 512K mbufs
or more.

Hi Robert,

Yes, I've been using 1GB pages for a while. My L3 cache is 20MB and mbufs
are 2240 bytes of size. So something strange indeed happens then we move
from ~200MB to ~400MB. Any ideas?

Regards,
Dmitry