On Friday 18 August 2006 04:25, Christoph Lameter wrote:

On Wed, 16 Aug 2006, Andi Kleen wrote:

quoted

That's not true on all NUMA systems (that they have a slow interconnect)
I think on x86-64 I would prefer if it was distributed evenly or maybe
even on the CPU who is finally going to process it.

-Andi "not all NUMA is an Altix"

The Altix NUMA interconnect has the same speed as far as I can recall as
Hypertransport. It is the distance (real physical cable length) that
creates latencies for huge systems. Sadly the Hypertransport is designed
to stay on the motherboard. Hypertransport can only be said to be fast
because its only used for tinzy winzy systems of a few processors. Are
you saying that the design limitations of Hypertransport are an
advantage?

Sorry, didn't want to state anything particular about advantages 
or disadvantages of different interconnects. I just wanted to say
that there are a lot of NUMA systems out there which have a very low
NUMA factor (for whatever reason, including them being quite small)
and that they should be considered for NUMA optimizations too.

So if you really want strict IO placement at least allow an easy way 
to turn it off even when CONFIG_NUMA is defined.

BTW there are large x86-64 NUMA systems that don't use HyperTransport
and have a varying NUMA factor, and also even HyperTransport
based systems have a widely varying NUMA factor depending on machine
size and hop distance (2-8 sockets and larger systems are in development)

So ideal would be something dynamic to turn on/off io placement, maybe based 
on node_distance() again, with the threshold tweakable per architecture?

Also I must say it's still not quite clear to me if it's better to place
network packets on the node the device is connected to or on the 
node which contains the CPU who processes the packet data 
For RX this can be three different nodes in the worst case
(CPU processing is often split on different CPUs between softirq
and user context), for TX  two. Do you have some experience that shows 
that a particular placement is better than the other?

-Andi