On Tue, 27 Jan 2009 00:10:44 +0100
Eric Dumazet [off-list ref] wrote:

Rick Jones a écrit :

quoted

Folks -

Under:

ftp://ftp.netperf.org/iptable_scaling

can be found netperf results and Caliper profiles for three scenarios on
a 32-core, 1.6 GHz 'Montecito' rx8640 system.  An rx8640 is what HP call
a "cell based" system in that it is comprised of "cell boards" on which
reside CPU and memory resources.  In this case there are four cell
boards, each with 4, dual-core Montecito processors and 1/4 of the
overall RAM.  The system was configured with a mix of cell-local and
global interleaved memory, where the global interleave is on a cacheline
(128 byte) boundary (IIRC).  Total RAM in the system is 256 GB.  The
cells are joined via cross-bar connections. (numactl --hardware output
is available under the URL above)

There was an "I/O expander" connected to the system.  This meant there
were as many distinct PCI-X domains as there were cells, and every cell
had a "local" set of PCI-X slots.

Into those slots I placed four HP AD385A PCI-X 10Gbit Ethernet NICs -
aka Neterion XFrame IIs.  These were then connected to an HP ProCurve
5806 switch, which was in turn connected to three, 4P/16C, 2.3 GHz HP
DL585 G5s, each of which had a pair of HP AD386A PCIe 10Gbit Ethernet
NICs (Aka Chelsio T3C-based).  They were running RHEL 5.2 I think.  Each
NIC was in either a PCI-X 2.0 266 MHz slot (rx8640) or a PCIe 1.mumble
x8 slot (DL585 G5)

The kernel is from DaveM's net-next tree ca last week, multiq enabled. 
The s2io driver is Neterion's out-of-tree version 2.0.36.15914 to get
multiq support.  It was loaded into the kernel via:

insmod ./s2io.ko tx_steering_type=3 tx_fifo_num=8

There were then 8 tx queues and 8 rx queues per interface in the
rx8640.  The "setaffinity.txt" script was used to set the IRQ affinities
to cores "closest" to the physical NIC. In all three tests all 32 cores
went to 100% utilization. At least for all incense and porpoises. (there
was some occasional idle reported by top on the full_iptables run)

A set of 64, concurrent "burst mode" netperf omni RR tests (tcp) with a
burst mode of 17 were run (ie 17 "transactions" outstanding on a
connection at one time,) with TCP_NODELAY set and the results gathered,
along with a set of Caliper profiles.  The script used to launch these
can be found in "runemomniagg2.sh.txt under the URL above.

I picked an "RR" test to maximize the trips up and down the stack while
minimizing the bandwidth consumed.

I picked a burst size of 16 because that was sufficient to saturate a
single core on the rx8640.

I picked 64 concurrent netperfs because I wanted to make sure I had
enough concurrent connections to get spread across all the cores/queues
by the algorithms in place.

I picked the combination of 64 and 16 rather than say 1024 and 0 (one
tran at a time) because I didn't want to run a context switching
benchmark :)

The rx8640 was picked because it was available and I was confident it
was not going to have any hardware scaling issues getting in the way.  I
wanted to see SW issues, not HW issues. I am ass-u-me-ing the rx8640 is
a reasonable analog for any "decent or better scaling" 32 core hardware
and that while there are ia64-specific routines present in the profiles,
they are there for platform-independent reasons.

The no_iptables/ data was run after a fresh boot, with no iptables
commands run and so no iptables related modules loaded into the kernel.

The empty_iptables/ data was run after an "iptables --list" command
which loaded one or two modules into the kernel.

The full_iptables/ data was run after an "iptables-restore" command
pointed at full_iptables/iptables.txt  which was created from what RH
creates by default when one enables firewall via their installer, with a
port range added by me to allow pretty much anything netperf would ask. 
As such, while it does excercise netfilter functionality, I cannot make
any claims as to its "real world" applicability.  (while the firewall
settings came from an RH setup, FWIW, the base bits running on the
rx8640 are Debian Lenny, with the net-next kernel on top)

The "cycles" profile is able to grab flat profile hits while interrupts
are disabled so it can see stuff happening while interrupts are
disabled.  The "scgprof" profile is an attempt to get some call graphs -
it does not have visibility into code running with interrupts disabled. 
The "cache" profile is a profile that looks to get some cache miss
information.

So, having said all that, details can be found under the previously
mentioned URL.  Some quick highlights:

no_iptables - ~22000 transactions/s/netperf.  Top of the cycles profile
looks like:

Function Summary
-----------------------------------------------------------------------
% Total
     IP  Cumulat             IP
Samples    % of         Samples
 (ETB)     Total         (ETB)   Function                          File
-----------------------------------------------------------------------
   5.70     5.70         37772   s2io.ko::tx_intr_handler
   5.14    10.84         34012   vmlinux::__ia64_readq
   4.88    15.72         32285   s2io.ko::s2io_msix_ring_handle
   4.63    20.34         30625   s2io.ko::rx_intr_handler
   4.60    24.94         30429   s2io.ko::s2io_xmit
   3.85    28.79         25488   s2io.ko::s2io_poll_msix
   2.87    31.65         18987   vmlinux::dev_queue_xmit
   2.51    34.16         16620   vmlinux::tcp_sendmsg
   2.51    36.67         16588   vmlinux::tcp_ack
   2.15    38.82         14221   vmlinux::__inet_lookup_established
   2.10    40.92         13937   vmlinux::ia64_spinlock_contention

empty_iptables - ~12000 transactions/s/netperf.  Top of the cycles
profile looks like:

Function Summary
-----------------------------------------------------------------------
% Total
     IP  Cumulat             IP
Samples    % of         Samples
 (ETB)     Total         (ETB)   Function                          File
-----------------------------------------------------------------------
  26.38    26.38        137458   vmlinux::_read_lock_bh
  10.63    37.01         55388   vmlinux::local_bh_enable_ip
   3.42    40.43         17812   s2io.ko::tx_intr_handler
   3.01    43.44         15691   ip_tables.ko::ipt_do_table
   2.90    46.34         15100   vmlinux::__ia64_readq
   2.72    49.06         14179   s2io.ko::rx_intr_handler
   2.55    51.61         13288   s2io.ko::s2io_xmit
   1.98    53.59         10329   s2io.ko::s2io_msix_ring_handle
   1.75    55.34          9104   vmlinux::dev_queue_xmit
   1.64    56.98          8546   s2io.ko::s2io_poll_msix
   1.52    58.50          7943   vmlinux::sock_wfree
   1.40    59.91          7302   vmlinux::tcp_ack

full_iptables - some test instances didn't complete, I think they got
starved. Of those which did complete, their performance ranged all the
way from 330 to 3100 transactions/s/netperf.  Top of the cycles profile
looks like:

Function Summary
-----------------------------------------------------------------------
% Total
     IP  Cumulat             IP
Samples    % of         Samples
 (ETB)     Total         (ETB)   Function                          File
-----------------------------------------------------------------------
  64.71    64.71        582171   vmlinux::_write_lock_bh
  18.43    83.14        165822   vmlinux::ia64_spinlock_contention
   2.86    85.99         25709   nf_conntrack.ko::init_module
   2.36    88.35         21194   nf_conntrack.ko::tcp_packet
   1.78    90.13         16009   vmlinux::_spin_lock_bh
   1.20    91.33         10810   nf_conntrack.ko::nf_conntrack_in
   1.20    92.52         10755   vmlinux::nf_iterate
   1.09    93.62          9833   vmlinux::default_idle
   0.26    93.88          2331   vmlinux::__ia64_readq
   0.25    94.12          2213   vmlinux::__interrupt
   0.24    94.37          2203   s2io.ko::tx_intr_handler

Suggestions as to things to look at/with and/or patches to try are
welcome.  I should have the HW available to me for at least a little
while, but not indefinitely.

rick jones

Hi Rick, nice hardware you have :)

Stephen had a patch to nuke read_lock() from iptables, using RCU and seqlocks.
I hit this contention point even with low cost hardware, and quite standard application.

I pinged him few days ago to try to finish the job with him, but it seems Stephen
is busy at the moment.

Then conntrack (tcp sessions) is awfull, since it uses a single rwlock_t tcp_lock
 that must be write_locked() for basically every handled tcp frame...

How long is "not indefinitely" ? 

Does anyone have a fix for this bottleneck.