Single core gets pegged on multi-core PPTP server

5 messages, 4 authors, 2012-01-20 · open the first message on its own page

Single core gets pegged on multi-core PPTP server

From: Bradley Peterson <hidden>
Date: 2012-01-19 22:35:59

Hello,

I am trying to test the capacity of a linux PPTP server, both in
number of connections, and in packets per second.  I am using kernel
2.6.38.8, with the ppp, pptp, and gre modules, and accel-pptp 0.8.3.
I have RPS, RFS, and XPS enabled on the network devices for SMP
support.

But I'm seeing one CPU get pegged out with soft interrupt, while the
others are almost completely idle.

In my current test, I'm starting 250 pptp connections from another
server, then running iperf across each connection.  The client machine
pegs out, sure, but I'm surprised the server pegs out a single CPU.
With RPS, I would expect softirq's to be more balanced.

Where could the bottleneck be?  Do all ppp packets need to be
processed serially?

Brad

Re: Single core gets pegged on multi-core PPTP server

From: Emanuil Hristov <hidden>
Date: 2012-01-19 22:55:37

Hi,
try with more recent version of accel-ppp (1.5 is lastest) and newer
kernel - there
are several critical bugs in older kernels


P.S the project accel-pptp has changed its name to accel-ppp

cheers

On Fri, Jan 20, 2012 at 12:35 AM, Bradley Peterson [off-list ref] wrote:
Hello,

I am trying to test the capacity of a linux PPTP server, both in
number of connections, and in packets per second.  I am using kernel
2.6.38.8, with the ppp, pptp, and gre modules, and accel-pptp 0.8.3.
I have RPS, RFS, and XPS enabled on the network devices for SMP
support.

But I'm seeing one CPU get pegged out with soft interrupt, while the
others are almost completely idle.

In my current test, I'm starting 250 pptp connections from another
server, then running iperf across each connection.  The client machine
pegs out, sure, but I'm surprised the server pegs out a single CPU.
With RPS, I would expect softirq's to be more balanced.

Where could the bottleneck be?  Do all ppp packets need to be
processed serially?

Brad
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Single core gets pegged on multi-core PPTP server

From: Alexey Osipov <hidden>
Date: 2012-01-19 23:28:16

Hi,

AFAIK, accel-ppp != accel-pptp.

accel-pptp - kernel module for speed-up pptp processing done by pppd.

accel-ppp - entirely new implementation of ppp written from scratch. No
pppd is used in this case.

So, if in your configuration, you need to use some other plugins for
pppd, you can't just switch from accel-pptp to accel-ppp, because
accel-ppp have it's own plugin API.

Regards,
Alexey.

В Птн, 20/01/2012 в 00:55 +0200, Emanuil Hristov пишет:
Hi,
try with more recent version of accel-ppp (1.5 is lastest) and newer
kernel - there
are several critical bugs in older kernels


P.S the project accel-pptp has changed its name to accel-ppp

cheers

On Fri, Jan 20, 2012 at 12:35 AM, Bradley Peterson [off-list ref] wrote:
quoted
Hello,

I am trying to test the capacity of a linux PPTP server, both in
number of connections, and in packets per second.  I am using kernel
2.6.38.8, with the ppp, pptp, and gre modules, and accel-pptp 0.8.3.
I have RPS, RFS, and XPS enabled on the network devices for SMP
support.

But I'm seeing one CPU get pegged out with soft interrupt, while the
others are almost completely idle.

In my current test, I'm starting 250 pptp connections from another
server, then running iperf across each connection.  The client machine
pegs out, sure, but I'm surprised the server pegs out a single CPU.
With RPS, I would expect softirq's to be more balanced.

Where could the bottleneck be?  Do all ppp packets need to be
processed serially?

Brad
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-ppp" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Single core gets pegged on multi-core PPTP server

From: Eric Dumazet <hidden>
Date: 2012-01-20 06:16:30

Le jeudi 19 janvier 2012 à 16:35 -0600, Bradley Peterson a écrit :
Hello,

I am trying to test the capacity of a linux PPTP server, both in
number of connections, and in packets per second.  I am using kernel
2.6.38.8, with the ppp, pptp, and gre modules, and accel-pptp 0.8.3.
I have RPS, RFS, and XPS enabled on the network devices for SMP
support.

But I'm seeing one CPU get pegged out with soft interrupt, while the
others are almost completely idle.

In my current test, I'm starting 250 pptp connections from another
server, then running iperf across each connection.  The client machine
pegs out, sure, but I'm surprised the server pegs out a single CPU.
With RPS, I would expect softirq's to be more balanced.

Where could the bottleneck be?  Do all ppp packets need to be
processed serially?
Hmmm, you need a more recent kernel or backport commit 
c6865cb3cc6f3c2857fa4c6f5fda2945d70b1e84
    rps: Inspect GRE encapsulated packets to get flow hash
    
    Crack open GRE packets in __skb_get_rxhash to compute 4-tuple hash on
    in encapsulated packet.  Note that this is used only when the
    __skb_get_rxhash is taken, in particular only when the device does
    not compute provide the rxhash (ie. feature is disabled).
    
    This was tested by creating a single GRE tunnel between two 16 core
    AMD machines.  200 netperf TCP_RR streams were ran with 1 byte
    request and response size.
    
    Without patch: 157497 tps, 50/90/99% latencies 1250/1292/1364 usecs
    With patch: 325896 tps, 50/90/99% latencies 603/848/1169
    
    Signed-off-by: Tom Herbert [off-list ref]
    Signed-off-by: David S. Miller [off-list ref]



And make sure you disabled hardware rxhash (if your NIC provides it)

Re: Single core gets pegged on multi-core PPTP server

From: Bradley Peterson <hidden>
Date: 2012-01-20 15:20:02

On Fri, Jan 20, 2012 at 12:16 AM, Eric Dumazet [off-list ref] wrote:
Le jeudi 19 janvier 2012 à 16:35 -0600, Bradley Peterson a écrit :
quoted
Hello,

I am trying to test the capacity of a linux PPTP server, both in
number of connections, and in packets per second.  I am using kernel
2.6.38.8, with the ppp, pptp, and gre modules, and accel-pptp 0.8.3.
I have RPS, RFS, and XPS enabled on the network devices for SMP
support.

But I'm seeing one CPU get pegged out with soft interrupt, while the
others are almost completely idle.

In my current test, I'm starting 250 pptp connections from another
server, then running iperf across each connection.  The client machine
pegs out, sure, but I'm surprised the server pegs out a single CPU.
With RPS, I would expect softirq's to be more balanced.

Where could the bottleneck be?  Do all ppp packets need to be
processed serially?
Hmmm, you need a more recent kernel or backport commit
c6865cb3cc6f3c2857fa4c6f5fda2945d70b1e84
   rps: Inspect GRE encapsulated packets to get flow hash

   Crack open GRE packets in __skb_get_rxhash to compute 4-tuple hash on
   in encapsulated packet.  Note that this is used only when the
   __skb_get_rxhash is taken, in particular only when the device does
   not compute provide the rxhash (ie. feature is disabled).

   This was tested by creating a single GRE tunnel between two 16 core
   AMD machines.  200 netperf TCP_RR streams were ran with 1 byte
   request and response size.

   Without patch: 157497 tps, 50/90/99% latencies 1250/1292/1364 usecs
   With patch: 325896 tps, 50/90/99% latencies 603/848/1169

   Signed-off-by: Tom Herbert [off-list ref]
   Signed-off-by: David S. Miller [off-list ref]



And make sure you disabled hardware rxhash (if your NIC provides it)
Thank you, I had missed the work on RPS with encapsulated packets.  I
will try a newer kernel.
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help