Re: DPDK testpmd forwarding performace degradation

From: Alexander Belyakov <hidden>
Date: 2015-01-29 12:43:45

On Wed, Jan 28, 2015 at 3:24 PM, Alexander Belyakov [off-list ref]
wrote:


On Tue, Jan 27, 2015 at 7:21 PM, De Lara Guarch, Pablo <
pablo.de.lara.guarch-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

quoted

On Tue, Jan 27, 2015 at 10:51 AM, Alexander Belyakov

quoted

[off-list ref] wrote:

quoted

Hi Pablo,

quoted

On Mon, Jan 26, 2015 at 5:22 PM, De Lara Guarch, Pablo

quoted

[off-list ref] wrote:

quoted

Hi Alexander,

quoted

-----Original Message-----

quoted

From: dev [mailto:dev-bounces-VfR2kkLFssw@public.gmane.org <redacted>] On

Behalf Of Alexander

quoted

Belyakov

quoted

Sent: Monday, January 26, 2015 10:18 AM

quoted

To: dev-VfR2kkLFssw@public.gmane.org

quoted

Subject: [dpdk-dev] DPDK testpmd forwarding performace degradation

quoted

quoted

quoted

quoted

recently I have found a case of significant performance degradation

for our

quoted

application (built on top of DPDK, of course). Surprisingly, similar

issue

quoted

is easily reproduced with default testpmd.

quoted

quoted

To show the case we need simple IPv4 UDP flood with variable UDP

quoted

payload

quoted

size. Saying "packet length" below I mean: Eth header length (14

bytes) +

quoted

IPv4 header length (20 bytes) + UPD header length (8 bytes) + UDP

payload

quoted

length (variable) + CRC (4 bytes). Source IP addresses and ports are

quoted

selected

quoted

randomly for each packet.

quoted

quoted

I have used DPDK with revisions 1.6.0r2 and 1.7.1. Both show the same

quoted

issue.

quoted

quoted

Follow "Quick start" guide (http://dpdk.org/doc/quick-start) to

build and

quoted

run testpmd. Enable testpmd forwarding ("start" command).

quoted

quoted

Table below shows measured forwarding performance depending on

quoted

packet

quoted

quoted

quoted

No. -- UDP payload length (bytes) -- Packet length (bytes) --

Forwarding

quoted

performance (Mpps) -- Expected theoretical performance (Mpps)

quoted

quoted

1. 0 -- 64 -- 14.8 -- 14.88

quoted

2. 34 -- 80 -- 12.4 -- 12.5

quoted

3. 35 -- 81 -- 6.2 -- 12.38 (!)

quoted

4. 40 -- 86 -- 6.6 -- 11.79

quoted

5. 49 -- 95 -- 7.6 -- 10.87

quoted

6. 50 -- 96 -- 10.7 -- 10.78 (!)

quoted

7. 60 -- 106 -- 9.4 -- 9.92

quoted

quoted

At line number 3 we have added 1 byte of UDP payload (comparing to

quoted

quoted

line) and got forwarding performance halved! 6.2 Mpps against 12.38

Mpps

quoted

quoted

expected theoretical maximum for this packet size.

quoted

quoted

That is the issue.

quoted

quoted

Significant performance degradation exists up to 50 bytes of UDP

payload

quoted

(96 bytes packet length), where it jumps back to theoretical maximum.

quoted

quoted

What is happening between 80 and 96 bytes packet length?

quoted

quoted

This issue is stable and 100% reproducible. At this point I am not

sure if

quoted

it is DPDK or NIC issue. These tests have been performed on Intel(R)

Eth

quoted

Svr Bypass Adapter X520-LR2 (X520LR2BP).

quoted

quoted

Is anyone aware of such strange behavior?

quoted

I cannot reproduce the issue using two ports on two different 82599EB

NICs,

quoted

using 1.7.1 and 1.8.0.

quoted

I always get either same or better linerate as I increase the packet

size.

quoted

Thank you for trying to reproduce the issue.

quoted

Actually, have you tried using 1.8.0?

quoted

I feel 1.8.0 is little bit immature and might require some post-release

quoted

patching. Even tespmd from this release is not forwarding packets

properly

quoted

on my setup. It is up and running without visible errors/warnings, TX/RX

quoted

counters are ticking but I can not see any packets at the output.



This is strange. Without  changing anything, forwarding works perfectly
for me

(so, RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC is enabled).

quoted

Please note, both 1.6.0r2 and 1.7.1 releases work (on the same setup)

out-of-the-box just

quoted

fine with only exception of this mysterious performance drop.

quoted

So it will take some time to figure out what is wrong with dpdk-1.8.0.

quoted

Meanwhile we could focus on stable dpdk-1.7.1.

quoted

Managed to get testpmd from dpdk-1.8.0 to work on my setup.

quoted

Unfortunately I had to disable RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC,

quoted

it is new comparing to 1.7.1 and somehow breaks testpmd forwarding. By

the

quoted

way, simply disabling RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC in

quoted

common_linuxapp config file breaks the build - had to make

quick'n'dirty fix

quoted

in struct igb_rx_queue as well.

quoted

Anyway, issue is still here.

quoted

Forwarding 80 bytes packets at 12.4 Mpps.

quoted

Forwarding 81 bytes packets at 7.2 Mpps.

quoted

Any ideas?

quoted

As for X520-LR2 NIC - it is dual port bypass adapter with device id

155d. I

quoted

believe it should be treated as 82599EB except bypass feature. I put

bypass

quoted

mode to "normal" in those tests.



I have used a 82599EB first, and now a X520-SR2. Same results.

I assume that X520-SR2 and X520-LR2 should give similar results

(only thing that is changed is the wavelength, but the controller is the
same).

It seems I found what was wrong, at least got a hint.

My build server machine type differs from test setup. Until now it was OK
to build DPDK with -march=native.

I found that building dpdk-1.8.0 with explicitly set core-avx-i (snb, ivb)
or bdver2 (amd) machine types almost eliminates performance drop. The same
goes for RTE_LIBRTE_IXGBE_RX_ALLOW_BULK_ALLOC option issues.

It seems DPDK performance and stability depends on machine type more than
I was expecting.

Thank you for your help.

Mysteries are still here.

While single RX queue forwarding shows almost no degradation now, multiple
RX queues still do.

Launching

./testpmd -c7 -n3 -- -i --rxq=2 --txq=2 --nb-cores=2 --nb-ports=2

shows the following result:

80 bytes packets are being forwarded at 12.46Mpps rate (which is fine)
81 bytes packets are being forwarded at 7.5Mpps rate (which is weird)

Pablo, could you please check on your setup if multiple RX queues shows no
performance degradation depending on packet size?

Additional information about packets I'm sending.
80 bytes packet is IPv4 UDP packet (with random source IP/port) and 34
payload bytes.
81 bytes packet is IPv4 UDP packet (with random source IP/port) and 35
payload bytes.

Thank you,
Alexander

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help