From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Lincoln Lavoie
Sent: Thursday, January 21, 2021 5:35 PM
To: Morten Brørup

Hi All,

Trying to follow the specific conversation.  It is correct, the lab
does
not list the specific throughput values achieved by the hardware, as
that
data can be sensitive to the hardware vendors, etc. The purpose of the
lab
is to check for degradations caused by patches, so the difference is
really
the important factor.  The comparison is against a prior run on the
same
hardware, via the DPDK main branch, so any delta should be caused by
the
specific patch changes (excluding statistical "wiggle").

Thank you for clarifying, Lincoln.

This sounds like a perfect solution to the meet the purpose.

I request that you change the output to show the relative difference (i.e. percentage above/below baseline), instead of the absolute difference (i.e. number of packets per second).

By showing a percentage, anyone reading the test report can understand if the difference is insignificant, or big enough to require further discussion before accepting the patch. Showing the difference in packets per second requires the reader of the test report to have prior knowledge about the expected performance.

If the group would prefer, we could calculate additional references if
desired (i.e. difference from the last official release, or a monthly
run
of the current, etc.).  We just need the community to define their
needs,
and we can add this to the development queue.

I retract my suggestion about adding additional references. You clearly explained how your baselining works, and I think it fully serves the needs of a regression test.

So please just put this small change in your development queue: Output the difference in percent with a few decimals after the comma, instead of the difference in packets per second.

For readability, performance drops should be shown as negative values, and increases as positive, e.g.:

Difference = (NewPPS - BaselinePPS) / BaselinePPS * 100.00 %.


If we want to compare performance against official releases, current or older, it should go elsewhere, not in the continuous testing environment. E.g. the release notes could include a report showing differences from the last few official releases. But that is a task for another day.

Cheers,
Lincoln


On Thu, Jan 21, 2021 at 4:29 AM Morten Brørup
[off-list ref]
wrote:

quoted

quoted

From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit
Sent: Thursday, January 21, 2021 10:19 AM

On 1/15/2021 6:39 PM, Ali Alnubani wrote:

quoted

Hi,
Adding Ferruh and Zhaoyan,

quoted

Ali,

You reported some performance regression, did you confirm it?
If I get no reply by monday, I'll proceed with this patch.

Sure I'll confirm by Monday.

Doesn't the regression also reproduce on the Lab's Intel servers?
Even though the check iol-intel-Performance isn't failing, I can

see

quoted

quoted

that the throughput differences from expected for this patch are

less

quoted

quoted

than those of another patch that was tested only 20 minutes

earlier.

quoted

quoted

Both patches were applied to the same tree:

quoted

https://mails.dpdk.org/archives/test-report/2021-

January/173927.html

quoted

quoted

quoted

quoted

| 64         | 512     | 1.571                               |

https://mails.dpdk.org/archives/test-report/2021-

January/173919.html

quoted

quoted

quoted

quoted

| 64         | 512     | 2.698                               |

Assuming that pw86457 doesn't have an effect on this test, it

looks

quoted

quoted

to me that this patch caused a regression in Intel hardware as

well.

quoted

quoted

quoted

Can someone update the baseline's expected values for the Intel

NICs

quoted

quoted

and rerun the test on this patch?

quoted

Zhaoyan said that the baseline is calculated dynamically,
what I understand is baseline set based on previous days

performance

quoted

quoted

result, so
it shouldn't require updating.

That sounds smart!

Perhaps another reference baseline could be added, for informational
purposes only:
Deviation from the performance of the last official release.

quoted

But cc'ed the lab for more details.

--
*Lincoln Lavoie*
Senior Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
<https://www.iol.unh.edu>