On 1/21/2021 4:35 PM, Lincoln Lavoie wrote:

Hi All,

Trying to follow the specific conversation.  It is correct, the lab does not 
list the specific throughput values achieved by the hardware, as that data can 
be sensitive to the hardware vendors, etc. The purpose of the lab is to check 
for degradations caused by patches, so the difference is really the important 
factor.  The comparison is against a prior run on the same hardware, via the 
DPDK main branch, so any delta should be caused by the specific patch changes 
(excluding statistical "wiggle").

If the group would prefer, we could calculate additional references if desired 
(i.e. difference from the last official release, or a monthly run of the 
current, etc.).  We just need the community to define their needs, and we can 
add this to the development queue.

Hi Brandon,

Can you also put above to the backlog, to display the performance difference to 
the a fixed point, like a previous release or a previous LTS?

Thanks,
ferruh

Cheers,
Lincoln


On Thu, Jan 21, 2021 at 4:29 AM Morten Brørup <mb@smartsharesystems.com 
<mailto:mb@smartsharesystems.com>> wrote:

     > From: dev [mailto:dev-bounces@dpdk.org <mailto:dev-bounces@dpdk.org>] On
    Behalf Of Ferruh Yigit
     > Sent: Thursday, January 21, 2021 10:19 AM
     >
     > On 1/15/2021 6:39 PM, Ali Alnubani wrote:
     > > Hi,
     > > Adding Ferruh and Zhaoyan,
     > >
     > >> Ali,
     > >>
     > >> You reported some performance regression, did you confirm it?
     > >> If I get no reply by monday, I'll proceed with this patch.
     > >
     > > Sure I'll confirm by Monday.
     > >
     > > Doesn't the regression also reproduce on the Lab's Intel servers?
     > > Even though the check iol-intel-Performance isn't failing, I can see
     > that the throughput differences from expected for this patch are less
     > than those of another patch that was tested only 20 minutes earlier.
     > Both patches were applied to the same tree:
     > >
     > > https://mails.dpdk.org/archives/test-report/2021-January/173927.html
    <https://mails.dpdk.org/archives/test-report/2021-January/173927.html>
     > >> | 64         | 512     | 1.571                               |
     > >
     > > https://mails.dpdk.org/archives/test-report/2021-January/173919.html
    <https://mails.dpdk.org/archives/test-report/2021-January/173919.html>
     > >> | 64         | 512     | 2.698                               |
     > >
     > > Assuming that pw86457 doesn't have an effect on this test, it looks
     > to me that this patch caused a regression in Intel hardware as well.
     > >
     > > Can someone update the baseline's expected values for the Intel NICs
     > and rerun the test on this patch?
     > >
     >
     > Zhaoyan said that the baseline is calculated dynamically,
     > what I understand is baseline set based on previous days performance
     > result, so
     > it shouldn't require updating.

    That sounds smart!

    Perhaps another reference baseline could be added, for informational
    purposes only:
    Deviation from the performance of the last official release.

     >
     > But cc'ed the lab for more details.



-- 
*Lincoln Lavoie*
Senior Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylavoie@iol.unh.edu <mailto:lylavoie@iol.unh.edu>
https://www.iol.unh.edu <https://www.iol.unh.edu>
+1-603-674-2755 (m)
<https://www.iol.unh.edu>