On Mon, Sep 18, 2017 at 03:50:30AM +0000, Yang, Qiming wrote:

Sorry, Gregory and Shijith

I misunderstand your advice.
If we only remove PCI reset at igbuio_pci_open, not include igbuio_pci_release, I think it is a good solution.
It reserve the functional of this patch and we have test on FVL 10G/40G/25G, all work fine.

Thanks Qiming. Glad to hear that removing pci_reset_function from igbuio_pci_open
fixed the issue for Intel NICs.

Harish,

Please let us know the behavior with QEDE. Reverting won't be necessary if the
fix works for all.

Thanks,
Shijith

quoted

-----Original Message-----
From: Zhang, Helin
Sent: Monday, September 18, 2017 10:39 AM
To: Yang, Qiming <redacted>; Gregory Etelson
[off-list ref]; thomas@monjalon.net
Cc: dev@dpdk.org; Tan, Jianfeng <redacted>; Thotton, Shijith
[off-list ref]; Hu, Xuekun [off-list ref]; Yigit,
Ferruh [off-list ref]; Patil, Harish [off-list ref]; Li,
Xiaoyun [off-list ref]
Subject: RE: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
and release of device file

Hi guys

May I suggest to revert the patch first, as it blocks existing functionalities.
Then the original authors can add it back with a fix?

Regards,
Helin

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yang, Qiming
Sent: Monday, September 18, 2017 10:21 AM
To: Gregory Etelson <redacted>
Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng
[off-list ref]; Thotton, Shijith [off-list ref]; Hu,
Xuekun [off-list ref]; Yigit, Ferruh [off-list ref]; Patil,
Harish [off-list ref]; Li, Xiaoyun [off-list ref]
Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
and release of device file

Yes, I have tried, and it works. But I think removing this function also makes the
patch meaningless.

Qiming
From: Gregory Etelson [mailto:gregory@weka.io]
Sent: Sunday, September 17, 2017 10:50 AM
To: Yang, Qiming <redacted>
Cc: thomas@monjalon.net; dev@dpdk.org; Tan, Jianfeng
[off-list ref]; Thotton, Shijith [off-list ref]; Hu,
Xuekun [off-list ref]; Yigit, Ferruh [off-list ref]; Patil,
Harish [off-list ref]
Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open
and release of device file


Hello Qiming,



Did you try to remove PCI reset at igbuio_pci_open ?

Reverting the patch entirely will get us back to VF issues



Regards,

Gregory





On Friday, 15 September 2017 11:04:16 IDT Yang, Qiming wrote:

quoted

Hi, Thomas

quoted

We meet a urgent issue because of Shijith's patch.

quoted

When use DPDK version 17.08, DPDK VF driver and kernel PF driver, i40e, ixgbe

and qede(Harish said) VF all meet initial failure.

quoted

This issue only occurred when use 17.08 igb_uio, vfio-pci and 17.05 igb_uio all

works well.

quoted

Considering this issue will block our vf driver develop and test work, could we

revert this patch first and find another appropriate way?

quoted

quoted

Qiming

quoted

quoted

quoted

-----Original Message-----

quoted

quoted

From: Patil, Harish [mailto:Harish.Patil@cavium.com]

quoted

quoted

Sent: Thursday, September 14, 2017 3:45 AM

quoted

quoted

To: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>;

Hu, Xuekun <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>;

quoted

quoted

Thotton, Shijith

<Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>; Yang,
Qiming

quoted

quoted

<qiming.yang@intel.com<mailto:qiming.yang@intel.com>>; Gregory Etelson

<gregory@weka.io<mailto:gregory@weka.io>>

quoted

quoted

Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng

<jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>

quoted

quoted

Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open

quoted

quoted

and release of device file

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

-----Original Message-----

quoted

quoted

From: dev <dev-bounces@dpdk.org<mailto:dev-bounces@dpdk.org>> on

behalf of Ferruh Yigit

quoted

quoted

<ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>

quoted

quoted

Date: Wednesday, September 13, 2017 at 10:06 AM

quoted

quoted

To: "Hu, Xuekun" <xuekun.hu@intel.com<mailto:xuekun.hu@intel.com>>,

"Thotton, Shijith"

quoted

quoted

<Shijith.Thotton@cavium.com<mailto:Shijith.Thotton@cavium.com>>,

"Yang, Qiming" <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>,

quoted

quoted

Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>>

quoted

quoted

Cc: "dev@dpdk.org<mailto:dev@dpdk.org>"

<dev@dpdk.org<mailto:dev@dpdk.org>>, "Tan, Jianfeng"
<jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>

quoted

quoted

Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open

quoted

quoted

and release of device file

quoted

quoted

quoted

quoted

quoted

On 9/13/2017 3:25 PM, Hu, Xuekun wrote:

quoted

quoted

quoted

quoted

I met the same issue too, only with i40e 2.1.26 PF kernel driver.

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

-----Original Message-----

quoted

quoted

quoted

quoted

From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit

quoted

quoted

quoted

quoted

Sent: Wednesday, September 13, 2017 7:04 PM

quoted

quoted

quoted

quoted

To: Shijith Thotton

<shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
.com>>; Yang,

quoted

quoted

quoted

quoted

Qiming <qiming.yang@intel.com<mailto:qiming.yang@intel.com>>;

Gregory Etelson <gregory@weka.io<mailto:gregory@weka.io>>

quoted

quoted

quoted

quoted

Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Tan, Jianfeng

<jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>

quoted

quoted

quoted

quoted

Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR

quoted

quoted

quoted

quoted

during open and release of device file

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

On 9/13/2017 11:48 AM, Shijith Thotton wrote:

quoted

quoted

quoted

quoted

quoted

On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote:

quoted

quoted

quoted

quoted

quoted

quoted

Hi, Shijith

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

VF init error will happen after apply your patch, error log as

quoted

quoted

quoted

quoted

quoted

quoted

below. If

quoted

quoted

quoted

quoted

quoted

quoted

revert your commit, all things work well. And this issue is not

quoted

quoted

quoted

quoted

quoted

quoted

only occur

quoted

quoted

quoted

quoted

quoted

quoted

in i40 VF but also ixgbe.

quoted

quoted

quoted

quoted

quoted

quoted

Hi Qiming,

quoted

quoted

quoted

quoted

quoted

quoted

I can reproduce the issue for the case:

quoted

quoted

quoted

quoted

quoted

quoted

1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k)

quoted

quoted

quoted

2- DPDK app run on host, so both PF and VF are in host. Linux driver

quoted

quoted

quoted

for PF and DPDK for VF. (When VF used in a VM, Linux PF on host and

quoted

quoted

quoted

DPDK VF on guest, this works fine)

quoted

quoted

quoted

quoted

quoted

quoted

And I confirm the function igbuio_pci_open() added with below patch is

quoted

quoted

quoted

causing it.

quoted

quoted

quoted

quoted

quoted

quoted

igbuio_pci_open() does:

quoted

quoted

quoted

pci_reset_function(dev);

quoted

quoted

quoted

pci_set_master(dev);

quoted

quoted

quoted

quoted

quoted

quoted

Don't know yet root cause of the error, will dig more.

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Could you help to check it soon?

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

[root@localhost app]# ./testpmd -c 7 -n 4 -- -i

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

EAL: Detected 10 lcore(s)

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

EAL: No free hugepages reported in hugepages-1048576kB

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

EAL: Probing VFIO support...

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using

quoted

quoted

quoted

quoted

quoted

quoted

unreliable clock cycles !

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

EAL: PCI device 0000:00:03.0 on NUMA socket -1

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

EAL: Invalid NUMA socket, default to 0

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

EAL: probe driver: 8086:154c net_i40e_vf

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

i40evf_init_vf(): init_adminq failed: -53

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

i40evf_dev_init(): Init vf failed

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

EAL: Requested device 0000:00:03.0 cannot be used

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

commit b58eedfc7dd57eef6d12e2c654a52c834f36084a

quoted

quoted

quoted

quoted

quoted

quoted

Author: Shijith Thotton

<shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
.com>>

quoted

quoted

quoted

quoted

quoted

quoted

Date: Fri Jul 7 16:43:51 2017 +0530

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

igb_uio: issue FLR during open and release of device file

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Set UIO info device file operations open and release. Call pci

quoted

quoted

quoted

quoted

quoted

quoted

reset

quoted

quoted

quoted

quoted

quoted

quoted

function inside open and release to clear device state at start

quoted

quoted

quoted

quoted

quoted

quoted

and end.

quoted

quoted

quoted

quoted

quoted

quoted

Copied this behaviour from vfio_pci kernel module code. With

quoted

quoted

quoted

quoted

quoted

quoted

this patch,

quoted

quoted

quoted

quoted

quoted

quoted

it is not mandatory to issue FLR by PMD's during init and close.

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Bus master enable and disable are added in open and release

quoted

quoted

quoted

quoted

quoted

quoted

respectively

quoted

quoted

quoted

quoted

quoted

quoted

to take care of device DMA.

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Signed-off-by: Shijith Thotton

<shijith.thotton@caviumnetworks.com<mailto:shijith.thotton@caviumnetworks
.com>>

quoted

quoted

quoted

quoted

quoted

quoted

Reviewed-by: Jianfeng Tan

<jianfeng.tan@intel.com<mailto:jianfeng.tan@intel.com>>

quoted

quoted

quoted

quoted

quoted

quoted

Acked-by: Ferruh Yigit

<ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>

quoted

quoted

quoted

quoted

quoted

quoted

Acked-by: Gregory Etelson

<gregory@weka.io<mailto:gregory@weka.io>>

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Best Regard,

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Yang Qiming

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Hi Yang Qiming,

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

We didn't face this issue while preparing patch. I think, Gregory

quoted

quoted

quoted

quoted

quoted

tested it for Intel NICs. Adding more people to the loop for help.

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

I will try to reproduce the issue.

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

quoted

Shijith

quoted

quoted

quoted

quoted

[Harish] We are seeing similar issue with the bringup of qede VF devices due

to

quoted

quoted

this patch.

quoted

quoted

Could we revert this and leave it to individual PMDs to manage it?

quoted

quoted

quoted

quoted

quoted

quoted