Re: [PATCH] PCI: Add no-D3 quirk for Mellanox ConnectX-[45]

On Fri, Jan 04, 2019 at 02:44:01PM +1100, David Gibson wrote:

On Thu, Dec 06, 2018 at 08:45:09AM +0200, Leon Romanovsky wrote:

quoted

On Thu, Dec 06, 2018 at 03:19:51PM +1100, David Gibson wrote:

quoted

Mellanox ConnectX-5 IB cards (MT27800) seem to cause a call trace when
unbound from their regular driver and attached to vfio-pci in order to pass
them through to a guest.

This goes away if the disable_idle_d3 option is used, so it looks like a
problem with the hardware handling D3 state.  To fix that more permanently,
use a device quirk to disable D3 state for these devices.

We do this by renaming the existing quirk_no_ata_d3() more generally and
attaching it to the ConnectX-[45] devices (0x15b3:0x1013).

Signed-off-by: David Gibson <redacted>
 drivers/pci/quirks.c | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

Hi David,

Thank for your patch,

I would like to reproduce the calltrace before moving forward,
but have trouble to reproduce the original issue.

I'm working with vfio-pci and CX-4/5 cards on daily basis,
tried manually enter into D3 state now, and it worked for me.

Interesting.  I've investigated this further, though I don't have as
many new clues as I'd like.  The problem occurs reliably, at least on
one particular type of machine (a POWER8 "Garrison" with ConnectX-4).
I don't yet know if it occurs with other machines, I'm having trouble
getting access to other machines with a suitable card.  I didn't
manage to reproduce it on a different POWER8 machine with a
ConnectX-5, but I don't know if it's the difference in machine or
difference in card revision that's important.

Make sure the card has the latest firmware is always good advice..

So possibilities that occur to me:
  * It's something specific about how the vfio-pci driver uses D3
    state - have you tried rebinding your device to vfio-pci?
  * It's something specific about POWER, either the kernel or the PCI
    bridge hardware
  * It's something specific about this particular type of machine

Does the EEH indicate what happend to actually trigger it?

Jason