@@ -7071,17 +7071,20 @@ static int rtl_alloc_irq(struct rtl8169_private *tp){unsignedintflags;-if(tp->mac_version<=RTL_GIGA_MAC_VER_06){+switch(tp->mac_version){+caseRTL_GIGA_MAC_VER_01...RTL_GIGA_MAC_VER_06:RTL_W8(tp,Cfg9346,Cfg9346_Unlock);RTL_W8(tp,Config2,RTL_R8(tp,Config2)&~MSIEnable);RTL_W8(tp,Cfg9346,Cfg9346_Lock);flags=PCI_IRQ_LEGACY;-}elseif(tp->mac_version==RTL_GIGA_MAC_VER_40){+break;+caseRTL_GIGA_MAC_VER_39...RTL_GIGA_MAC_VER_40:/* This version was reported to have issues with resume*fromsuspendwhenusingMSI-X*/flags=PCI_IRQ_LEGACY|PCI_IRQ_MSI;-}else{+break;+default:flags=PCI_IRQ_ALL_TYPES;}
From: David Miller <davem@davemloft.net> Date: 2018-08-16 19:21:34
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks.
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
Heiner
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
It could be that we need to reprogram some register(s) on resume,
which normally might not be needed, and that is what is causing the
problem with some chips.
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
It could be that we need to reprogram some register(s) on resume,
which normally might not be needed, and that is what is causing the
problem with some chips.
Indeed. That's what I'm checking with Realtek.
In the register list in the r8169 driver there's one entry which
seems to indicate that there are MSI-X specific settings.
However this register isn't used, and the r8168 vendor driver
uses only MSI. And there are no public datasheets.
From: Jian-Hong Pan <hidden> Date: 2018-08-17 05:08:07
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
[ 21.848357] libphy: r8169: probed
[ 21.848473] r8169 0000:02:00.0 eth0: RTL8106e, 0c:9d:92:32:67:b4, XID
44900000, IRQ 127
[ 22.518860] r8169 0000:02:00.0 enp2s0: renamed from eth0
[ 29.458041] Generic PHY r8169-200:00: attached PHY driver [Generic
PHY] (mii_bus:phy_addr=r8169-200:00, irq=IGNORE)
[ 63.227398] r8169 0000:02:00.0 enp2s0: Link is Up - 100Mbps/Full -
flow control off
[ 124.514648] Generic PHY r8169-200:00: attached PHY driver [Generic
PHY] (mii_bus:phy_addr=r8169-200:00, irq=IGNORE)
Here is the ethernet controller in detail:
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd.
RTL8101/2/6E PCI Express Fast/Gigabit Ethernet controller [10ec:8136]
(rev 07)
Subsystem: ASUSTeK Computer Inc. RTL810xE PCI Express Fast
Ethernet controller [1043:200f]
Flags: bus master, fast devsel, latency 0, IRQ 16
I/O ports at e000 [size=256]
Memory at ef100000 (64-bit, non-prefetchable) [size=4K]
Memory at e0000000 (64-bit, prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: r8169
Kernel modules: r8169
Falling back to MSI fixes the issue.
Fixes: 6c6aa15fdea5 ("r8169: improve interrupt handling")
Signed-off-by: Jian-Hong Pan <redacted>
---
Changes in v2:
- Make the commit message shorter
- Add "Fixes" tag in the commit message
drivers/net/ethernet/realtek/r8169.c | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
@@ -7071,17 +7071,20 @@ static int rtl_alloc_irq(struct rtl8169_private *tp){unsignedintflags;-if(tp->mac_version<=RTL_GIGA_MAC_VER_06){+switch(tp->mac_version){+caseRTL_GIGA_MAC_VER_01...RTL_GIGA_MAC_VER_06:RTL_W8(tp,Cfg9346,Cfg9346_Unlock);RTL_W8(tp,Config2,RTL_R8(tp,Config2)&~MSIEnable);RTL_W8(tp,Cfg9346,Cfg9346_Lock);flags=PCI_IRQ_LEGACY;-}elseif(tp->mac_version==RTL_GIGA_MAC_VER_40){+break;+caseRTL_GIGA_MAC_VER_39...RTL_GIGA_MAC_VER_40:/* This version was reported to have issues with resume*fromsuspendwhenusingMSI-X*/flags=PCI_IRQ_LEGACY|PCI_IRQ_MSI;-}else{+break;+default:flags=PCI_IRQ_ALL_TYPES;}
From: David Miller <davem@davemloft.net> Date: 2018-08-19 18:01:47
From: Jian-Hong Pan <redacted>
Date: Fri, 17 Aug 2018 13:07:35 +0800
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
...
Here is the ethernet controller in detail:
...
Falling back to MSI fixes the issue.
Fixes: 6c6aa15fdea5 ("r8169: improve interrupt handling")
Signed-off-by: Jian-Hong Pan <redacted>
---
Changes in v2:
- Make the commit message shorter
- Add "Fixes" tag in the commit message
I'm going to apply this for now, and queue it up for -stable.
If we hear back from Realtek on something we can do to make MSI-X
work on these chips, we can deal with it as a follow-up.
Thanks.
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
It could be that we need to reprogram some register(s) on resume,
which normally might not be needed, and that is what is causing the
problem with some chips.
Indeed. That's what I'm checking with Realtek.
In the register list in the r8169 driver there's one entry which
seems to indicate that there are MSI-X specific settings.
However this register isn't used, and the r8168 vendor driver
uses only MSI. And there are no public datasheets.
Do we have any information about these chip versions in other systems?
Or other devices using MSI-X in the same ASUS system? It seems
possible that there's some PCI core or suspend/resume issue with MSI-X
and this patch just avoids it without fixing the root cause.
It might be useful to have a kernel.org bugzilla with the complete
dmesg, "sudo lspci -vv" output, and /proc/interrupts contents archived
for future reference.
[1] https://lkml.kernel.org/r/20180815062110.16155-1-jian-hong@endlessm.com
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
It could be that we need to reprogram some register(s) on resume,
which normally might not be needed, and that is what is causing the
problem with some chips.
Indeed. That's what I'm checking with Realtek.
In the register list in the r8169 driver there's one entry which
seems to indicate that there are MSI-X specific settings.
However this register isn't used, and the r8168 vendor driver
uses only MSI. And there are no public datasheets.
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
It could be that we need to reprogram some register(s) on resume,
which normally might not be needed, and that is what is causing the
problem with some chips.
Indeed. That's what I'm checking with Realtek.
In the register list in the r8169 driver there's one entry which
seems to indicate that there are MSI-X specific settings.
However this register isn't used, and the r8168 vendor driver
uses only MSI. And there are no public datasheets.
Do we have any information about these chip versions in other systems?
Or other devices using MSI-X in the same ASUS system? It seems
possible that there's some PCI core or suspend/resume issue with MSI-X
and this patch just avoids it without fixing the root cause.
I'm in contact with Realtek and according to them few chip versions
seem to clear MSI-X table entries on resume from suspend. Checking
with them how this could be fixed / worked around.
Worst case we may have to disable MSI-X in general.
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
It could be that we need to reprogram some register(s) on resume,
which normally might not be needed, and that is what is causing the
problem with some chips.
Indeed. That's what I'm checking with Realtek.
In the register list in the r8169 driver there's one entry which
seems to indicate that there are MSI-X specific settings.
However this register isn't used, and the r8168 vendor driver
uses only MSI. And there are no public datasheets.
I don't think it's related, because falling back to MSI fixes the issue for
the reporter. And some chip versions report a generic Realtek PHY ID which
isn't covered by any Realtek PHY driver. These chip versions seem to work
fine with the generic PHY driver. So he may have Realtek PHY drivers enabled
or not. But indeed, would be good to have this info to get the full picture.
See also the mail I wrote few minutes ago, there it's described what we know
about the reason of the MSI-X issue so far.
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
It could be that we need to reprogram some register(s) on resume,
which normally might not be needed, and that is what is causing the
problem with some chips.
Indeed. That's what I'm checking with Realtek.
In the register list in the r8169 driver there's one entry which
seems to indicate that there are MSI-X specific settings.
However this register isn't used, and the r8168 vendor driver
uses only MSI. And there are no public datasheets.
Do we have any information about these chip versions in other systems?
Or other devices using MSI-X in the same ASUS system? It seems
possible that there's some PCI core or suspend/resume issue with MSI-X
and this patch just avoids it without fixing the root cause.
It might be useful to have a kernel.org bugzilla with the complete
dmesg, "sudo lspci -vv" output, and /proc/interrupts contents archived
for future reference.
The one system I have with a Realtek chip seems happy enough with MSI-X,
but it never gets suspended. There is comment in the patch that I don't
quite get:
It is the IRQ 127 - PCI-MSI used by enp2s0. However, lspci lists MSI is
disabled and MSI-X is enabled which conflicts to the interrupt table.
What do you mean by "conflicts"? With what? Another question is whether
you've loaded any firmware (some versions of the Realtek HW seem to require
it).
For the posterity, some data from my own system, which I don't know if it
has any relevance to the problem at hand.
Thanks,
M.
[ 2.624963] r8169 0000:02:00.0 eth0: RTL8168g/8111g, 5a:fe:ad:ce:11:00, XID 4c000800, IRQ 26
[ 2.633398] r8169 0000:02:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
26: 50 997005 0 0 MSI 1048576 Edge enp2s0
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 25
Region 0: I/O ports at 1000 [size=256]
Region 2: Memory at 100004000 (64-bit, prefetchable) [size=4K]
Region 4: Memory at 100000000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE#
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [b0] MSI-X: Enable+ Count=4 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00000800
Capabilities: [d0] Vital Product Data
pcilib: sysfs_read_vpd: read failed: Input/output error
Not readable
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Capabilities: [160 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [170 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Kernel driver in use: r8169
--
Jazz is not dead. It just smells funny...
I'm in contact with Realtek and according to them few chip versions
seem to clear MSI-X table entries on resume from suspend. Checking
with them how this could be fixed / worked around.
Worst case we may have to disable MSI-X in general.
I worry that if the chip does this, and somehow MSI-X is enabled and
an interrupt is generated, the chip will write to the cleared out
MSI-X address. This will either write garbage into memory or cause
a bus error and require PCI error recovery.
It also looks like your test patch doesn't fix things for people who
have tested it.
Hmmm...
I'm in contact with Realtek and according to them few chip versions
seem to clear MSI-X table entries on resume from suspend. Checking
with them how this could be fixed / worked around.
Worst case we may have to disable MSI-X in general.
I worry that if the chip does this, and somehow MSI-X is enabled and
an interrupt is generated, the chip will write to the cleared out
MSI-X address. This will either write garbage into memory or cause
a bus error and require PCI error recovery.
It also looks like your test patch doesn't fix things for people who
have tested it.
The test patch was based on the first info from Realtek which made me
think that the base address of the MSI-X table is cleared, what
obviously is not the case.
After some further tests it seems that the solution isn't as simple
as storing the MSI-X table entries on suspend and restore them on
resume. On my system (where MSI-X works fine) MSI-X table entries
on resume are partially different from the ones on suspend.
Unfortunately I don't have affected test hardware, currently I'm
waiting for further feedback from Realtek.
From: <redacted>
Date: Wed, 15 Aug 2018 14:21:10 +0800
quoted
Found the ethernet network on ASUS X441UAR doesn't come back on resume
from suspend when using MSI-X. The chip is RTL8106e - version 39.
Heiner, please take a look at this.
You recently disabled MSI-X on RTL8168g for similar reasons.
Now that we've seen two chips like this, maybe there is some other
problem afoot.
Thanks for the hint. I saw it already and just contacted Realtek
whether they are aware of any MSI-X issues with particular chip
versions. With the chip versions I have access to MSI-X works fine.
There's also the theoretical option that the issues are caused by
broken BIOS's. But so far only chip versions have been reported
which are very similar, at least with regard to version number
(2x VER_40, 1x VER_39). So they may share some buggy component.
Let's see whether Realtek can provide some hint.
If more chip versions are reported having problems with MSI-X,
then we could switch to a whitelist or disable MSI-X in general.
It could be that we need to reprogram some register(s) on resume,
which normally might not be needed, and that is what is causing the
problem with some chips.
Indeed. That's what I'm checking with Realtek.
In the register list in the r8169 driver there's one entry which
seems to indicate that there are MSI-X specific settings.
However this register isn't used, and the r8168 vendor driver
uses only MSI. And there are no public datasheets.
Do we have any information about these chip versions in other systems?
Or other devices using MSI-X in the same ASUS system? It seems
possible that there's some PCI core or suspend/resume issue with MSI-X
and this patch just avoids it without fixing the root cause.
It might be useful to have a kernel.org bugzilla with the complete
dmesg, "sudo lspci -vv" output, and /proc/interrupts contents archived
for future reference.
The one system I have with a Realtek chip seems happy enough with MSI-X,
but it never gets suspended.
Other owners of affected chip versiosn made the same experience, MSI-X
works fine until resume from suspend.
There is comment in the patch that I don't quite get:
quoted
It is the IRQ 127 - PCI-MSI used by enp2s0. However, lspci lists MSI is
disabled and MSI-X is enabled which conflicts to the interrupt table.
What do you mean by "conflicts"? With what? Another question is whether
you've loaded any firmware (some versions of the Realtek HW seem to require
it).
These "conflicts" were a misunderstanding which was clarified with the
reporter. "PCI-MSI" as irq chip name in /proc/interrupts output was
interpreted in a way that a MSI irq is used, not a MSI-X irq.
The firmware is for the PHY only, that's at least my experience on
the chip versions I have for testing.
I'm in contact with Realtek and according to them few chip versions
seem to clear MSI-X table entries on resume from suspend. Checking
with them how this could be fixed / worked around.
Worst case we may have to disable MSI-X in general.
I worry that if the chip does this, and somehow MSI-X is enabled and
an interrupt is generated, the chip will write to the cleared out
MSI-X address. This will either write garbage into memory or cause
a bus error and require PCI error recovery.
It also looks like your test patch doesn't fix things for people who
have tested it.
The test patch was based on the first info from Realtek which made me
think that the base address of the MSI-X table is cleared, what
obviously is not the case.
After some further tests it seems that the solution isn't as simple
as storing the MSI-X table entries on suspend and restore them on
resume. On my system (where MSI-X works fine) MSI-X table entries
on resume are partially different from the ones on suspend.
Which is not a surprise. Please don't try to fiddle with that at the driver
level. The irq and PCI core code are the ones in charge and if you'd
restore at the wrong point then hell breaks lose.
Can you please do the following:
1) Store the PCI config space at suspend time
2) Compare the PCI config space at resume time and print the difference
Do that on a working and a non-working version of Realtek NICs.
Thanks,
tglx
I'm in contact with Realtek and according to them few chip versions
seem to clear MSI-X table entries on resume from suspend. Checking
with them how this could be fixed / worked around.
Worst case we may have to disable MSI-X in general.
I worry that if the chip does this, and somehow MSI-X is enabled and
an interrupt is generated, the chip will write to the cleared out
MSI-X address. This will either write garbage into memory or cause
a bus error and require PCI error recovery.
It also looks like your test patch doesn't fix things for people who
have tested it.
The test patch was based on the first info from Realtek which made me
think that the base address of the MSI-X table is cleared, what
obviously is not the case.
After some further tests it seems that the solution isn't as simple
as storing the MSI-X table entries on suspend and restore them on
resume. On my system (where MSI-X works fine) MSI-X table entries
on resume are partially different from the ones on suspend.
Which is not a surprise. Please don't try to fiddle with that at the driver
level. The irq and PCI core code are the ones in charge and if you'd
restore at the wrong point then hell breaks lose.
Instead of spending a lot of effort on a workaround which may not be
acceptable, it may be better to fall back to MSI on all affected chip
versions. For two chip versions which were reported to have this issues
we're doing this already. I asked Realtek whether they have an overview
which chip versions are affected, let's see ..
The Realtek chips provide an alternative, register-based way to access
the MSI-X table, and their Windows driver seems to use it. See here:
https://patchwork.kernel.org/patch/4149171/
But as we handle all MSI-X basics in the PCI core, this isn't an option.
Can you please do the following:
1) Store the PCI config space at suspend time
2) Compare the PCI config space at resume time and print the difference
Do that on a working and a non-working version of Realtek NICs.
Thanks,
tglx
I'm in contact with Realtek and according to them few chip versions
seem to clear MSI-X table entries on resume from suspend. Checking
with them how this could be fixed / worked around.
Worst case we may have to disable MSI-X in general.
I worry that if the chip does this, and somehow MSI-X is enabled and
an interrupt is generated, the chip will write to the cleared out
MSI-X address. This will either write garbage into memory or cause
a bus error and require PCI error recovery.
It also looks like your test patch doesn't fix things for people who
have tested it.
The test patch was based on the first info from Realtek which made me
think that the base address of the MSI-X table is cleared, what
obviously is not the case.
After some further tests it seems that the solution isn't as simple
as storing the MSI-X table entries on suspend and restore them on
resume. On my system (where MSI-X works fine) MSI-X table entries
on resume are partially different from the ones on suspend.
Which is not a surprise. Please don't try to fiddle with that at the driver
level. The irq and PCI core code are the ones in charge and if you'd
restore at the wrong point then hell breaks lose.
Instead of spending a lot of effort on a workaround which may not be
acceptable, it may be better to fall back to MSI on all affected chip
versions. For two chip versions which were reported to have this issues
we're doing this already. I asked Realtek whether they have an overview
which chip versions are affected, let's see ..
The Realtek chips provide an alternative, register-based way to access
the MSI-X table, and their Windows driver seems to use it. See here:
https://patchwork.kernel.org/patch/4149171/
But as we handle all MSI-X basics in the PCI core, this isn't an option.
On Thu, Aug 23, 2018 at 06:46:28PM +0800, Jian-Hong Pan wrote:
quoted
On 22.08.2018 13:44, Thomas Gleixner wrote:
quoted
Can you please do the following:
Tested on ASUS X441AUR equipped with RTL8106e.
This is the laptop whose ethernet does not come back after resume, if
it does not fallback to MSI.
...
dev@endless:~$ sudo lspci -xnnvvs 02:00.0
...
00: ec 10 36 81 07 04 10 00 07 00 00 02 10 00 00 00
10: 01 e0 00 00 00 00 00 00 04 00 10 ef 00 00 00 00
20: 0c 00 00 e0 00 00 00 00 00 00 00 00 43 10 0f 20
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
After comparing, there is no difference between before suspend and
after resume.
It'd be better to compare the hex data directly and ignore the lspci
decoding, since lspci doesn't decode everything. You only dumped the
first 0x40 bytes of config space, and all capabilities, including the
MSI and MSI-X capabilities, are past that:
On Thu, Aug 23, 2018 at 06:46:28PM +0800, Jian-Hong Pan wrote:
quoted
quoted
On 22.08.2018 13:44, Thomas Gleixner wrote:
quoted
Can you please do the following:
Tested on ASUS X441AUR equipped with RTL8106e.
This is the laptop whose ethernet does not come back after resume, if
it does not fallback to MSI.
...
quoted
dev@endless:~$ sudo lspci -xnnvvs 02:00.0
...
00: ec 10 36 81 07 04 10 00 07 00 00 02 10 00 00 00
10: 01 e0 00 00 00 00 00 00 04 00 10 ef 00 00 00 00
20: 0c 00 00 e0 00 00 00 00 00 00 00 00 43 10 0f 20
30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00
After comparing, there is no difference between before suspend and
after resume.
It'd be better to compare the hex data directly and ignore the lspci
decoding, since lspci doesn't decode everything. You only dumped the
first 0x40 bytes of config space, and all capabilities, including the
MSI and MSI-X capabilities, are past that: