Hello Everyone-
So we tracked it down to IOMMU causing CPU affinity getting broken[1].
Can we provide any further details or is this a known issue?
Thank You,
Brandon
[1] https://github.com/coreos/bugs/issues/1275#issuecomment-219866601
On Tue, May 17, 2016 at 12:44 PM, Brandon Philips [off-list ref] wrote:
Hello ixgbe team-
With Linux v4.6 and the ixgbe driver (details below) a user is reporting
ksoftirqd consuming 100% of the CPU on all cores after a moderate ~20-50
number of TCP connections. They are unable to reproduce this issue with
Cisco hardware.
With Kernel v3.19 they cannot reproduce[1] the issue. Disabling IOMMU
(intel_iommu=off) does "fix" the issue[2].
Thank You,
Brandon
[1] https://github.com/coreos/bugs/issues/1275#issuecomment-219157803
[2] https://github.com/coreos/bugs/issues/1275#issuecomment-219819986
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
ethtool -i eno1
driver: ixgbe
version: 4.0.1-k
firmware-version: 0x800004e0
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
CPU
Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
I'm guessing the issue is lock contention on the IOMMU resource table.
I resolved most of that for the Rx side back when we implemented the
Rx page reuse but the Tx side still has to perform a DMA mapping for
each individual buffer. Depending on the needs of the user if they
still need the IOMMU enabled for use with something like KVM one thing
they may try doing is use the kernel parameter "iommu=pt" to allow
host devices to access memory without the penalty for having to
allocate/free resources and still provide guests with IOMMU isolation.
- Alex
On Tue, May 24, 2016 at 9:40 AM, Brandon Philips [off-list ref] wrote:
Hello Everyone-
So we tracked it down to IOMMU causing CPU affinity getting broken[1].
Can we provide any further details or is this a known issue?
Thank You,
Brandon
[1] https://github.com/coreos/bugs/issues/1275#issuecomment-219866601
On Tue, May 17, 2016 at 12:44 PM, Brandon Philips [off-list ref] wrote:
quoted
Hello ixgbe team-
With Linux v4.6 and the ixgbe driver (details below) a user is reporting
ksoftirqd consuming 100% of the CPU on all cores after a moderate ~20-50
number of TCP connections. They are unable to reproduce this issue with
Cisco hardware.
With Kernel v3.19 they cannot reproduce[1] the issue. Disabling IOMMU
(intel_iommu=off) does "fix" the issue[2].
Thank You,
Brandon
[1] https://github.com/coreos/bugs/issues/1275#issuecomment-219157803
[2] https://github.com/coreos/bugs/issues/1275#issuecomment-219819986
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
ethtool -i eno1
driver: ixgbe
version: 4.0.1-k
firmware-version: 0x800004e0
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
CPU
Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
On Tue, 24 May 2016 12:46:56 -0700
Alexander Duyck [off-list ref] wrote:
I'm guessing the issue is lock contention on the IOMMU resource table.
I resolved most of that for the Rx side back when we implemented the
Rx page reuse but the Tx side still has to perform a DMA mapping for
each individual buffer. Depending on the needs of the user if they
still need the IOMMU enabled for use with something like KVM one thing
they may try doing is use the kernel parameter "iommu=pt" to allow
host devices to access memory without the penalty for having to
allocate/free resources and still provide guests with IOMMU isolation.
Listen to Alex, he knows what his is talking about.
My longer term plan for getting rid of the dma_map/unmap overhead is to
_keep_ the pages DMA mapped and recycle them back via page-pool.
Details in my slides, see slide 5:
http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf
Alex'es RX recycle trick for the Intel drivers are described on
slide14. It seems like, in your use-case the pages might be held "too"
long for the RX recycling trick to work.
If you want to understand the IOMMU problem in details, I recommend to
read the article "True IOMMU Protection from DMA Attacks"
http://www.cs.technion.ac.il/~mad/publications/asplos2016-iommu.pdf
(My solution is different, but they desc the problem very well)
--Jesper
On Tue, May 24, 2016 at 9:40 AM, Brandon Philips [off-list ref] wrote:
quoted
Hello Everyone-
So we tracked it down to IOMMU causing CPU affinity getting broken[1].
Can we provide any further details or is this a known issue?
Thank You,
Brandon
[1] https://github.com/coreos/bugs/issues/1275#issuecomment-219866601
On Tue, May 17, 2016 at 12:44 PM, Brandon Philips [off-list ref] wrote:
quoted
Hello ixgbe team-
With Linux v4.6 and the ixgbe driver (details below) a user is reporting
ksoftirqd consuming 100% of the CPU on all cores after a moderate ~20-50
number of TCP connections. They are unable to reproduce this issue with
Cisco hardware.
With Kernel v3.19 they cannot reproduce[1] the issue. Disabling IOMMU
(intel_iommu=off) does "fix" the issue[2].
Thank You,
Brandon
[1] https://github.com/coreos/bugs/issues/1275#issuecomment-219157803
[2] https://github.com/coreos/bugs/issues/1275#issuecomment-219819986
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
ethtool -i eno1
driver: ixgbe
version: 4.0.1-k
firmware-version: 0x800004e0
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
CPU
Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer