From: Meelis Roos <hidden> Date: 2014-09-15 23:15:04
Somewhere between 3.17.0-rc3 and 3.17.0-rc5 I started seeing dropped ssh
connections to a couple of test servers with dual AthlonMP (32-bit) and
3C90x family of NICs (3Com Corporation 3c980-C 10/100baseTX NIC
[Python-T] (rev 78) in one server and 3Com Corporation 3c905C-TX/TX-M
[Tornado] (rev 78) in the other server). Bisect leads to the following
commit:
98ea232cf63961fad734cc8c5e07e8915ec73073 is the first bad commit
commit 98ea232cf63961fad734cc8c5e07e8915ec73073
Author: Neil Horman [off-list ref]
Date: Thu Sep 4 06:13:38 2014 -0400
3c59x: avoid panic in boomerang_start_xmit when finding page address:
...
--
Meelis Roos (mroos@linux.ee)
From: Neil Horman <nhorman@tuxdriver.com> Date: 2014-09-16 10:17:22
On Tue, Sep 16, 2014 at 02:14:54AM +0300, Meelis Roos wrote:
Somewhere between 3.17.0-rc3 and 3.17.0-rc5 I started seeing dropped ssh
connections to a couple of test servers with dual AthlonMP (32-bit) and
3C90x family of NICs (3Com Corporation 3c980-C 10/100baseTX NIC
[Python-T] (rev 78) in one server and 3Com Corporation 3c905C-TX/TX-M
[Tornado] (rev 78) in the other server). Bisect leads to the following
commit:
98ea232cf63961fad734cc8c5e07e8915ec73073 is the first bad commit
commit 98ea232cf63961fad734cc8c5e07e8915ec73073
Author: Neil Horman [off-list ref]
Date: Thu Sep 4 06:13:38 2014 -0400
3c59x: avoid panic in boomerang_start_xmit when finding page address:
...
--
Meelis Roos (mroos@linux.ee)
I'm guessing the above change has uncovered another bug, mostly likely an
exhaustion of dma space on your system. Nothing in the transmit path there does
any error checking for successful dma mapping, which it really should. I'd be
willing to be that any dma mapping error leads to a leak in the mapping table.
Does your system have an iommu, or does it use swiotlb? If its the latter, can
you increase the swiotlb table space and see if that relieves the problem? In
the interim, I'll start adding some error checking to the transmit path.
Neil
From: Meelis Roos <hidden> Date: 2014-09-16 10:32:35
[ Fixed Steffen Klasserts bouncing email address as per the bounce message ]
quoted
Somewhere between 3.17.0-rc3 and 3.17.0-rc5 I started seeing dropped ssh
connections to a couple of test servers with dual AthlonMP (32-bit) and
3C90x family of NICs (3Com Corporation 3c980-C 10/100baseTX NIC
[Python-T] (rev 78) in one server and 3Com Corporation 3c905C-TX/TX-M
[Tornado] (rev 78) in the other server). Bisect leads to the following
commit:
98ea232cf63961fad734cc8c5e07e8915ec73073 is the first bad commit
commit 98ea232cf63961fad734cc8c5e07e8915ec73073
Author: Neil Horman [off-list ref]
Date: Thu Sep 4 06:13:38 2014 -0400
3c59x: avoid panic in boomerang_start_xmit when finding page address:
...
I'm guessing the above change has uncovered another bug, mostly likely an
exhaustion of dma space on your system. Nothing in the transmit path there does
any error checking for successful dma mapping, which it really should. I'd be
willing to be that any dma mapping error leads to a leak in the mapping table.
Does your system have an iommu, or does it use swiotlb? If its the latter, can
you increase the swiotlb table space and see if that relieves the problem? In
the interim, I'll start adding some error checking to the transmit path.
# CONFIG_IOMMU_SUPPORT is not set
nothing matching iommu or iotlb in dmesg. I thought the system does not
have any - K7 CPUs and AMD 760MP chipset with normal AGP gart.
[ 3.763519] agpgart-amdk7 0000:00:00.0: AMD 760MP chipset
[ 3.782995] agpgart-amdk7 0000:00:00.0: AGP aperture is 64M @ 0xf8000000
--
Meelis Roos (mroos@linux.ee)
From: Neil Horman <nhorman@tuxdriver.com> Date: 2014-09-16 14:31:38
On Tue, Sep 16, 2014 at 01:32:31PM +0300, Meelis Roos wrote:
[ Fixed Steffen Klasserts bouncing email address as per the bounce message ]
quoted
quoted
Somewhere between 3.17.0-rc3 and 3.17.0-rc5 I started seeing dropped ssh
connections to a couple of test servers with dual AthlonMP (32-bit) and
3C90x family of NICs (3Com Corporation 3c980-C 10/100baseTX NIC
[Python-T] (rev 78) in one server and 3Com Corporation 3c905C-TX/TX-M
[Tornado] (rev 78) in the other server). Bisect leads to the following
commit:
98ea232cf63961fad734cc8c5e07e8915ec73073 is the first bad commit
commit 98ea232cf63961fad734cc8c5e07e8915ec73073
Author: Neil Horman [off-list ref]
Date: Thu Sep 4 06:13:38 2014 -0400
3c59x: avoid panic in boomerang_start_xmit when finding page address:
...
I'm guessing the above change has uncovered another bug, mostly likely an
exhaustion of dma space on your system. Nothing in the transmit path there does
any error checking for successful dma mapping, which it really should. I'd be
willing to be that any dma mapping error leads to a leak in the mapping table.
Does your system have an iommu, or does it use swiotlb? If its the latter, can
you increase the swiotlb table space and see if that relieves the problem? In
the interim, I'll start adding some error checking to the transmit path.
# CONFIG_IOMMU_SUPPORT is not set
nothing matching iommu or iotlb in dmesg. I thought the system does not
have any - K7 CPUs and AMD 760MP chipset with normal AGP gart.
[ 3.763519] agpgart-amdk7 0000:00:00.0: AMD 760MP chipset
[ 3.782995] agpgart-amdk7 0000:00:00.0: AGP aperture is 64M @ 0xf8000000
Then I think you are likely very lmiited in how much DMA memory you can map (may
be limited to ZONE_DMA), which is likely from where this problem is stemming.
Try adding swiotlb= setup to the kernel command line and play with the size to
see if you can avoid the problem. I also have a patch I'm sending you to test.
Neil
--
Meelis Roos (mroos@linux.ee)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Neil Horman <nhorman@tuxdriver.com> Date: 2014-09-16 14:32:43
Noted that 3c59x has no checks on transmit for failed DMA mappings, and no
ability to unmap fragments when a single map fails in the middle of a transmit.
This patch provides error checking to ensure that dma mappings work properly,
and unrolls an skb mapping if a fragmented skb transmission has a mapping
failure to prevent leaks.
Please test this patch out. It may fix the problem your experiencing, and, if
you enable dynamic debug in the driver, it will certainly confirm that is the
problem you are experiencing.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
---
drivers/net/ethernet/3com/3c59x.c | 50 ++++++++++++++++++++++++++++++++-------
1 file changed, 41 insertions(+), 9 deletions(-)
@@ -2217,7 +2245,11 @@ boomerang_start_xmit(struct sk_buff *skb, struct net_device *dev)skb_tx_timestamp(skb);iowrite16(DownUnstall,ioaddr+EL3_CMD);spin_unlock_irqrestore(&vp->lock,flags);+out:returnNETDEV_TX_OK;+out_dma_err:+dev_err(&VORTEX_PCI(vp)->dev,"Error mapping dma buffer\n");+gotoout;}/* The interrupt handler does all of the Rx thread work and cleans up
I'm guessing the above change has uncovered another bug,
Neil, read your patch carefully, I think it added the bug.
The ->page_offset of the frag gets applied two times.
skb_dma_map_frag() already takes frag->page_offset into consideration,
you then pass it in as the 'offset' argument and it then gets added to
itself to compute th final offset.
I'm guessing the above change has uncovered another bug,
Neil, read your patch carefully, I think it added the bug.
The ->page_offset of the frag gets applied two times.
skb_dma_map_frag() already takes frag->page_offset into consideration,
you then pass it in as the 'offset' argument and it then gets added to
itself to compute th final offset.
Shit, you're right, sorry about that. Its odd, I'm running it here, and its not
causing problems, but thats obviously wrong. Meelis, please add the above fix
to your test and confirm that it sovles the problem. If you could keep the
previous patch in place too that would be great, as we should probably add the
dma error checking anyway.
[PATCH] 3c59x: Fix bad offset spec in skb_frag_dma_map
Recently aded the use of skb_frag_dma_map to 3c59x, but didn't realize it
automatically included the frag_offset internally, as well as provided an option
to specify an extra offset in the parameter list. We need to specify an offset
of 0 in the parameter list to avoid skb corruption that results in lost
connections.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
---
drivers/net/ethernet/3com/3c59x.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Shit, you're right, sorry about that. Its odd, I'm running it here, and its not
causing problems, but thats obviously wrong. Meelis, please add the above fix
to your test and confirm that it sovles the problem. If you could keep the
previous patch in place too that would be great, as we should probably add the
dma error checking anyway.
[PATCH] 3c59x: Fix bad offset spec in skb_frag_dma_map
Tested 2 variants: only this patch (backported to old state) and both
patches together.
Both work fine.
--
Meelis Roos (mroos@linux.ee)
From: Neil Horman <nhorman@tuxdriver.com> Date: 2014-09-17 12:57:03
On Wed, Sep 17, 2014 at 03:43:54PM +0300, mroos@linux.ee wrote:
quoted
Shit, you're right, sorry about that. Its odd, I'm running it here, and its not
causing problems, but thats obviously wrong. Meelis, please add the above fix
to your test and confirm that it sovles the problem. If you could keep the
previous patch in place too that would be great, as we should probably add the
dma error checking anyway.
[PATCH] 3c59x: Fix bad offset spec in skb_frag_dma_map
Tested 2 variants: only this patch (backported to old state) and both
patches together.
Both work fine.
Thank you, Meelis. Dave, I'll propose these patches officially in just a bit
sorry for the
From: Neil Horman <nhorman@tuxdriver.com> Date: 2014-09-17 13:05:19
Noted that 3c59x has no checks on transmit for failed DMA mappings, and no
ability to unmap fragments when a single map fails in the middle of a transmit.
This patch provides error checking to ensure that dma mappings work properly,
and unrolls an skb mapping if a fragmented skb transmission has a mapping
failure to prevent leaks.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: Linux Kernel list <redacted>
CC: "David S. Miller" <davem@davemloft.net>
CC: Meelis Roos <redacted>
Tested-by: Meelis Roos <redacted>
---
drivers/net/ethernet/3com/3c59x.c | 50 ++++++++++++++++++++++++++++++++-------
1 file changed, 41 insertions(+), 9 deletions(-)
@@ -2217,7 +2245,11 @@ boomerang_start_xmit(struct sk_buff *skb, struct net_device *dev)skb_tx_timestamp(skb);iowrite16(DownUnstall,ioaddr+EL3_CMD);spin_unlock_irqrestore(&vp->lock,flags);+out:returnNETDEV_TX_OK;+out_dma_err:+dev_err(&VORTEX_PCI(vp)->dev,"Error mapping dma buffer\n");+gotoout;}/* The interrupt handler does all of the Rx thread work and cleans up
From: Neil Horman <nhorman@tuxdriver.com> Date: 2014-09-17 13:05:48
Recently aded the use of skb_frag_dma_map to 3c59x, but didn't realize it
automatically included the frag_offset internally, as well as provided an option
to specify an extra offset in the parameter list. We need to specify an offset
of 0 in the parameter list to avoid skb corruption that results in lost
connections.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: Linux Kernel list <redacted>
CC: "David S. Miller" <davem@davemloft.net>
CC: Meelis Roos <redacted>
Tested-by: Meelis Roos <redacted>
---
drivers/net/ethernet/3com/3c59x.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
Noted that 3c59x has no checks on transmit for failed DMA mappings, and no
ability to unmap fragments when a single map fails in the middle of a transmit.
This patch provides error checking to ensure that dma mappings work properly,
and unrolls an skb mapping if a fragmented skb transmission has a mapping
failure to prevent leaks.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: Linux Kernel list <redacted>
CC: "David S. Miller" <davem@davemloft.net>
CC: Meelis Roos <redacted>
Tested-by: Meelis Roos <redacted>
Recently aded the use of skb_frag_dma_map to 3c59x, but didn't realize it
automatically included the frag_offset internally, as well as provided an option
to specify an extra offset in the parameter list. We need to specify an offset
of 0 in the parameter list to avoid skb corruption that results in lost
connections.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: Linux Kernel list <redacted>
CC: "David S. Miller" <davem@davemloft.net>
CC: Meelis Roos <redacted>
Tested-by: Meelis Roos <redacted>
Noted that 3c59x has no checks on transmit for failed DMA mappings, and no
ability to unmap fragments when a single map fails in the middle of a transmit.
This patch provides error checking to ensure that dma mappings work properly,
and unrolls an skb mapping if a fragmented skb transmission has a mapping
failure to prevent leaks.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
CC: Linux Kernel list <redacted>
CC: "David S. Miller" <davem@davemloft.net>
CC: Meelis Roos <redacted>
Tested-by: Meelis Roos <redacted>