--- Exception: 501 at .pseries_dedicated_idle_sleep+0xe4/0x210
LR = .pseries_dedicated_idle_sleep+0x8c/0x210
[c000000000dffd00] [c00000000005b194] .pseries_dedicated_idle_sleep+0x194/0x210
(unreliable)
[c000000000dffdc0] [c000000000018c84] .cpu_idle+0x164/0x210
[c000000000dffe70] [c00000000000b0d0] .rest_init+0x90/0xb0
[c000000000dffef0] [c000000000830bc0] .start_kernel+0x54c/0x56c
[c000000000dfff90] [c00000000000953c] .start_here_common+0x1c/0x60
Its caused when skb_gro_reset_offset attempts to call PageHighMem on
skb_shinfo(skb)->frags[0].page, when the frags array was left uninitalized.
This can happen in the ixgbe driver if the hardware reports a zero length rx
descriptor ni the middle of a packet split receive transaction. I've consulted
with Jesse Brandeburg on this, who is attempting to root cause the issue at
Intel, but it seems prudent to add this check to the driver to discard frames of
that encounter this error to avoid the opps
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Signed-off-by: Thadeu Lima de Souza Cascardo <redacted>
CC: Jesse Brandeburg <redacted>
CC: Alexander Duyck <redacted>
CC: John Fastabend <redacted>
CC: Jeff Kirsher <redacted>
CC: David S. Miller <davem@davemloft.net>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 17 +++++++++++------
1 files changed, 11 insertions(+), 6 deletions(-)
--- Exception: 501 at .pseries_dedicated_idle_sleep+0xe4/0x210
LR = .pseries_dedicated_idle_sleep+0x8c/0x210
[c000000000dffd00] [c00000000005b194] .pseries_dedicated_idle_sleep
+0x194/0x210
(unreliable)
[c000000000dffdc0] [c000000000018c84] .cpu_idle+0x164/0x210
[c000000000dffe70] [c00000000000b0d0] .rest_init+0x90/0xb0
[c000000000dffef0] [c000000000830bc0] .start_kernel+0x54c/0x56c
[c000000000dfff90] [c00000000000953c] .start_here_common+0x1c/0x60
Its caused when skb_gro_reset_offset attempts to call PageHighMem on
skb_shinfo(skb)->frags[0].page, when the frags array was left
uninitalized.
This can happen in the ixgbe driver if the hardware reports a zero
length rx
descriptor ni the middle of a packet split receive transaction. I've
consulted
with Jesse Brandeburg on this, who is attempting to root cause the
issue at
Intel, but it seems prudent to add this check to the driver to discard
frames of
that encounter this error to avoid the opps
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Signed-off-by: Thadeu Lima de Souza Cascardo
[off-list ref]
CC: Jesse Brandeburg <redacted>
CC: Alexander Duyck <redacted>
CC: John Fastabend <redacted>
CC: Jeff Kirsher <redacted>
CC: David S. Miller <davem@davemloft.net>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 17
+++++++++++------
1 files changed, 11 insertions(+), 6 deletions(-)
Thanks Neil, I have added the patch to my queue of ixgbe patches.
This patch was made against net-next, was this issue only seen on the
net-next kernel?
From: Alexander Duyck <hidden> Date: 2011-09-02 16:16:25
This kind of fix just opens up a whole can of security related worms. If you are going to discard a packet you should do it after we have reached the EOP in the series. My advice would be to determine what traits identify this packet and add those to the check for the IXGBE_RXDADV_ERR_FRAME_ERR_MASK check further down in the code. Likely what you are seeing is skb_headlen(skb) will be equal to 0.
I'm suspecting this is some sort of read corruption. It looks like in order to trigger it you have to either be reading rx_buffer_info->dma as 0, or the header length is being read as 0. Do you know if you actually have header split enabled when this is occuring? Are you running with jumbo frames enabled to see the issue? If not then packet split wouldn't be enabled.
Is this occurring on net-next or on an older kernel? I just want to be sure since we added a read memory barrier in 2.6.34 to address the fact that the length and descriptor DD bits were being read in the wrong order resulting in the length being corrupted on PowerPC systems. The fact that we are now seeing another length error on PowerPC seems very odd.
Thanks,
Alex
On 09/02/2011 07:03 AM, Neil Horman wrote:
--- Exception: 501 at .pseries_dedicated_idle_sleep+0xe4/0x210
LR = .pseries_dedicated_idle_sleep+0x8c/0x210
[c000000000dffd00] [c00000000005b194] .pseries_dedicated_idle_sleep+0x194/0x210
(unreliable)
[c000000000dffdc0] [c000000000018c84] .cpu_idle+0x164/0x210
[c000000000dffe70] [c00000000000b0d0] .rest_init+0x90/0xb0
[c000000000dffef0] [c000000000830bc0] .start_kernel+0x54c/0x56c
[c000000000dfff90] [c00000000000953c] .start_here_common+0x1c/0x60
Its caused when skb_gro_reset_offset attempts to call PageHighMem on
skb_shinfo(skb)->frags[0].page, when the frags array was left uninitalized.
This can happen in the ixgbe driver if the hardware reports a zero length rx
descriptor ni the middle of a packet split receive transaction. I've consulted
with Jesse Brandeburg on this, who is attempting to root cause the issue at
Intel, but it seems prudent to add this check to the driver to discard frames of
that encounter this error to avoid the opps
Signed-off-by: Neil Horman<nhorman@tuxdriver.com>
Signed-off-by: Thadeu Lima de Souza Cascardo<redacted>
CC: Jesse Brandeburg<redacted>
CC: Alexander Duyck<redacted>
CC: John Fastabend<redacted>
CC: Jeff Kirsher<redacted>
CC: David S. Miller<davem@davemloft.net>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 17 +++++++++++------
1 files changed, 11 insertions(+), 6 deletions(-)
--- Exception: 501 at .pseries_dedicated_idle_sleep+0xe4/0x210
LR = .pseries_dedicated_idle_sleep+0x8c/0x210
[c000000000dffd00] [c00000000005b194] .pseries_dedicated_idle_sleep
+0x194/0x210
(unreliable)
[c000000000dffdc0] [c000000000018c84] .cpu_idle+0x164/0x210
[c000000000dffe70] [c00000000000b0d0] .rest_init+0x90/0xb0
[c000000000dffef0] [c000000000830bc0] .start_kernel+0x54c/0x56c
[c000000000dfff90] [c00000000000953c] .start_here_common+0x1c/0x60
Its caused when skb_gro_reset_offset attempts to call PageHighMem on
skb_shinfo(skb)->frags[0].page, when the frags array was left
uninitalized.
This can happen in the ixgbe driver if the hardware reports a zero
length rx
descriptor ni the middle of a packet split receive transaction. I've
consulted
with Jesse Brandeburg on this, who is attempting to root cause the
issue at
Intel, but it seems prudent to add this check to the driver to discard
frames of
that encounter this error to avoid the opps
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Signed-off-by: Thadeu Lima de Souza Cascardo
[off-list ref]
CC: Jesse Brandeburg <redacted>
CC: Alexander Duyck <redacted>
CC: John Fastabend <redacted>
CC: Jeff Kirsher <redacted>
CC: David S. Miller <davem@davemloft.net>
---
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 17
+++++++++++------
1 files changed, 11 insertions(+), 6 deletions(-)
Thanks Neil, I have added the patch to my queue of ixgbe patches.
This patch was made against net-next, was this issue only seen on the
net-next kernel?
From: Neil Horman <nhorman@tuxdriver.com> Date: 2011-09-02 16:55:31
On Fri, Sep 02, 2011 at 09:17:40AM -0700, Alexander Duyck wrote:
This kind of fix just opens up a whole can of security related
worms. If you are going to discard a packet you should do it after
we have reached the EOP in the series. My advice would be to
determine what traits identify this packet and add those to the
check for the IXGBE_RXDADV_ERR_FRAME_ERR_MASK check further down in
the code. Likely what you are seeing is skb_headlen(skb) will be
equal to 0.
Well, the traits of the bogus descriptor are almost exactly as you describe
them, i.e. rx_buffer_info->dma is zero, which the driver takes to mean packet
split is enabled, and this is a buffer in the middle of that operation
(according to the comments in ixgbe_clean_rx_irq), and the upper_len value we
read from the rx_descriptior rx_dex->wb.upper.length is zero. This implies we
have a frame which is in the middle of a packet split receive, and one of the
page long buffers has a length value of zero, which is non-sensical. I suppose
we could wait until the next frame with EOP set to discard the whole thing, but
I'm not sure how that amounts to anything different than just skipping to the
next descriptor.
I'm suspecting this is some sort of read corruption. It looks like
in order to trigger it you have to either be reading
rx_buffer_info->dma as 0, or the header length is being read as 0.
Correct, which drops us into the else clause of the if(rx_buffer_info->dma)
conditional in ixgbe_clean_rx_irq.
Do you know if you actually have header split enabled when this is
occuring? Are you running with jumbo frames enabled to see the
Yes, packet split is enabled. and no, Jumbo frames are not in use.
issue? If not then packet split wouldn't be enabled.
Is this occurring on net-next or on an older kernel? I just want to
be sure since we added a read memory barrier in 2.6.34 to address
the fact that the length and descriptor DD bits were being read in
the wrong order resulting in the length being corrupted on PowerPC
systems. The fact that we are now seeing another length error on
PowerPC seems very odd.
According to the bz:
https://bugzilla.redhat.com/show_bug.cgi?id=683611
This appears to be happening on RHEL, and on upstream kernels, as well as the
sourceforge driver. Don't quote me on the SF driver though, because I never got
a clear answer on that. Although, fwiw, the RHEL version of the driver in which
we were definately seeing this problem has a read memory barrrier at the top of
the loop in ixgbe_clean_rx_irq, pulled in from commit
3c945e5b3719bcc18c6ddd31bbcae8ef94f3d19a, so I think thats handled.
Regards
Neil
From: Alexander Duyck <hidden> Date: 2011-09-02 17:53:10
On 09/02/2011 09:55 AM, Neil Horman wrote:
On Fri, Sep 02, 2011 at 09:17:40AM -0700, Alexander Duyck wrote:
quoted
This kind of fix just opens up a whole can of security related
worms. If you are going to discard a packet you should do it after
we have reached the EOP in the series. My advice would be to
determine what traits identify this packet and add those to the
check for the IXGBE_RXDADV_ERR_FRAME_ERR_MASK check further down in
the code. Likely what you are seeing is skb_headlen(skb) will be
equal to 0.
Well, the traits of the bogus descriptor are almost exactly as you describe
them, i.e. rx_buffer_info->dma is zero, which the driver takes to mean packet
split is enabled, and this is a buffer in the middle of that operation
(according to the comments in ixgbe_clean_rx_irq), and the upper_len value we
read from the rx_descriptior rx_dex->wb.upper.length is zero. This implies we
have a frame which is in the middle of a packet split receive, and one of the
page long buffers has a length value of zero, which is non-sensical. I suppose
we could wait until the next frame with EOP set to discard the whole thing, but
I'm not sure how that amounts to anything different than just skipping to the
next descriptor.
quoted
I'm suspecting this is some sort of read corruption. It looks like
in order to trigger it you have to either be reading
rx_buffer_info->dma as 0, or the header length is being read as 0.
Correct, which drops us into the else clause of the if(rx_buffer_info->dma)
conditional in ixgbe_clean_rx_irq.
quoted
Do you know if you actually have header split enabled when this is
occuring? Are you running with jumbo frames enabled to see the
Yes, packet split is enabled. and no, Jumbo frames are not in use.
quoted
issue? If not then packet split wouldn't be enabled.
Is this occurring on net-next or on an older kernel? I just want to
be sure since we added a read memory barrier in 2.6.34 to address
the fact that the length and descriptor DD bits were being read in
the wrong order resulting in the length being corrupted on PowerPC
systems. The fact that we are now seeing another length error on
PowerPC seems very odd.
According to the bz:
https://bugzilla.redhat.com/show_bug.cgi?id=683611
This appears to be happening on RHEL, and on upstream kernels, as well as the
sourceforge driver. Don't quote me on the SF driver though, because I never got
a clear answer on that. Although, fwiw, the RHEL version of the driver in which
we were definately seeing this problem has a read memory barrrier at the top of
the loop in ixgbe_clean_rx_irq, pulled in from commit
3c945e5b3719bcc18c6ddd31bbcae8ef94f3d19a, so I think thats handled.
Regards
Neil
I'll review the bugzilla and submit my comments there.
Thanks,
Alex
--- Exception: 501 at .pseries_dedicated_idle_sleep+0xe4/0x210
LR = .pseries_dedicated_idle_sleep+0x8c/0x210
[c000000000dffd00] [c00000000005b194] .pseries_dedicated_idle_sleep+0x194/0x210
(unreliable)
[c000000000dffdc0] [c000000000018c84] .cpu_idle+0x164/0x210
[c000000000dffe70] [c00000000000b0d0] .rest_init+0x90/0xb0
[c000000000dffef0] [c000000000830bc0] .start_kernel+0x54c/0x56c
[c000000000dfff90] [c00000000000953c] .start_here_common+0x1c/0x60
Its caused when skb_gro_reset_offset attempts to call PageHighMem on
skb_shinfo(skb)->frags[0].page, when the frags array was left uninitalized.
This can happen in the ixgbe driver if the hardware reports a zero length rx
descriptor ni the middle of a packet split receive transaction. I've consulted
with Jesse Brandeburg on this, who is attempting to root cause the issue at
Intel, but it seems prudent to add this check to the driver to discard frames of
that encounter this error to avoid the opps
Sorry, I need to rescind this patch. Looks like this is turning out to be an
issue with an ideosyncracy in the dma hardware on this platform.
Thanks
Neil
--- Exception: 501 at .pseries_dedicated_idle_sleep+0xe4/0x210
LR = .pseries_dedicated_idle_sleep+0x8c/0x210
[c000000000dffd00] [c00000000005b194] .pseries_dedicated_idle_sleep+0x194/0x210
(unreliable)
[c000000000dffdc0] [c000000000018c84] .cpu_idle+0x164/0x210
[c000000000dffe70] [c00000000000b0d0] .rest_init+0x90/0xb0
[c000000000dffef0] [c000000000830bc0] .start_kernel+0x54c/0x56c
[c000000000dfff90] [c00000000000953c] .start_here_common+0x1c/0x60
Its caused when skb_gro_reset_offset attempts to call PageHighMem on
skb_shinfo(skb)->frags[0].page, when the frags array was left uninitalized.
This can happen in the ixgbe driver if the hardware reports a zero length rx
descriptor ni the middle of a packet split receive transaction. I've consulted
with Jesse Brandeburg on this, who is attempting to root cause the issue at
Intel, but it seems prudent to add this check to the driver to discard frames of
that encounter this error to avoid the opps
Sorry, I need to rescind this patch. Looks like this is turning out to be an
issue with an ideosyncracy in the dma hardware on this platform.
Thanks
Neil
Thanks for the update Neil. I have dropped the patch from my queue.