Hi Maxime,
On 5/24/2021 6:01 AM, Maxime Ripard wrote:
Hi Doug, Florian,
I've been running a RaspberryPi4 with a mainline kernel for a while,
booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
I'm getting a kernel panic around the time init is started.
I was debugging a kernel based on drm-misc-next-2021-05-17 today with
KASAN enabled and got this, which looks related:
Is there a known good version that could be used for bisection or you
just started to do this test and you have no reference point?
How stable in terms of clocking is the configuration that you are using?
I could try to fire up a similar test on a Pi4 at home, or use one of
our 72112 systems which is the closest we have to a Pi4 and see if that
happens there as well.
From: Maxime Ripard <hidden> Date: 2021-05-24 15:24:20
Hi Florian,
On Mon, May 24, 2021 at 07:49:25AM -0700, Florian Fainelli wrote:
Hi Maxime,
On 5/24/2021 6:01 AM, Maxime Ripard wrote:
quoted
Hi Doug, Florian,
I've been running a RaspberryPi4 with a mainline kernel for a while,
booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
I'm getting a kernel panic around the time init is started.
I was debugging a kernel based on drm-misc-next-2021-05-17 today with
KASAN enabled and got this, which looks related:
Is there a known good version that could be used for bisection or you
just started to do this test and you have no reference point?
I've had this issue for over a year and never (I think?) got a good
version, so while it might be a regression, it's not a recent one.
How stable in terms of clocking is the configuration that you are using?
I could try to fire up a similar test on a Pi4 at home, or use one of
our 72112 systems which is the closest we have to a Pi4 and see if that
happens there as well.
I'm not really sure about the clocking. Is there any clock you want to
look at in particular?
My setup is fairly simple: the firmware and kernel are loaded over TFTP
and the rootfs is mounted over NFS, and the crash always occur around
init start, so I guess when it actually starts to transmit a decent
amount of data?
Maxime
Hi Florian,
On Mon, May 24, 2021 at 07:49:25AM -0700, Florian Fainelli wrote:
quoted
Hi Maxime,
On 5/24/2021 6:01 AM, Maxime Ripard wrote:
quoted
Hi Doug, Florian,
I've been running a RaspberryPi4 with a mainline kernel for a while,
booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
I'm getting a kernel panic around the time init is started.
I was debugging a kernel based on drm-misc-next-2021-05-17 today with
KASAN enabled and got this, which looks related:
Is there a known good version that could be used for bisection or you
just started to do this test and you have no reference point?
I've had this issue for over a year and never (I think?) got a good
version, so while it might be a regression, it's not a recent one.
OK, this helps and does not really help.
quoted
How stable in terms of clocking is the configuration that you are using?
I could try to fire up a similar test on a Pi4 at home, or use one of
our 72112 systems which is the closest we have to a Pi4 and see if that
happens there as well.
I'm not really sure about the clocking. Is there any clock you want to
look at in particular?
ARM, DDR, AXI, anything that could cause some memory corruption to occur
essentially. GENET clocks are fairly fixed, you have a 250MHz clock and
a 125MHz clock feeding the data path.
My setup is fairly simple: the firmware and kernel are loaded over TFTP
and the rootfs is mounted over NFS, and the crash always occur around
init start, so I guess when it actually starts to transmit a decent
amount of data?
Do you reproduce this problem with KASAN disabled, do you eventually
have a crash pointing back to the same location?
I have a suspicion that this is all Pi4 specific because we regularly
run the GENET driver through various kernel versions (4.9, 5.4 and 5.10
and mainline) and did not run into that.
--
Florian
Hi Florian,
On Mon, May 24, 2021 at 07:49:25AM -0700, Florian Fainelli wrote:
quoted
Hi Maxime,
On 5/24/2021 6:01 AM, Maxime Ripard wrote:
quoted
Hi Doug, Florian,
I've been running a RaspberryPi4 with a mainline kernel for a while,
booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
I'm getting a kernel panic around the time init is started.
I was debugging a kernel based on drm-misc-next-2021-05-17 today with
KASAN enabled and got this, which looks related:
Is there a known good version that could be used for bisection or you
just started to do this test and you have no reference point?
I've had this issue for over a year and never (I think?) got a good
version, so while it might be a regression, it's not a recent one.
OK, this helps and does not really help.
quoted
quoted
How stable in terms of clocking is the configuration that you are using?
I could try to fire up a similar test on a Pi4 at home, or use one of
our 72112 systems which is the closest we have to a Pi4 and see if that
happens there as well.
I'm not really sure about the clocking. Is there any clock you want to
look at in particular?
ARM, DDR, AXI, anything that could cause some memory corruption to occur
essentially. GENET clocks are fairly fixed, you have a 250MHz clock and
a 125MHz clock feeding the data path.
quoted
My setup is fairly simple: the firmware and kernel are loaded over TFTP
and the rootfs is mounted over NFS, and the crash always occur around
init start, so I guess when it actually starts to transmit a decent
amount of data?
Do you reproduce this problem with KASAN disabled, do you eventually
have a crash pointing back to the same location?
I have a suspicion that this is all Pi4 specific because we regularly
run the GENET driver through various kernel versions (4.9, 5.4 and 5.10
and mainline) and did not run into that.
I have not had time to get a set-up to reproduce what you are seeing,
could you share your .config meanwhile? Thanks
--
Florian
From: Maxime Ripard <hidden> Date: 2021-05-28 16:32:43
hi Florian,
On Fri, May 28, 2021 at 09:21:27AM -0700, Florian Fainelli wrote:
On 5/24/21 8:37 AM, Florian Fainelli wrote:
quoted
On 5/24/2021 8:13 AM, Maxime Ripard wrote:
quoted
Hi Florian,
On Mon, May 24, 2021 at 07:49:25AM -0700, Florian Fainelli wrote:
quoted
Hi Maxime,
On 5/24/2021 6:01 AM, Maxime Ripard wrote:
quoted
Hi Doug, Florian,
I've been running a RaspberryPi4 with a mainline kernel for a while,
booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
I'm getting a kernel panic around the time init is started.
I was debugging a kernel based on drm-misc-next-2021-05-17 today with
KASAN enabled and got this, which looks related:
Is there a known good version that could be used for bisection or you
just started to do this test and you have no reference point?
I've had this issue for over a year and never (I think?) got a good
version, so while it might be a regression, it's not a recent one.
OK, this helps and does not really help.
quoted
quoted
How stable in terms of clocking is the configuration that you are using?
I could try to fire up a similar test on a Pi4 at home, or use one of
our 72112 systems which is the closest we have to a Pi4 and see if that
happens there as well.
I'm not really sure about the clocking. Is there any clock you want to
look at in particular?
ARM, DDR, AXI, anything that could cause some memory corruption to occur
essentially. GENET clocks are fairly fixed, you have a 250MHz clock and
a 125MHz clock feeding the data path.
quoted
My setup is fairly simple: the firmware and kernel are loaded over TFTP
and the rootfs is mounted over NFS, and the crash always occur around
init start, so I guess when it actually starts to transmit a decent
amount of data?
Do you reproduce this problem with KASAN disabled, do you eventually
have a crash pointing back to the same location?
I have a suspicion that this is all Pi4 specific because we regularly
run the GENET driver through various kernel versions (4.9, 5.4 and 5.10
and mainline) and did not run into that.
I have not had time to get a set-up to reproduce what you are seeing,
could you share your .config meanwhile? Thanks
Sorry, I didn't have the time to check how the clock were behaving.
You'll find attached my config.txt file and .config
I'm booting the board entirely from TFTP (which might introduce some
issues in the "handoff" from the bootloader to the kernel), you'll find
some guide there:
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md
Maxime
hi Florian,
On Fri, May 28, 2021 at 09:21:27AM -0700, Florian Fainelli wrote:
quoted
On 5/24/21 8:37 AM, Florian Fainelli wrote:
quoted
On 5/24/2021 8:13 AM, Maxime Ripard wrote:
quoted
Hi Florian,
On Mon, May 24, 2021 at 07:49:25AM -0700, Florian Fainelli wrote:
quoted
Hi Maxime,
On 5/24/2021 6:01 AM, Maxime Ripard wrote:
quoted
Hi Doug, Florian,
I've been running a RaspberryPi4 with a mainline kernel for a while,
booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
I'm getting a kernel panic around the time init is started.
I was debugging a kernel based on drm-misc-next-2021-05-17 today with
KASAN enabled and got this, which looks related:
Is there a known good version that could be used for bisection or you
just started to do this test and you have no reference point?
I've had this issue for over a year and never (I think?) got a good
version, so while it might be a regression, it's not a recent one.
OK, this helps and does not really help.
quoted
quoted
How stable in terms of clocking is the configuration that you are using?
I could try to fire up a similar test on a Pi4 at home, or use one of
our 72112 systems which is the closest we have to a Pi4 and see if that
happens there as well.
I'm not really sure about the clocking. Is there any clock you want to
look at in particular?
ARM, DDR, AXI, anything that could cause some memory corruption to occur
essentially. GENET clocks are fairly fixed, you have a 250MHz clock and
a 125MHz clock feeding the data path.
quoted
My setup is fairly simple: the firmware and kernel are loaded over TFTP
and the rootfs is mounted over NFS, and the crash always occur around
init start, so I guess when it actually starts to transmit a decent
amount of data?
Do you reproduce this problem with KASAN disabled, do you eventually
have a crash pointing back to the same location?
I have a suspicion that this is all Pi4 specific because we regularly
run the GENET driver through various kernel versions (4.9, 5.4 and 5.10
and mainline) and did not run into that.
I have not had time to get a set-up to reproduce what you are seeing,
could you share your .config meanwhile? Thanks
Sorry, I didn't have the time to check how the clock were behaving.
You'll find attached my config.txt file and .config
I'm booting the board entirely from TFTP (which might introduce some
issues in the "handoff" from the bootloader to the kernel), you'll find
some guide there:
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
--
Florian
hi Florian,
On Fri, May 28, 2021 at 09:21:27AM -0700, Florian Fainelli wrote:
quoted
On 5/24/21 8:37 AM, Florian Fainelli wrote:
quoted
On 5/24/2021 8:13 AM, Maxime Ripard wrote:
quoted
Hi Florian,
On Mon, May 24, 2021 at 07:49:25AM -0700, Florian Fainelli wrote:
quoted
Hi Maxime,
On 5/24/2021 6:01 AM, Maxime Ripard wrote:
quoted
Hi Doug, Florian,
I've been running a RaspberryPi4 with a mainline kernel for a while,
booting from NFS. Every once in a while (I'd say ~20-30% of all boots),
I'm getting a kernel panic around the time init is started.
I was debugging a kernel based on drm-misc-next-2021-05-17 today with
KASAN enabled and got this, which looks related:
Is there a known good version that could be used for bisection or you
just started to do this test and you have no reference point?
I've had this issue for over a year and never (I think?) got a good
version, so while it might be a regression, it's not a recent one.
OK, this helps and does not really help.
quoted
quoted
How stable in terms of clocking is the configuration that you are using?
I could try to fire up a similar test on a Pi4 at home, or use one of
our 72112 systems which is the closest we have to a Pi4 and see if that
happens there as well.
I'm not really sure about the clocking. Is there any clock you want to
look at in particular?
ARM, DDR, AXI, anything that could cause some memory corruption to occur
essentially. GENET clocks are fairly fixed, you have a 250MHz clock and
a 125MHz clock feeding the data path.
quoted
My setup is fairly simple: the firmware and kernel are loaded over TFTP
and the rootfs is mounted over NFS, and the crash always occur around
init start, so I guess when it actually starts to transmit a decent
amount of data?
Do you reproduce this problem with KASAN disabled, do you eventually
have a crash pointing back to the same location?
I have a suspicion that this is all Pi4 specific because we regularly
run the GENET driver through various kernel versions (4.9, 5.4 and 5.10
and mainline) and did not run into that.
I have not had time to get a set-up to reproduce what you are seeing,
could you share your .config meanwhile? Thanks
Sorry, I didn't have the time to check how the clock were behaving.
You'll find attached my config.txt file and .config
I'm booting the board entirely from TFTP (which might introduce some
issues in the "handoff" from the bootloader to the kernel), you'll find
some guide there:
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net_tutorial.md
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
--
Florian
From: nicolas saenz julienne <nsaenz@kernel.org> Date: 2021-06-01 09:33:25
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Regards,
Nicolas
From: Maxime Ripard <hidden> Date: 2021-06-02 13:28:29
On Tue, Jun 01, 2021 at 11:33:18AM +0200, nicolas saenz julienne wrote:
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Yeah, it looks like it's only from a cold boot and comes in "bursts",
where you would get like 5 in a row and be done with it for a while.
Maxime
On Tue, Jun 01, 2021 at 11:33:18AM +0200, nicolas saenz julienne wrote:
quoted
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Yeah, it looks like it's only from a cold boot and comes in "bursts",
where you would get like 5 in a row and be done with it for a while.
Here are two patches that you could try exclusive from one another
1) Limit GENET to a single queue
From: Maxime Ripard <hidden> Date: 2021-06-25 12:59:14
Hi Florian,
Sorry for the late reply
On Thu, Jun 10, 2021 at 02:33:17PM -0700, Florian Fainelli wrote:
quoted hunk
On 6/2/2021 6:28 AM, Maxime Ripard wrote:
quoted
On Tue, Jun 01, 2021 at 11:33:18AM +0200, nicolas saenz julienne wrote:
quoted
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Yeah, it looks like it's only from a cold boot and comes in "bursts",
where you would get like 5 in a row and be done with it for a while.
Here are two patches that you could try exclusive from one another
1) Limit GENET to a single queue
bcmgenet_priv *priv,
/* Returns a reusable dma control register value */
static u32 bcmgenet_dma_disable(struct bcmgenet_priv *priv)
{
+ unsigned int i;
u32 reg;
u32 dma_ctrl;
/* disable DMA */
dma_ctrl = 1 << (DESC_INDEX + DMA_RING_BUF_EN_SHIFT) | DMA_EN;
+ for (i = 0; i < priv->hw_params->tx_queues; i++)
+ dma_ctrl |= (1 << (i + DMA_RING_BUF_EN_SHIFT));
reg = bcmgenet_tdma_readl(priv, DMA_CTRL);
reg &= ~dma_ctrl;
bcmgenet_tdma_writel(priv, reg, DMA_CTRL);
+ dma_ctrl = 1 << (DESC_INDEX + DMA_RING_BUF_EN_SHIFT) | DMA_EN;
+ for (i = 0; i < priv->hw_params->rx_queues; i++)
+ dma_ctrl |= (1 << (i + DMA_RING_BUF_EN_SHIFT));
reg = bcmgenet_rdma_readl(priv, DMA_CTRL);
reg &= ~dma_ctrl;
bcmgenet_rdma_writel(priv, reg, DMA_CTRL);
I had a bunch of issues popping up today so I took the occasion to test
those patches. The first one doesn't change anything, I still had the
crash occurring with it. With the second applied (in addition), it seems
like it's fixed. I'll keep testing and will let you know.
Maxime
Hey Maxime,
On 6/25/2021 5:59 AM, Maxime Ripard wrote:
Hi Florian,
Sorry for the late reply
On Thu, Jun 10, 2021 at 02:33:17PM -0700, Florian Fainelli wrote:
quoted
On 6/2/2021 6:28 AM, Maxime Ripard wrote:
quoted
On Tue, Jun 01, 2021 at 11:33:18AM +0200, nicolas saenz julienne wrote:
quoted
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Yeah, it looks like it's only from a cold boot and comes in "bursts",
where you would get like 5 in a row and be done with it for a while.
Here are two patches that you could try exclusive from one another
1) Limit GENET to a single queue
bcmgenet_priv *priv,
/* Returns a reusable dma control register value */
static u32 bcmgenet_dma_disable(struct bcmgenet_priv *priv)
{
+ unsigned int i;
u32 reg;
u32 dma_ctrl;
/* disable DMA */
dma_ctrl = 1 << (DESC_INDEX + DMA_RING_BUF_EN_SHIFT) | DMA_EN;
+ for (i = 0; i < priv->hw_params->tx_queues; i++)
+ dma_ctrl |= (1 << (i + DMA_RING_BUF_EN_SHIFT));
reg = bcmgenet_tdma_readl(priv, DMA_CTRL);
reg &= ~dma_ctrl;
bcmgenet_tdma_writel(priv, reg, DMA_CTRL);
+ dma_ctrl = 1 << (DESC_INDEX + DMA_RING_BUF_EN_SHIFT) | DMA_EN;
+ for (i = 0; i < priv->hw_params->rx_queues; i++)
+ dma_ctrl |= (1 << (i + DMA_RING_BUF_EN_SHIFT));
reg = bcmgenet_rdma_readl(priv, DMA_CTRL);
reg &= ~dma_ctrl;
bcmgenet_rdma_writel(priv, reg, DMA_CTRL);
I had a bunch of issues popping up today so I took the occasion to test
those patches. The first one doesn't change anything, I still had the
crash occurring with it. With the second applied (in addition), it seems
like it's fixed. I'll keep testing and will let you know.
Did this patch survive more days of testing? I am tempted to send it
regardless of your testing because it is a correctness issue that is
being fixed. There is a global DMA enable bit which should "cut" any
TX/RX queues, but still, for symmetry with other code paths all queues
should be disabled.
Thanks!
--
Florian
From: Maxime Ripard <hidden> Date: 2021-07-06 08:16:56
Hi Florian,
On Fri, Jul 02, 2021 at 09:49:31AM -0700, Florian Fainelli wrote:
On 6/25/2021 5:59 AM, Maxime Ripard wrote:
quoted
Hi Florian,
Sorry for the late reply
On Thu, Jun 10, 2021 at 02:33:17PM -0700, Florian Fainelli wrote:
quoted
On 6/2/2021 6:28 AM, Maxime Ripard wrote:
quoted
On Tue, Jun 01, 2021 at 11:33:18AM +0200, nicolas saenz julienne wrote:
quoted
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Yeah, it looks like it's only from a cold boot and comes in "bursts",
where you would get like 5 in a row and be done with it for a while.
Here are two patches that you could try exclusive from one another
1) Limit GENET to a single queue
bcmgenet_priv *priv,
/* Returns a reusable dma control register value */
static u32 bcmgenet_dma_disable(struct bcmgenet_priv *priv)
{
+ unsigned int i;
u32 reg;
u32 dma_ctrl;
/* disable DMA */
dma_ctrl = 1 << (DESC_INDEX + DMA_RING_BUF_EN_SHIFT) | DMA_EN;
+ for (i = 0; i < priv->hw_params->tx_queues; i++)
+ dma_ctrl |= (1 << (i + DMA_RING_BUF_EN_SHIFT));
reg = bcmgenet_tdma_readl(priv, DMA_CTRL);
reg &= ~dma_ctrl;
bcmgenet_tdma_writel(priv, reg, DMA_CTRL);
+ dma_ctrl = 1 << (DESC_INDEX + DMA_RING_BUF_EN_SHIFT) | DMA_EN;
+ for (i = 0; i < priv->hw_params->rx_queues; i++)
+ dma_ctrl |= (1 << (i + DMA_RING_BUF_EN_SHIFT));
reg = bcmgenet_rdma_readl(priv, DMA_CTRL);
reg &= ~dma_ctrl;
bcmgenet_rdma_writel(priv, reg, DMA_CTRL);
I had a bunch of issues popping up today so I took the occasion to test
those patches. The first one doesn't change anything, I still had the
crash occurring with it. With the second applied (in addition), it seems
like it's fixed. I'll keep testing and will let you know.
Did this patch survive more days of testing? I am tempted to send it
regardless of your testing because it is a correctness issue that is being
fixed. There is a global DMA enable bit which should "cut" any TX/RX queues,
but still, for symmetry with other code paths all queues should be disabled.
Unfortunately, I haven't spent too much time working on mainline
recently, so I didn't really have the occasion to test further that
patch.
It seems to make sense anyway like you said, so you can definitely send
it, with my Tested-by :)
Maxime
From: Maxime Ripard <hidden> Date: 2022-05-13 14:57:11
Hi Florian,
Sorry for reviving this old thread...
On Thu, Jun 10, 2021 at 02:33:17PM -0700, Florian Fainelli wrote:
quoted hunk
On 6/2/2021 6:28 AM, Maxime Ripard wrote:
quoted
On Tue, Jun 01, 2021 at 11:33:18AM +0200, nicolas saenz julienne wrote:
quoted
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Yeah, it looks like it's only from a cold boot and comes in "bursts",
where you would get like 5 in a row and be done with it for a while.
Here are two patches that you could try exclusive from one another
1) Limit GENET to a single queue
Hi Florian,
Sorry for reviving this old thread...
On Thu, Jun 10, 2021 at 02:33:17PM -0700, Florian Fainelli wrote:
quoted
On 6/2/2021 6:28 AM, Maxime Ripard wrote:
quoted
On Tue, Jun 01, 2021 at 11:33:18AM +0200, nicolas saenz julienne wrote:
quoted
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Yeah, it looks like it's only from a cold boot and comes in "bursts",
where you would get like 5 in a row and be done with it for a while.
Here are two patches that you could try exclusive from one another
1) Limit GENET to a single queue
How do I reproduce this reliably? What version of GCC did you build your
kernel with? How often does that happen? What config.txt file are you
using for your Pi4 B?
--
Florian
From: Maxime Ripard <hidden> Date: 2022-05-17 07:53:11
On Sat, May 14, 2022 at 09:35:42AM -0700, Florian Fainelli wrote:
On 5/13/2022 7:56 AM, Maxime Ripard wrote:
quoted
Hi Florian,
Sorry for reviving this old thread...
On Thu, Jun 10, 2021 at 02:33:17PM -0700, Florian Fainelli wrote:
quoted
On 6/2/2021 6:28 AM, Maxime Ripard wrote:
quoted
On Tue, Jun 01, 2021 at 11:33:18AM +0200, nicolas saenz julienne wrote:
quoted
On Mon, 2021-05-31 at 19:36 -0700, Florian Fainelli wrote:
quoted
quoted
That is also how I boot my Pi4 at home, and I suspect you are right, if
the VPU does not shut down GENET's DMA, and leaves buffer addresses in
the on-chip descriptors that point to an address space that is managed
totally differently by Linux, then we can have a serious problem and
create some memory corruption when the ring is being reclaimed. I will
run a few experiments to test that theory and there may be a solution
using the SW_INIT reset controller to have a big reset of the controller
before handing it over to the Linux driver.
Adding a WARN_ON(reg & DMA_EN) in bcmgenet_dma_disable() has not shown
that the TX or RX DMA have been left running during the hand over from
the VPU to the kernel. I checked out drm-misc-next-2021-05-17 to reduce
as much as possible the differences between your set-up and my set-up
but so far have not been able to reproduce the crash in booting from NFS
repeatedly, I will try again.
FWIW I can reproduce the error too. That said it's rather hard to reproduce,
something in the order of 1 failure every 20 tries.
Yeah, it looks like it's only from a cold boot and comes in "bursts",
where you would get like 5 in a row and be done with it for a while.
Here are two patches that you could try exclusive from one another
1) Limit GENET to a single queue
It's not really 100% reliable, but happens 30%-50% of the time at boot
when KASAN is enabled. It seems like enabling KASAN increases that
likelihood though, it went unnoticed for some time before I started
having those issues again when I enabled it for something unrelated.
It looks like it happens in bursts though, so I would get 10-15 boots
fine, and then 4-5 boots with that crash.
Cold boot vs reboot doesn't seem to affect it in one way or the other.
What version of GCC did you build your kernel with?
The arm64 cross-compiler packaged by Fedora, which is GCC 11.2
at the moment.
How often does that happen? What config.txt file are you using
for your Pi4 B?
You'll find my config.txt and kernel .config attached
Thanks!
Maxime
It's not really 100% reliable, but happens 30%-50% of the time at boot
when KASAN is enabled. It seems like enabling KASAN increases that
likelihood though, it went unnoticed for some time before I started
having those issues again when I enabled it for something unrelated.
It looks like it happens in bursts though, so I would get 10-15 boots
fine, and then 4-5 boots with that crash.
Cold boot vs reboot doesn't seem to affect it in one way or the other.
quoted
What version of GCC did you build your kernel with?
The arm64 cross-compiler packaged by Fedora, which is GCC 11.2
at the moment.
quoted
How often does that happen? What config.txt file are you using
for your Pi4 B?
You'll find my config.txt and kernel .config attached
OK, so this is what I have been able to reproduce so far but this does
not appear to be very reliable to reproduce, I will try my best to hold
on to that lead though, thanks for your patience.
# udhcpc -i eth0
udhcpc: started, v1.35.0
[ 34.355086] bcmgenet fd580000.ethernet: configuring instance for
external RGMII (RX delay)
[ 34.363758]
==================================================================
[ 34.371106] BUG: KASAN: user-memory-access in put_page+0x10/0x64
[ 34.377227] Read of size 4 at addr 01000085 by task ifconfig/165
[ 34.383338]
[ 34.384857] CPU: 0 PID: 165 Comm: ifconfig Tainted: G W
5.19.0 #43
[ 34.392560] Hardware name: BCM2711
[ 34.396020] unwind_backtrace from show_stack+0x18/0x1c
[ 34.401354] show_stack from dump_stack_lvl+0x40/0x4c
[ 34.406502] dump_stack_lvl from kasan_report+0x8c/0xa4
[ 34.411825] kasan_report from put_page+0x10/0x64
[ 34.416615] put_page from skb_release_data+0x84/0x13c
[ 34.421847] skb_release_data from __kfree_skb+0x14/0x20
[ 34.427256] __kfree_skb from bcmgenet_rx_poll+0x504/0x6f8
[ 34.432846] bcmgenet_rx_poll from __napi_poll.constprop.0+0x50/0x1c0
[ 34.439407] __napi_poll.constprop.0 from net_rx_action+0x278/0x488
[ 34.445787] net_rx_action from __do_softirq+0x268/0x390
[ 34.451197] __do_softirq from __irq_exit_rcu+0x88/0xf8
[ 34.456521] __irq_exit_rcu from irq_exit+0x10/0x18
[ 34.461492] irq_exit from call_with_stack+0x18/0x20
[ 34.466553] call_with_stack from __irq_svc+0x84/0x94
[ 34.471696] Exception stack(0xf0d337f8 to 0xf0d33840)
[ 34.476835] 37e0:
c5548580 00000003
[ 34.485156] 3800: 00002000 f0a40808 c5548000 c5548580 00000000
c554b000 c5548580 c554bdd0
[ 34.493474] 3820: 00000000 00000004 c5548580 f0d33848 c094329c
c09432bc 00070013 ffffffff
[ 34.501788] __irq_svc from bcmgenet_open+0xe1c/0x1094
[ 34.507023] bcmgenet_open from __dev_open+0x1e4/0x21c
[ 34.512258] __dev_open from __dev_change_flags+0x228/0x25c
[ 34.517931] __dev_change_flags from dev_change_flags+0x48/0x88
[ 34.523958] dev_change_flags from devinet_ioctl+0x3ac/0x834
[ 34.529723] devinet_ioctl from inet_ioctl+0x250/0x2a4
[ 34.534956] inet_ioctl from sock_ioctl+0x1dc/0x410
[ 34.539927] sock_ioctl from vfs_ioctl+0x50/0x64
[ 34.544632] vfs_ioctl from sys_ioctl+0x134/0xa7c
[ 34.549422] sys_ioctl from ret_fast_syscall+0x0/0x4c
[ 34.554565] Exception stack(0xf0d33fa8 to 0xf0d33ff0)
[ 34.559705] 3fa0: 0051fd98 0053f9dc 00000003
00008914 b6dc5c4c b6dc5bd0
[ 34.568025] 3fc0: 0051fd98 0053f9dc b6dc5f55 00000036 b6dc5e48
00000003 aed11d00 aed12010
[ 34.576341] 3fe0: 00000036 b6dc5bb8 aec4c2f3 aebdda66
[ 34.581475]
==================================================================
[ 34.588882] Disabling lock debugging due to kernel taint
[ 34.594288] 8<--- cut here ---
[ 34.597412] Unable to handle kernel paging request at virtual address
01000085
[ 34.604775] [01000085] *pgd=01982003, *pmd=00000000
[ 34.609751] Internal error: Oops: 206 [#1] SMP ARM
[ 34.614624] Modules linked in:
[ 34.617734] CPU: 0 PID: 165 Comm: ifconfig Tainted: G B W
5.19.0 #43
[ 34.625435] Hardware name: BCM2711
[ 34.628892] PC is at put_page+0x14/0x64
[ 34.632800] LR is at kasan_report+0x98/0xa4
[ 34.637056] pc : [<c0b4bee4>] lr : [<c047ea5c>] psr: 60070113
[ 34.643427] sp : f0803d50 ip : 00000000 fp : c554bfd8
[ 34.648739] r10: 00007f5e r9 : c694f582 r8 : c1fef15e
[ 34.654052] r7 : c694f5b8 r6 : c694f580 r5 : 01000081 r4 : c1fef100
[ 34.660689] r3 : 00000000 r2 : c1f047c0 r1 : 00000004 r0 : 00000001
[ 34.667325] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM
Segment user
[ 34.674582] Control: 30c5383d Table: 0606b700 DAC: fffffffd
[ 34.680422] Register r0 information: non-paged memory
[ 34.685565] Register r1 information: non-paged memory
[ 34.690705] Register r2 information: slab task_struct start c1f047c0
pointer offset 0
[ 34.698690] Register r3 information: NULL pointer
[ 34.703477] Register r4 information: slab skbuff_head_cache start
c1fef100 pointer offset 0 size 48
[ 34.712699] Register r5 information: non-paged memory
[ 34.717839] Register r6 information: non-slab/vmalloc memory
[ 34.723595] Register r7 information: non-slab/vmalloc memory
[ 34.729352] Register r8 information: slab skbuff_head_cache start
c1fef100 pointer offset 94 size 48
[ 34.738662] Register r9 information: non-slab/vmalloc memory
[ 34.744419] Register r10 information: non-paged memory
[ 34.749646] Register r11 information: non-slab/vmalloc memory
[ 34.755492] Register r12 information: NULL pointer
[ 34.760366] Process ifconfig (pid: 165, stack limit = 0xf517d551)
[ 34.766573] Stack: (0xf0803d50 to 0xf0804000)
[ 34.771005] 3d40: c1fef100
00000001 c694f580 c0b4dc74
[ 34.779325] 3d60: c1fef100 c5548000 c5548580 c1fef100 f0803e40
7f5e0001 00007f5e c0b4db24
[ 34.787644] 3d80: c554bdd0 c0940f84 0bc80000 b4c23195 c2cb12c0
c0efdab0 c2cb12c0 00000001
[ 34.795963] 3da0: 00000000 00000040 00000004 c554bec4 1e1007bc
c554beb8 c5548588 00000004
[ 34.804282] 3dc0: c55498bc c554bec8 c02d5684 00000003 00000000
c02b6e10 e7df0980 c02bf390
[ 34.812601] 3de0: 41b58ab3 c15fec7a c0940a80 c1f047c0 00070113
257ac000 e7de97cc ffff982d
[ 34.820919] 3e00: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 b4c23195
[ 34.829237] 3e20: c1f047c0 e7de8680 00000000 c1f047c0 00000000
c076733c e7de9ad8 00000000
[ 34.837556] 3e40: e7de97d4 c613e0a0 00000001 c554bdd0 00000001
00000040 f0803ef0 c554bdd8
[ 34.845875] 3e60: 257ac000 c2805d40 e7df0d00 c0b70f24 c554bdd0
f0803ef0 00000000 e7df0b40
[ 34.854195] 3e80: f0803f60 bd1007d8 c554bdd0 c2644b40 257ac000
c0b7130c 0000012c e7df0d0c
[ 34.862513] 3ea0: ffff9839 f0803ef0 81d99054 c554bdd4 0000002c
257ac000 c26433c8 c0840554
[ 34.870832] 3ec0: 41b58ab3 c1612850 c0b71094 c2cb12c0 e7df0980
c02d8a5c ea8ed400 c02d8ae0
[ 34.879150] 3ee0: 41b58ab3 c15f3580 c08403c4 00000010 c554bd00
c554bdd8 00000000 00000010
[ 34.887470] 3f00: f0803f00 f0803f00 c5548580 00002000 c554bdd0
c554b580 0000010a c093e0b8
[ 34.895788] 3f20: f0803f20 f0803f20 0000002c c093df98 c2806f18
c029f4ac 00000000 00000007
[ 34.904108] 3f40: e7de9780 c02a4218 00000104 c4dca800 00000001
c4dca824 c4dca86c c4dca86c
[ 34.912427] 3f60: c4dca848 f0803fc8 f0d337f0 b4c23195 c4dca800
c1f047c0 c280508c 00000008
[ 34.920747] 3f80: c2643dc0 c1f047c4 00000003 00000100 c1f049d4
c02014d8 c4dca800 c1f047c0
[ 34.929066] 3fa0: 00400100 0000000a ffff9838 00000004 c263c3c8
257ac000 c26433c0 c1f047c0
[ 34.937385] 3fc0: c2643dc0 c1f047c4 257ac000 257ac000 c1f047c0
00000000 f0d337f0 c02312c4
[ 34.945704] 3fe0: c09432bc 00070013 ffffffff f0d3382c c5548580
c0231418 c09432bc c07559fc
[ 34.954019] put_page from skb_release_data+0x84/0x13c
[ 34.959252] skb_release_data from __kfree_skb+0x14/0x20
[ 34.964660] __kfree_skb from bcmgenet_rx_poll+0x504/0x6f8
[ 34.970250] bcmgenet_rx_poll from __napi_poll.constprop.0+0x50/0x1c0
[ 34.976812] __napi_poll.constprop.0 from net_rx_action+0x278/0x488
[ 34.983192] net_rx_action from __do_softirq+0x268/0x390
[ 34.988602] __do_softirq from __irq_exit_rcu+0x88/0xf8
[ 34.993927] __irq_exit_rcu from irq_exit+0x10/0x18
[ 34.998899] irq_exit from call_with_stack+0x18/0x20
[ 35.003958] call_with_stack from __irq_svc+0x84/0x94
[ 35.009101] Exception stack(0xf0d337f8 to 0xf0d33840)
[ 35.014238] 37e0:
c5548580 00000003
[ 35.022557] 3800: 00002000 f0a40808 c5548000 c5548580 00000000
c554b000 c5548580 c554bdd0
[ 35.030877] 3820: 00000000 00000004 c5548580 f0d33848 c094329c
c09432bc 00070013 ffffffff
[ 35.039192] __irq_svc from bcmgenet_open+0xe1c/0x1094
[ 35.044427] bcmgenet_open from __dev_open+0x1e4/0x21c
[ 35.049661] __dev_open from __dev_change_flags+0x228/0x25c
[ 35.055334] __dev_change_flags from dev_change_flags+0x48/0x88
[ 35.061361] dev_change_flags from devinet_ioctl+0x3ac/0x834
[ 35.067125] devinet_ioctl from inet_ioctl+0x250/0x2a4
[ 35.072359] inet_ioctl from sock_ioctl+0x1dc/0x410
[ 35.077330] sock_ioctl from vfs_ioctl+0x50/0x64
[ 35.082034] vfs_ioctl from sys_ioctl+0x134/0xa7c
[ 35.086825] sys_ioctl from ret_fast_syscall+0x0/0x4c
[ 35.091969] Exception stack(0xf0d33fa8 to 0xf0d33ff0)
[ 35.097109] 3fa0: 0051fd98 0053f9dc 00000003
00008914 b6dc5c4c b6dc5bd0
[ 35.105428] 3fc0: 0051fd98 0053f9dc b6dc5f55 00000036 b6dc5e48
00000003 aed11d00 aed12010
[ 35.113744] 3fe0: 00000036 b6dc5bb8 aec4c2f3 aebdda66
[ 35.118883] Code: e1a05000 e2800004 ebe4cca7 e3a01004 (e5953004)
[ 35.125104] ---[ end trace 0000000000000000 ]---
[ 35.129801] Kernel panic - not syncing: Fatal exception in interrupt
[ 35.136260] CPU3: stopping
[ 35.139009] CPU: 3 PID: 27 Comm: migration/3 Tainted: G B D W
5.19.0 #43
[ 35.146872] Hardware name: BCM2711
[ 35.150318] Stopper: multi_cpu_stop+0x0/0x140 <-
stop_machine_cpuslocked+0x180/0x1e4
[ 35.158197] unwind_backtrace from show_stack+0x18/0x1c
[ 35.163509] show_stack from dump_stack_lvl+0x40/0x4c
[ 35.168643] dump_stack_lvl from do_handle_IPI+0x150/0x2a8
[ 35.174218] do_handle_IPI from ipi_handler+0x1c/0x28
[ 35.179351] ipi_handler from handle_percpu_devid_irq+0x94/0x150
[ 35.185454] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48
[ 35.191820] handle_irq_desc from gic_handle_irq+0x6c/0x78
[ 35.197393] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c
[ 35.203671] generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 35.210038] call_with_stack from __irq_svc+0x84/0x94
[ 35.215168] Exception stack(0xf0913e98 to 0xf0913ee0)
[ 35.220293] 3e80:
e7e20a10 00000000
[ 35.228594] 3ea0: 00000000 257dc000 e7e1ec68 f0913ee8 257dc000
00000000 c2806f18 60070013
[ 35.236896] 3ec0: f0863d70 f0863d74 f0863d70 f0913ee8 c02bebd4
c02bebe8 60070013 ffffffff
[ 35.245192] __irq_svc from rcu_momentary_dyntick_idle+0x2c/0x9c
[ 35.251296] rcu_momentary_dyntick_idle from multi_cpu_stop+0xd4/0x140
[ 35.257931] multi_cpu_stop from cpu_stopper_thread+0x120/0x1d8
[ 35.263947] cpu_stopper_thread from smpboot_thread_fn+0x25c/0x264
[ 35.270228] smpboot_thread_fn from kthread+0x12c/0x140
[ 35.275539] kthread from ret_from_fork+0x14/0x1c
[ 35.280317] Exception stack(0xf0913fb0 to 0xf0913ff8)
[ 35.285441] 3fa0: 00000000
00000000 00000000 00000000
[ 35.293739] 3fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[ 35.302037] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 35.308746] CPU2: stopping
[ 35.311492] CPU: 2 PID: 22 Comm: migration/2 Tainted: G B D W
5.19.0 #43
[ 35.319355] Hardware name: BCM2711
[ 35.322803] Stopper: multi_cpu_stop+0x0/0x140 <-
stop_machine_cpuslocked+0x180/0x1e4
[ 35.330677] unwind_backtrace from show_stack+0x18/0x1c
[ 35.335988] show_stack from dump_stack_lvl+0x40/0x4c
[ 35.341122] dump_stack_lvl from do_handle_IPI+0x150/0x2a8
[ 35.346697] do_handle_IPI from ipi_handler+0x1c/0x28
[ 35.351830] ipi_handler from handle_percpu_devid_irq+0x94/0x150
[ 35.357932] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48
[ 35.364298] handle_irq_desc from gic_handle_irq+0x6c/0x78
[ 35.369870] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c
[ 35.376148] generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 35.382515] call_with_stack from __irq_svc+0x84/0x94
[ 35.387646] Exception stack(0xf08ebea8 to 0xf08ebef0)
[ 35.392773] bea0: f0863d70 00000003 00000000
00000001 f0863d60 00000000
[ 35.401074] bec0: 00000001 00000000 c2806f18 600c0013 f0863d70
f0863d74 f0863d70 f08ebef8
[ 35.409372] bee0: c030acac c02bebbc 600c0013 ffffffff
[ 35.414495] __irq_svc from rcu_momentary_dyntick_idle+0x0/0x9c
[ 35.420511] rcu_momentary_dyntick_idle from 0xc31d0000
[ 35.425820] CPU1: stopping
[ 35.428568] CPU: 1 PID: 17 Comm: migration/1 Tainted: G B D W
5.19.0 #43
[ 35.436430] Hardware name: BCM2711
[ 35.439879] Stopper: multi_cpu_stop+0x0/0x140 <-
stop_machine_cpuslocked+0x180/0x1e4
[ 35.447752] unwind_backtrace from show_stack+0x18/0x1c
[ 35.453064] show_stack from dump_stack_lvl+0x40/0x4c
[ 35.458198] dump_stack_lvl from do_handle_IPI+0x150/0x2a8
[ 35.463772] do_handle_IPI from ipi_handler+0x1c/0x28
[ 35.468905] ipi_handler from handle_percpu_devid_irq+0x94/0x150
[ 35.475006] handle_percpu_devid_irq from handle_irq_desc+0x38/0x48
[ 35.481373] handle_irq_desc from gic_handle_irq+0x6c/0x78
[ 35.486945] gic_handle_irq from generic_handle_arch_irq+0x28/0x3c
[ 35.493222] generic_handle_arch_irq from call_with_stack+0x18/0x20
[ 35.499590] call_with_stack from __irq_svc+0x84/0x94
[ 35.504721] Exception stack(0xf08c3e98 to 0xf08c3ee0)
[ 35.509847] 3e80:
e7e00a10 00000000
[ 35.518148] 3ea0: 00000000 257bc000 e7dfec68 f08c3ee8 257bc000
00000000 c2806f18 600f0013
[ 35.526449] 3ec0: f0863d70 f0863d74 f0863d70 f08c3ee8 c02bebd4
c02bebe8 600f0013 ffffffff
[ 35.534745] __irq_svc from rcu_momentary_dyntick_idle+0x2c/0x9c
[ 35.540849] rcu_momentary_dyntick_idle from multi_cpu_stop+0xd4/0x140
[ 35.547483] multi_cpu_stop from cpu_stopper_thread+0x120/0x1d8
[ 35.553499] cpu_stopper_thread from smpboot_thread_fn+0x25c/0x264
[ 35.559780] smpboot_thread_fn from kthread+0x12c/0x140
[ 35.565090] kthread from ret_from_fork+0x14/0x1c
[ 35.569868] Exception stack(0xf08c3fb0 to 0xf08c3ff8)
[ 35.574992] 3fa0: 00000000
00000000 00000000 00000000
[ 35.583292] 3fc0: 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000
[ 35.591589] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[ 35.599291] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---
--
Florian
From: Maxime Ripard <hidden> Date: 2022-08-15 07:07:41
Hi Florian,
On Thu, Aug 11, 2022 at 08:33:58PM -0700, Florian Fainelli wrote:
On 5/17/2022 12:52 AM, Maxime Ripard wrote:
quoted
It's not really 100% reliable, but happens 30%-50% of the time at boot
when KASAN is enabled. It seems like enabling KASAN increases that
likelihood though, it went unnoticed for some time before I started
having those issues again when I enabled it for something unrelated.
It looks like it happens in bursts though, so I would get 10-15 boots
fine, and then 4-5 boots with that crash.
Cold boot vs reboot doesn't seem to affect it in one way or the other.
quoted
What version of GCC did you build your kernel with?
The arm64 cross-compiler packaged by Fedora, which is GCC 11.2
at the moment.
quoted
How often does that happen? What config.txt file are you using
for your Pi4 B?
You'll find my config.txt and kernel .config attached
OK, so this is what I have been able to reproduce so far but this does not
appear to be very reliable to reproduce, I will try my best to hold on to
that lead though, thanks for your patience.
# udhcpc -i eth0
udhcpc: started, v1.35.0
[ 34.355086] bcmgenet fd580000.ethernet: configuring instance for external
RGMII (RX delay)
[ 34.363758]
==================================================================
[ 34.371106] BUG: KASAN: user-memory-access in put_page+0x10/0x64
[ 34.377227] Read of size 4 at addr 01000085 by task ifconfig/165
[ 34.383338]
[ 34.384857] CPU: 0 PID: 165 Comm: ifconfig Tainted: G W 5.19.0
#43
[ 34.392560] Hardware name: BCM2711
[ 34.396020] unwind_backtrace from show_stack+0x18/0x1c
[ 34.401354] show_stack from dump_stack_lvl+0x40/0x4c
[ 34.406502] dump_stack_lvl from kasan_report+0x8c/0xa4
[ 34.411825] kasan_report from put_page+0x10/0x64
[ 34.416615] put_page from skb_release_data+0x84/0x13c
[ 34.421847] skb_release_data from __kfree_skb+0x14/0x20
[ 34.427256] __kfree_skb from bcmgenet_rx_poll+0x504/0x6f8
[ 34.432846] bcmgenet_rx_poll from __napi_poll.constprop.0+0x50/0x1c0
[ 34.439407] __napi_poll.constprop.0 from net_rx_action+0x278/0x488
[ 34.445787] net_rx_action from __do_softirq+0x268/0x390
[ 34.451197] __do_softirq from __irq_exit_rcu+0x88/0xf8
[ 34.456521] __irq_exit_rcu from irq_exit+0x10/0x18
[ 34.461492] irq_exit from call_with_stack+0x18/0x20
[ 34.466553] call_with_stack from __irq_svc+0x84/0x94
[ 34.471696] Exception stack(0xf0d337f8 to 0xf0d33840)
It looks fairly close indeed.
There's a bunch of notable differences though (user-memory-access vs
wild-memory-access, the read size) but the type of memory access error
can just be due to the randomness of the memory address we try to
access, and the read 4 vs 8 could be because you're running on ARM and
I'm running on arm64?
Thanks again for looking into it
Maxime