FEC_IEVENT_RFIFO_ERROR

Hi!

I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
for MPC5200 chip on a custom board that is almost lite5200 compatible.
I noticed a couple of times I have a strange error at bootup.
It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
went trough without problems but since today system just hangs.
Sometimes with several printouts of this error.
---boot sequence ------
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
....

I traced a problem a bit and found that this happenes at
mpc52xx_fec_probe() function in fec.c at this point:
-----------------------------------------------------------------------------------------
      /* Get the IRQ we need one by one */
               /* Control */
       dev->irq = ocp->def->irq;
-->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT,                        "mpc52xx_fec_ctrl", dev)) {
               printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request failed\n");
               ret = -EBUSY;
               dev->irq = -1;  /* Don't try to free it */
               goto probe_error;
       }
------------------------------------------------------------------------------------------

When fec_interrupt() is called also from fec.c.

------------------------------------------------------------------------------------------
static irqreturn_t fec_interrupt(int irq, void *dev_id, struct pt_regs *regs)
{
       struct net_device *dev = (struct net_device *)dev_id;
       struct fec_priv *priv = (struct fec_priv *)dev->priv;
       struct mpc52xx_fec *fec = priv->fec;
       int ievent;

       ievent = in_be32(&fec->ievent);
       out_be32(&fec->ievent, ievent);         /* clear pending events */

       if (ievent & (FEC_IEVENT_RFIFO_ERROR | FEC_IEVENT_XFIFO_ERROR)) {
               if (ievent & FEC_IEVENT_RFIFO_ERROR)
-->                        printk(KERN_WARNING "FEC_IEVENT_RFIFO_ERROR\n");
               if (ievent & FEC_IEVENT_XFIFO_ERROR)
                       printk(KERN_WARNING "FEC_IEVENT_XFIFO_ERROR\n");
               fec_reinit(dev);
       }
       else if (ievent & FEC_IEVENT_MII)
               fec_mii(dev);
       return IRQ_HANDLED;
}
-------------------------------------------------------------------------------------------------


This is what I found in MPC5200 Users Manual:
Receive FIFO Error--indicates error occurred within the forest green version
RX FIFO. When RFIFO_ERROR bit is set, ECNTRL.ETHER_EN is cleared,
halting FEC frame processing. When this occurs, software must ensure both
the FIFO Controller and BestComm are soft-reset.

Any ideas on what could be causing this?

Hi

I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
for MPC5200 chip on a custom board that is almost lite5200 compatible.
I noticed a couple of times I have a strange error at bootup.
It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
went trough without problems but since today system just hangs.
Sometimes with several printouts of this error.
---boot sequence ------
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
....

Theses are definitly not "normal" but you said "since today it just hangs",
did something change in the environment of the card ?

I traced a problem a bit and found that this happenes at
mpc52xx_fec_probe() function in fec.c at this point:
-----------------------------------------------------------------------------------------

      /* Get the IRQ we need one by one */
               /* Control */
       dev->irq = ocp->def->irq;
-->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT,                        "mpc52xx_fec_ctrl", dev)) {
               printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request failed\n");
               ret = -EBUSY;
               dev->irq = -1;  /* Don't try to free it */
               goto probe_error;
       }
------------------------------------------------------------------------------------------

It ovbiously can't happen before since the message it printed in that interrupt
handler. But it should not happen there either (not so early) !

This error globally says : "Somthing got wrong with the receive buffer". But
at this point, frame reception is not yet enabled, how could it go wrong ? Unless
your bootloader don't take care of shutting down the fec, then frames may be
stuck in the fifo between the bootloader and the fec init ...

This is what I found in MPC5200 Users Manual:
Receive FIFO Error--indicates error occurred within the forest green version
RX FIFO. When RFIFO_ERROR bit is set, ECNTRL.ETHER_EN is cleared,
halting FEC frame processing. When this occurs, software must ensure both
the FIFO Controller and BestComm are soft-reset.

Any ideas on what could be causing this?

I can't explain why this happen so early at init (as I said before) but other things that could
cause such an event :
- We don't have enough buffer descriptors : The bescomm task just fill them all and runs out
  of them before the interrupt is handled.
- The bestcomm engine don't flush the RX fifo quicly enough. Currently the only tasks
- Bestcomm stopped processing for whatever reason ...
- Something else that I don't see at the moment. I'll try to "stress test" network a little bit,
see if I can reproduce the issue.

In the mean time, pull the latest change, I just pushed some fixes related to frame reception,
I don't think it's related to your issue but ...



   Sylvain

Sylvain Munaut wrote:

Hi

quoted

I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
for MPC5200 chip on a custom board that is almost lite5200 compatible.
I noticed a couple of times I have a strange error at bootup.
It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
went trough without problems but since today system just hangs.
Sometimes with several printouts of this error.
---boot sequence ------
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
....

Theses are definitly not "normal" but you said "since today it just hangs",
did something change in the environment of the card ?

I changed the card yesterday and I got fresh bootloader on. I don't compile
bootloader, I receive it with the board. Maybe I should change this practice.
Anyway as I said I got the new board yesterday with some hardware fixes and I
managed to get it up after I set up bootloader environment. It worked until this mornig.
I was working on my ( noob :D ) custom drivers and executed another
cycle of make/copy image/reboot  and noticed that for some reason
fec.c and fec_phy.c got also recompiled. After that, things stoped working.
I really don't know why those two got into the make routine but that was
the end of fun. :D
I'm stuck at this error now and lot's of new stuff to learn. :D I don't mind that
though although I got completely of track with what I was doing before.

quoted

I traced a problem a bit and found that this happenes at
mpc52xx_fec_probe() function in fec.c at this point:
-----------------------------------------------------------------------------------------

      /* Get the IRQ we need one by one */
               /* Control */
       dev->irq = ocp->def->irq;
-->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT,                        "mpc52xx_fec_ctrl", dev)) {
               printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request failed\n");
               ret = -EBUSY;
               dev->irq = -1;  /* Don't try to free it */
               goto probe_error;
       }
------------------------------------------------------------------------------------------

It ovbiously can't happen before since the message it printed in that interrupt
handler. But it should not happen there either (not so early) !

This error globally says : "Somthing got wrong with the receive buffer". But
at this point, frame reception is not yet enabled, how could it go wrong ? Unless
your bootloader don't take care of shutting down the fec, then frames may be
stuck in the fifo between the bootloader and the fec init ...

I think I understand what you're saying.
Hmmm. I really don't know what bootloader takes care of.
What should I do to check this? Should I try with another
version of bootloader? BTW I got the board with U-Boot 1.1.1.

quoted

This is what I found in MPC5200 Users Manual:
Receive FIFO Error--indicates error occurred within the forest green version
RX FIFO. When RFIFO_ERROR bit is set, ECNTRL.ETHER_EN is cleared,
halting FEC frame processing. When this occurs, software must ensure both
the FIFO Controller and BestComm are soft-reset.

Any ideas on what could be causing this?

I can't explain why this happen so early at init (as I said before) but other things that could
cause such an event :
- We don't have enough buffer descriptors : The bescomm task just fill them all and runs out
  of them before the interrupt is handled.
- The bestcomm engine don't flush the RX fifo quicly enough. Currently the only tasks
- Bestcomm stopped processing for whatever reason ...
- Something else that I don't see at the moment. I'll try to "stress test" network a little bit,
see if I can reproduce the issue.

In the mean time, pull the latest change, I just pushed some fixes related to frame reception,
I don't think it's related to your issue but ...

Ok. I will try with this now.

Sylvain Munaut wrote:

In the mean time, pull the latest change, I just pushed some fixes related to frame reception,
I don't think it's related to your issue but ...



   Sylvain

No luck unfortunatly. I pulled/recompiled and still have the same error. :(

On Thu, Mar 03, 2005 at 01:14:54PM +0000, Babarovic Ivica wrote:

I run 2.6.10-rc2 kernel (http://www.246tNt.com/mpc52xx/)
for MPC5200 chip on a custom board that is almost lite5200 compatible.
I noticed a couple of times I have a strange error at bootup.
It was FEC_IEVENT_RFIFO_ERROR. Most of the times this
went trough without problems but since today system just hangs.
Sometimes with several printouts of this error.
---boot sequence ------
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
FEC_IEVENT_RFIFO_ERROR
....

I traced a problem a bit and found that this happenes at
mpc52xx_fec_probe() function in fec.c at this point:
--------------------------------------------------------------------------
      /* Get the IRQ we need one by one */
               /* Control */
       dev->irq = ocp->def->irq;
-->       if (request_irq(dev->irq, &fec_interrupt, SA_INTERRUPT, 
                       "mpc52xx_fec_ctrl", dev)) {
               printk(KERN_ERR "mpc52xx_fec: ctrl interrupt request 
failed\n");
               ret = -EBUSY;
               dev->irq = -1;  /* Don't try to free it */
               goto probe_error;
       }
--------------------------------------------------------------------------

It looks like the bootloader left the FEC enabled.  You might try
the following (untested) patch which resets the FEC and disables
its interrupts before requesting the irq.

-Dale

===== drivers/net/fec_mpc52xx/fec.c 1.1 vs edited =====

--- 1.1/drivers/net/fec_mpc52xx/fec.c	2004-11-20 15:26:33 -07:00
+++ edited/drivers/net/fec_mpc52xx/fec.c	2005-03-03 09:43:08 -07:00

@@ -85,18 +85,13 @@
 	return 0;
 }
 
-/* This function is called to start or restart the FEC during a link
- * change.  This happens on fifo errors or when switching between half
- * and full duplex.
- */
-static void fec_restart(struct net_device *dev, int duplex)
+static void fec_reset(struct net_device *dev)
 {
 	struct fec_priv *priv = (struct fec_priv *)dev->priv;
 	struct mpc52xx_fec *fec = priv->fec;
-	u32 rcntrl;
-	u32 tcntrl;
 	int i;
 
+	out_be32(&priv->fec->imask, 0);		/* mask all interrupts */
 	out_be32(&fec->rfifo_status, in_be32(&fec->rfifo_status) & 0x700000);
 	out_be32(&fec->tfifo_status, in_be32(&fec->tfifo_status) & 0x700000);
 	out_be32(&fec->reset_cntrl, 0x1000000);

@@ -110,6 +105,20 @@
 	}
 	if (i == FEC_RESET_DELAY)
 		printk (KERN_ERR "FEC Reset timeout!\n");
+}
+
+/* This function is called to start or restart the FEC during a link
+ * change.  This happens on fifo errors or when switching between half
+ * and full duplex.
+ */
+static void fec_restart(struct net_device *dev, int duplex)
+{
+	struct fec_priv *priv = (struct fec_priv *)dev->priv;
+	struct mpc52xx_fec *fec = priv->fec;
+	u32 rcntrl;
+	u32 tcntrl;
+
+	fec_reset(dev);
 
 	/* Set station address. */
 	fec_set_paddr(dev, dev->dev_addr);

@@ -645,6 +654,8 @@
 	ret = sdma_fec_tx_init(priv->tx_sdma, priv->tx_fifo);
 	if (ret < 0)
 		goto probe_error;
+
+	fec_reset(dev);
 
 	/* Get the IRQ we need one by one */
 		/* Control */

Hi!

I tried the patch you've sent and It's still the same.
FEC_IEVENT_RFIFO_ERROR is still here.
Can you/anyone think of something that I could do to make
analyzing this problem easier?



Dale Farnsworth wrote:

Hi

I tried the patch you've sent and It's still the same.
FEC_IEVENT_RFIFO_ERROR is still here.
Can you/anyone think of something that I could do to make
analyzing this problem easier?

Resetting the FEC early is a good thing so that patch should be there anyway, so we don't setup irq handlers for a hardware in unknow state.

However looking at fec code, there is another problem with it : The FEC is enabled in probe, once for all but the DMA tasks that should empty
the buffers are only enabled when the interface is up, so during it's down time, any received packed ends up in filling the FIFO ...

This problem goes away when the interface is up tough, so you should not experience 'crash' ... Do you nfs boot ? Try with a static image, just
to see, maybe having the error at boot screw something. If it doesn't boot with a static image then there is still another problem.


   Sylvain