Re: at91sam9260 MACB problem with IP fragmentation

Erwin,

On 12/06/2012 12:32 PM, Erwin Rol :

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1
copy of the Atmel EK).

The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The
test case is that a udp echo client (PC) sends datagrams with increasing
length to the AT91 until the max length of the UDP datagram is reached.
When there is no IP fragmentation everything is fine, but when the
datagrams are starting to get fragmented the AT91 will not reply
anymore. But as soon as some network traffic happens it goes on again,
and non of the data is lost.

With wireshark the effect can be easily seen (192.168.1.4 is the PC echo
client, and 192.168.1.133 is the at91 echo server) After the first
request there comes no reply. After a 5 second timeout the second
request is send. And then both replies are returned.

When I enabled debugging output it all started to work. So I tried some
udelays in the driver instead of printk and with a 1ms delay in the irq
handler it started working. Of course that is an unacceptable fix, but
it looks like that is some weird race condition that causes the sending
to stall. The only difference with normal MTU sized datagrams I can
think of is that the fragmented packets can be passed very quickly to
the macb tx function, because the kernel has all 5 skb's ready.

I would be very interested to hear if someone else could reproduce this
problem. Or even better, has seen this problem and has a fix for it.

I tried several kernels including the test version from Nicolas that he
posted on LKML in October. They all show the same effect.

[..]

It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html

I am working on the macb driver right now, so I will try to reproduce
and track this issue on my side.

Best regards,
-- 
Nicolas Ferre

Hey Nicolas,

On 6-12-2012 14:27, Nicolas Ferre wrote:

Erwin,

On 12/06/2012 12:32 PM, Erwin Rol :

quoted

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1
copy of the Atmel EK).

 <snip>

[..]

It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html

The difference seems to be that in Matteo's case the receiving stalls.
In my case it is the sending that stalls. I see the UDP datagram in
userspace and the sendto call also returns without error, but the data
does not end up on the network (until the next packet is send)

I am working on the macb driver right now, so I will try to reproduce
and track this issue on my side.

That would be really great, and thank you for the quick reply. If you
have anything that I should try or test on my hardware just let me know.

BTW: A quick check on a at91sam9263 board did not show the problem. I
will try to verify if it really does work on a 9263, cause maybe it just
more rare on a 9263.

- Erwin

Hallo Nicolas,

On 6-12-2012 14:27, Nicolas Ferre wrote:

Erwin,

On 12/06/2012 12:32 PM, Erwin Rol :

quoted

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1
copy of the Atmel EK).

The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The
test case is that a udp echo client (PC) sends datagrams with increasing
length to the AT91 until the max length of the UDP datagram is reached.
When there is no IP fragmentation everything is fine, but when the
datagrams are starting to get fragmented the AT91 will not reply
anymore. But as soon as some network traffic happens it goes on again,
and non of the data is lost.

<snip>

quoted

I tried several kernels including the test version from Nicolas that he
posted on LKML in October. They all show the same effect.

[..]

It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html

I tried Matteo's patch and it seems to work. But I don't know if the
patch is really the right solution. I checked again with wireshark and
it really seems the sending that stalls not the receiving. But as soon
as a ethernet frame is received the sending "un-stalls". So maybe the
patch just causes an MACB IRQ at certain moments that causes the sending
to continue?

I am working on the macb driver right now, so I will try to reproduce
and track this issue on my side.

Any luck reproducing it ?


- Erwin

On 12/20/2012 10:17 AM, Erwin Rol :

Hallo Nicolas,

On 6-12-2012 14:27, Nicolas Ferre wrote:

quoted

Erwin,

On 12/06/2012 12:32 PM, Erwin Rol :

quoted

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1
copy of the Atmel EK).

The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The
test case is that a udp echo client (PC) sends datagrams with increasing
length to the AT91 until the max length of the UDP datagram is reached.
When there is no IP fragmentation everything is fine, but when the
datagrams are starting to get fragmented the AT91 will not reply
anymore. But as soon as some network traffic happens it goes on again,
and non of the data is lost.

<snip>

quoted

quoted

I tried several kernels including the test version from Nicolas that he
posted on LKML in October. They all show the same effect.

[..]

It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html

I tried Matteo's patch and it seems to work. But I don't know if the
patch is really the right solution. I checked again with wireshark and
it really seems the sending that stalls not the receiving. But as soon
as a ethernet frame is received the sending "un-stalls". So maybe the
patch just causes an MACB IRQ at certain moments that causes the sending
to continue?

Any digging is interesting for me.

quoted

I am working on the macb driver right now, so I will try to reproduce
and track this issue on my side.

Any luck reproducing it ?

Yes, I see unexpected things happening but as I am connected to a whole
company network so maybe some broadcast packets are unlocking the
interface...
Anyway, I am continuing to investigate.

Best regards,--
Nicolas Ferre

When interrupts are disabled, an RX condition can occur but
it is not reported when enabling interrupts again. We need to check
RSR and use napi_reschedule() if condition is met.

Signed-off-by: Nicolas Ferre <redacted>
---
 drivers/net/ethernet/cadence/macb.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/net/ethernet/cadence/macb.c b/drivers/net/ethernet/cadence/macb.c
index a9b0830..b9d4bb9 100644
--- a/drivers/net/ethernet/cadence/macb.c
+++ b/drivers/net/ethernet/cadence/macb.c

@@ -693,6 +693,11 @@ static int macb_poll(struct napi_struct *napi, int budget)
 		 * get notified when new packets arrive.
 		 */
 		macb_writel(bp, IER, MACB_RX_INT_FLAGS);
+
+		/* Packets received while interrupts were disabled */
+		status = macb_readl(bp, RSR);
+		if (unlikely(status))
+			napi_reschedule(napi);
 	}
 
 	/* TODO: Handle errors */

-- 
1.8.0

From: Nicolas Ferre <redacted>
Date: Tue, 12 Feb 2013 11:08:48 +0100

When interrupts are disabled, an RX condition can occur but
it is not reported when enabling interrupts again. We need to check
RSR and use napi_reschedule() if condition is met.

Signed-off-by: Nicolas Ferre <redacted>

Applied.