Toggling link state breaks network connectivity

Hello,

I am using the following drivers for Ethernet connectivity.
drivers/net/ethernet/aurora/nb8800.c
drivers/net/phy/at803x.c

Pulling the cable and plugging it back works as expected.
(I can ping both before and after.)

However, if I toggle the link state in software (using ip link set),
the board loses network connectivity.

# Statically assign IP address
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
# Set link state to "up"
ip link set eth0 up
# ping -c 3 172.27.64.1 > /tmp/v1

PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms

--- 172.27.64.1 ping statistics ---

3 packets transmitted, 1 packets received, 66% packet loss
round-trip min/avg/max = 18.321/18.321/18.321 ms


172.27.64.1 is a desktop system.
Running
% tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
on the desktop, I get:

15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64
15:01:50.213232 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 2, length 64
15:01:51.198563 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
15:01:51.209586 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
15:01:51.209598 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46

Packet #1: the board asks for the desktop's MAC address
Packet #2: the desktop replies instantly
Packet #3: the board sends the first ping
Packet #4: the desktop replies instantly
Then the board goes quiet for a long time (why???)
Packet #5: the desktop asks for the board's MAC address (doesn't it have it already?)
Packet #6: this seems to unwedge the board, which sends the second ping
Packet #7: the desktop replies instantly
Packet #8: the board sends the third ping
Packet #9: the desktop replies instantly
Packet #10: the desktop asks again for the board's MAC address
Packet #11 and #12: the board answers twice (for the old and new requests?)

Some oddities, but it seems to work.

Now toggle the link state:

% ip link set eth0 down

5 more lines

--- 172.27.64.1 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss


On the desktop, I see

15:14:03.900162 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:14:03.900175 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:14:05.017189 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:14:05.017200 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:14:06.030531 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:14:06.030541 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28

So basically, the board is asking the desktop for its MAC address,
and the desktop is answering immediately. But the board doesn't seem
to be getting the replies... Any ideas, or words of wisdom, as they say?

Regards.

On 06/12/2017 06:22 AM, Mason wrote:

Hello,

I am using the following drivers for Ethernet connectivity.
drivers/net/ethernet/aurora/nb8800.c
drivers/net/phy/at803x.c

Pulling the cable and plugging it back works as expected.
(I can ping both before and after.)

However, if I toggle the link state in software (using ip link set),
the board loses network connectivity.

# Statically assign IP address
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
# Set link state to "up"
ip link set eth0 up
# ping -c 3 172.27.64.1 > /tmp/v1

PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms

This delay seems abnormally long unless you are purposely introducing
delay (e.g: with cls_netem) or this is a really remote host, does not
seem to be based on your traces later on.

23 more quoted lines

- check the Ethernet MAC counters to see if there is packet loss, or
error, or both

- consult with your HW engineers for possible flaws in your
ndo_open/ndo_close paths and possible interactions with the MAC/PHY
clocks, or reset etc.

- see if your PHY needs a complete re-init after an up/down sequence and
if you are doing this properly

-- 
Florian

Hello Florian,

On 12/06/2017 18:38, Florian Fainelli wrote:

On 06/12/2017 06:22 AM, Mason wrote:

quoted

I am using the following drivers for Ethernet connectivity.
drivers/net/ethernet/aurora/nb8800.c
drivers/net/phy/at803x.c

Pulling the cable and plugging it back works as expected.
(I can ping both before and after.)

However, if I toggle the link state in software (using ip link set),
the board loses network connectivity.

# Statically assign IP address
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
# Set link state to "up"
ip link set eth0 up
# ping -c 3 172.27.64.1 > /tmp/v1

PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms

This delay seems abnormally long unless you are purposely introducing
delay (e.g: with cls_netem) or this is a really remote host, does not
seem to be based on your traces later on.

172.27.64.1 and 172.27.64.77 are connected to the
same switch. Purely local traffic. It seems to me
that the ARP request/reply could explain the delay.

Start op at 45.187346
Receive ICMP echo reply at 45.194662
Hmmm, that's only 7 ms

quoted

172.27.64.1 is a desktop system.
Running
% tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
on the desktop, I get:

15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64
15:01:50.213232 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 2, length 64
15:01:51.198563 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
15:01:51.209586 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46
15:01:51.209598 ARP, Reply 172.27.64.77 is-at 00:16:e8:4d:7f:c4, length 46

Packet #1: the board asks for the desktop's MAC address
Packet #2: the desktop replies instantly
Packet #3: the board sends the first ping
Packet #4: the desktop replies instantly
Then the board goes quiet for a long time (why???)
Packet #5: the desktop asks for the board's MAC address (doesn't it have it already?)
Packet #6: this seems to unwedge the board, which sends the second ping
Packet #7: the desktop replies instantly
Packet #8: the board sends the third ping
Packet #9: the desktop replies instantly
Packet #10: the desktop asks again for the board's MAC address
Packet #11 and #12: the board answers twice (for the old and new requests?)

Some oddities, but it seems to work.

Now toggle the link state:

% ip link set eth0 down
% ip link set eth0 up
% ping -c 3 172.27.64.1 > /tmp/v2

PING 172.27.64.1 (172.27.64.1): 56 data bytes

20 more quoted lines

I'll take a look, but I don't expect any packet loss
(LAN traffic on an idle switch).

- consult with your HW engineers for possible flaws in your
ndo_open/ndo_close paths and possible interactions with the MAC/PHY
clocks, or reset etc.

(The HW engineers have no knowledge of Linux use-cases.)
The crazy thing is that I can use the same driver on the
previous chip, and I don't see this behavior... Will
retest tomorrow to be sure. What does change between
the two chips are a few clock frequencies though.
So maybe some race is now consistently lost on the
new chip...

- see if your PHY needs a complete re-init after an up/down sequence and
if you are doing this properly

Thanks for these suggestions. I'll take a closer look
tomorrow.

Regards.

On 12/06/17 15:22, Mason wrote:

46 more quoted lines

Hi
You might want to read this thread:
https://www.mail-archive.com/netdev@vger.kernel.org/msg133896.html
The symptoms you describe are pretty much what we had at one point.

BR
Matthias

On 13/06/2017 11:39, Matthias May wrote:

On 12/06/17 15:22, Mason wrote:

quoted

Hello,

I am using the following drivers for Ethernet connectivity.
drivers/net/ethernet/aurora/nb8800.c
drivers/net/phy/at803x.c

Pulling the cable and plugging it back works as expected.
(I can ping both before and after.)

However, if I toggle the link state in software (using ip link set),
the board loses network connectivity.

# Statically assign IP address
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
# Set link state to "up"
ip link set eth0 up
# ping -c 3 172.27.64.1 > /tmp/v1

PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms

--- 172.27.64.1 ping statistics ---

3 packets transmitted, 1 packets received, 66% packet loss
round-trip min/avg/max = 18.321/18.321/18.321 ms


172.27.64.1 is a desktop system.
Running
% tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
on the desktop, I get:

15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64
15:01:50.213217 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 2, length 64

48 more quoted lines

Hello Matthias,

Yes, I remember discussing the issue with Zefir.
https://www.mail-archive.com/netdev@vger.kernel.org/msg147369.html

Do you think I should apply Zefir's patch?

In an attempt to remove the Atheros driver from the equation,
I used the generic PHY driver (I didn't build the Atheros
driver at all). But this had no impact on the issue.

Regards.

On 13/06/17 16:20, Mason wrote:

On 13/06/2017 11:39, Matthias May wrote:

quoted

On 12/06/17 15:22, Mason wrote:

quoted

Hello,

I am using the following drivers for Ethernet connectivity.
drivers/net/ethernet/aurora/nb8800.c
drivers/net/phy/at803x.c

Pulling the cable and plugging it back works as expected.
(I can ping both before and after.)

However, if I toggle the link state in software (using ip link set),
the board loses network connectivity.

# Statically assign IP address
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
# Set link state to "up"
ip link set eth0 up
# ping -c 3 172.27.64.1 > /tmp/v1

PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms

--- 172.27.64.1 ping statistics ---

3 packets transmitted, 1 packets received, 66% packet loss
round-trip min/avg/max = 18.321/18.321/18.321 ms


172.27.64.1 is a desktop system.
Running
% tcpdump -n -i eth1-boards ether host 00:16:e8:4d:7f:c4
on the desktop, I get:

15:01:45.187346 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:01:45.187359 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:01:45.194633 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 0, length 64
15:01:45.194662 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 0, length 64
15:01:50.198564 ARP, Request who-has 172.27.64.77 tell 172.27.64.1, length 28
15:01:50.205929 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41219, seq 1, length 64
15:01:50.205951 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41219, seq 1, length 64

63 more quoted lines

Isn't this patch already accepted?
I sent the link more as reference to try to check what we checked to detect the underlying issue.

On 12/06/2017 18:38, Florian Fainelli wrote:

On 06/12/2017 06:22 AM, Mason wrote:

quoted

I am using the following drivers for Ethernet connectivity.
drivers/net/ethernet/aurora/nb8800.c
drivers/net/phy/at803x.c

Pulling the cable and plugging it back works as expected.
(I can ping both before and after.)

However, if I toggle the link state in software (using ip link set),
the board loses network connectivity.

# Statically assign IP address
ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
# Set link state to "up"
ip link set eth0 up
# ping -c 3 172.27.64.1 > /tmp/v1

PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=18.321 ms

This delay seems abnormally long unless you are purposely introducing
delay (e.g: with cls_netem) or this is a really remote host, does not
seem to be based on your traces later on.

I think the delay is due to calling ping before the link
is actually up. For example, if I ping immediately after
setting the link up, the first 4 packets are lost.

PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=4 ttl=64 time=0.235 ms
64 bytes from 172.27.64.1: seq=5 ttl=64 time=0.142 ms
64 bytes from 172.27.64.1: seq=6 ttl=64 time=0.110 ms
64 bytes from 172.27.64.1: seq=7 ttl=64 time=0.095 ms
64 bytes from 172.27.64.1: seq=8 ttl=64 time=0.139 ms
64 bytes from 172.27.64.1: seq=9 ttl=64 time=0.120 ms

--- 172.27.64.1 ping statistics ---

10 packets transmitted, 6 packets received, 40% packet loss
round-trip min/avg/max = 0.095/0.140/0.235 ms

quoted

So basically, the board is asking the desktop for its MAC address,
and the desktop is answering immediately. But the board doesn't seem
to be getting the replies... Any ideas, or words of wisdom, as they say?

- check the Ethernet MAC counters to see if there is packet loss, or
error, or both

- consult with your HW engineers for possible flaws in your
ndo_open/ndo_close paths and possible interactions with the MAC/PHY
clocks, or reset etc.

- see if your PHY needs a complete re-init after an up/down sequence and
if you are doing this properly

I'm using the following test script:

ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
ip link set eth0 up
sleep 3  ## hopefully autoneg is complete

ethtool -S eth0 > /tmp/s0
ping -c 10 172.27.64.1 > /tmp/v1
ethtool -S eth0 > /tmp/s1
diff -U0 /tmp/s0 /tmp/s1

ip link set eth0 down
sleep 1
ip link set eth0 up
sleep 1

ethtool -S eth0 > /tmp/s0
ping -c 10 172.27.64.1 > /tmp/v2
ethtool -S eth0 > /tmp/s1
diff -U0 /tmp/s0 /tmp/s1

Testing with a generic PHY driver (no Atheros 8035 support built).
Apparently, ethtool doesn't report any packet loss or error.

First time:

# diff -U0 /tmp/s0 /tmp/s1

--- /tmp/s0
+++ /tmp/s1

@@ -2,2 +2,2 @@
-     rx_bytes_ok: 0
-     rx_frames_ok: 0
+     rx_bytes_ok: 1084
+     rx_frames_ok: 11

@@ -6,2 +6,2 @@
-     rx_64_byte_frames: 0
-     rx_127_byte_frames: 0
+     rx_64_byte_frames: 1
+     rx_127_byte_frames: 10

@@ -22,6 +22,6 @@
-     rx_bytes: 0
-     rx_frames: 0
-     tx_bytes_ok: 0
-     tx_frames_ok: 0
-     tx_64_byte_frames: 0
-     tx_127_byte_frames: 0
+     rx_bytes: 1084
+     rx_frames: 11
+     tx_bytes_ok: 1084
+     tx_frames_ok: 11
+     tx_64_byte_frames: 1
+     tx_127_byte_frames: 10

@@ -33 +33 @@
-     tx_broadcast_frames: 0
+     tx_broadcast_frames: 1

@@ -43,2 +43,2 @@
-     tx_bytes: 0
-     tx_frames: 0
+     tx_bytes: 1084
+     tx_frames: 11

Second time:

# diff -U0 /tmp/s0 /tmp/s1

--- /tmp/s0
+++ /tmp/s1

@@ -2,2 +2,2 @@
-     rx_bytes_ok: 1276
-     rx_frames_ok: 14
+     rx_bytes_ok: 1779
+     rx_frames_ok: 19

@@ -6 +6 @@
-     rx_64_byte_frames: 4
+     rx_64_byte_frames: 8

@@ -8 +8 @@
-     rx_255_byte_frames: 0
+     rx_255_byte_frames: 1

@@ -14 +14 @@
-     rx_broadcast_frames: 0
+     rx_broadcast_frames: 1

@@ -22,5 +22,5 @@
-     rx_bytes: 1276
-     rx_frames: 14
-     tx_bytes_ok: 1276
-     tx_frames_ok: 14
-     tx_64_byte_frames: 4
+     rx_bytes: 1779
+     rx_frames: 19
+     tx_bytes_ok: 1724
+     tx_frames_ok: 21
+     tx_64_byte_frames: 11

@@ -33 +33 @@
-     tx_broadcast_frames: 1
+     tx_broadcast_frames: 8

@@ -43,2 +43,2 @@
-     tx_bytes: 1276
-     tx_frames: 14
+     tx_bytes: 1724
+     tx_frames: 21

I did note something that seems important.

If I toggle the link state in software, then connectivity breaks.

If I unplug the ethernet cable, and replug, connectivity remains.

The difference is that plugging/unplugging doesn't call the
.ndo_stop callback. But 'ip link set eth0 down' does call it.

Should the .ndo_stop callback be symmetric to the .ndo_open callback?
In other words, should .ndo_open(); .ndo_stop(); be a NOP?

Regards.

On 13/06/2017 17:07, Mason wrote:

I did note something that seems important.

If I toggle the link state in software, then connectivity breaks.

If I unplug the ethernet cable, and replug, connectivity remains.

The difference is that plugging/unplugging doesn't call the
.ndo_stop callback. But 'ip link set eth0 down' does call it.

Should the .ndo_stop callback be symmetric to the .ndo_open callback?
In other words, should .ndo_open(); .ndo_stop(); be a NOP?

I changed the ndo_open callback to a wrapper that calls:

	nb8800_open(dev);
	nb8800_stop(dev);
	nb8800_open(dev);

With this change, connectivity is broken from the start.
Valid ARP requests are correctly *sent* but the corresponding
ARP replies are not received.

I'm hoping this limits the scope of what needs to be investigated.

Regards.

On 06/13/2017 08:07 AM, Mason wrote:> I did note something that seems
important.

If I toggle the link state in software, then connectivity breaks.

If I unplug the ethernet cable, and replug, connectivity remains.

What does that actually mean? If you disconnect the cable a link state
should be notified and another link state should be notified, even if
that happens faster than the PHYLIB polling time (1s).

The difference is that plugging/unplugging doesn't call the
.ndo_stop callback. But 'ip link set eth0 down' does call it.

Should the .ndo_stop callback be symmetric to the .ndo_open callback?

Yes obviously.

In other words, should .ndo_open(); .ndo_stop(); be a NOP?

-- 
Florian

On 13/06/2017 17:36, Florian Fainelli wrote:

On 06/13/2017 08:07 AM, Mason wrote:

quoted

I did note something that seems important.
If I toggle the link state in software, then connectivity breaks.
If I unplug the ethernet cable, and replug, connectivity remains.

What does that actually mean? If you disconnect the cable a link state
should be notified and another link state should be notified, even if
that happens faster than the PHYLIB polling time (1s).

Sorry for being unclear.

If I unplug/replug the cable, ping still works afterward.
(Packet RX appears to be *not* wedged)

If I toggle the link state in SW (set link down/set link up),
I can no longer ping afterward.
(Packet RX appears to be wedged, so ARP replies are not seen)

Maybe the two experiments are too unrelated to consider
the different results relevant in any way?

Regards.

On 06/13/2017 08:47 AM, Mason wrote:

On 13/06/2017 17:36, Florian Fainelli wrote:

quoted

On 06/13/2017 08:07 AM, Mason wrote:

quoted

I did note something that seems important.
If I toggle the link state in software, then connectivity breaks.
If I unplug the ethernet cable, and replug, connectivity remains.

What does that actually mean? If you disconnect the cable a link state
should be notified and another link state should be notified, even if
that happens faster than the PHYLIB polling time (1s).

Sorry for being unclear.

If I unplug/replug the cable, ping still works afterward.
(Packet RX appears to be *not* wedged)

If I toggle the link state in SW (set link down/set link up),
I can no longer ping afterward.
(Packet RX appears to be wedged, so ARP replies are not seen)

Maybe the two experiments are too unrelated to consider
the different results relevant in any way?

No, they are not, this really tells you that whatever your ndo_open()
and ndo_stop() functions do, they are currently broken with your
particular HW. I have seen numerous problems on some of our older
platforms using bcmgenet where the PHY needs to be reset *before* any
MAC activity occurs, and any MAC activity even qualifies as a reset of
the MAC itself, we can perform a PHY reset there because the PHY is
internal, that may not be an option, and your HW is different anyway.
-- 
Florian

On 13/06/2017 17:50, Florian Fainelli wrote:

On 06/13/2017 08:47 AM, Mason wrote:

quoted

If I unplug/replug the cable, ping still works afterward.
(Packet RX appears to be *not* wedged)

If I toggle the link state in SW (set link down/set link up),
I can no longer ping afterward.
(Packet RX appears to be wedged, so ARP replies are not seen)

Maybe the two experiments are too unrelated to consider
the different results relevant in any way?

No, they are not, this really tells you that whatever your ndo_open()
and ndo_stop() functions do, they are currently broken with your
particular HW. I have seen numerous problems on some of our older
platforms using bcmgenet where the PHY needs to be reset *before* any
MAC activity occurs, and any MAC activity even qualifies as a reset of
the MAC itself, we can perform a PHY reset there because the PHY is
internal, that may not be an option, and your HW is different anyway.

I have logged *all* register writes, to compare state after probe,
after open, and after close. I do see a few odd values...
I'm still deep in the maze, but I may find an exit eventually.

What state is probe supposed to leave the HW in?
What state is open  supposed to leave the HW in?

I'm thinking that I could reset the entire block in close.

Regards.


Found ethernet phy : AR8035
[    0.852746] libphy: Fixed MDIO Bus: probed
[    0.856975] ++ETH++ gw8  0xf0026424 0x00		# ETH_SW_RESET
[    0.870942] ++ETH++ gw8  0xf0026424 0x01		# ETH_SW_RESET
[    0.874894] ++ETH++ gw16 0xf0026420 0x0050		# ETH_MDIO_CLK_DIV
[    0.879101] libphy: nb8800-mii: probed
[    0.885005] ++ETH++ gw8  0xf0026000 0x1c		# ETH_MAC_CORE_TRANSMIT_CONTROL
[    0.888967] ++ETH++ gw8  0xf0026001 0x05		# ETH_MAC_CORE_TRANSMIT_CONTROL
[    0.892911] ++ETH++ gw8  0xf0026004 0x22		# ETH_MAC_CORE_RECEIVE_CONTROL
[    0.896855] ++ETH++ gw8  0xf0026008 0x04		# ETH_MAC_CORE_RANDOM_SEED
[    0.900815] ++ETH++ gw8  0xf0026014 0x0c		# ETH_MAC_CORE_TRANSMIT_SINGLE_DEFERRAL
[    0.904758] ++ETH++ gw8  0xf0026051 0x00		# ETH_MAC_CORE_THRESHOLDS
[    0.908700] ++ETH++ gw8  0xf0026052 0xff		# ETH_MAC_CORE_THRESHOLDS
[    0.912643] ++ETH++ gw8  0xf0026054 0x40		# ETH_MAC_CORE_BUFFER_SIZE_TX_AND_ETH_MAC_CORE_FIFO_CONTROL
[    0.916586] ++ETH++ gw32 0xf0026100 0x000002fe	# TRANSMIT_CHANNEL_CONTROL
[    0.921054] ++ETH++ gw32 0xf0026118 0x003cc4a4	# TRANSMIT_INTERRUPT_TIME
[    0.925521] ++ETH++ gw32 0xf0026200 0x000002fe	# RECEIVE_CHANNEL_CONTROL
[    0.929990] ++ETH++ gw32 0xf0026218 0x00004dc8	# RECEIVE_INTERRUPT_TIME
[    0.934456] ++ETH++ gw8  0xf0026060 0x00		# ETH_MAC_CORE_PAUSE_QUANTA
[    0.938398] ++ETH++ gw8  0xf0026061 0xc3		# ETH_MAC_CORE_PAUSE_QUANTA
[    0.942338] ++ETH++ gw8  0xf002602e 0x00		# ETH_MAC_CORE_MULTICAST_ADDRESS1
[    0.946283] ++ETH++ gw8  0xf0026400 0x01		# ETH_GPIO_PAD_MODE
[    0.950226] ++ETH++ gw32 0xf0026200 0x0000028e	# RECEIVE_CHANNEL_CONTROL
[    0.954695] ++ETH++ gw8  0xf002606a 0x00		# ETH_MAC_CORE_SOURCE_ADDRESS0
[    0.958638] ++ETH++ gw8  0xf002606b 0x16		# ETH_MAC_CORE_SOURCE_ADDRESS0
[    0.962581] ++ETH++ gw8  0xf002606c 0xe8		# ETH_MAC_CORE_SOURCE_ADDRESS1
[    0.966523] ++ETH++ gw8  0xf002606d 0x4d		# ETH_MAC_CORE_SOURCE_ADDRESS1

87 more lines

Mason [off-list ref] writes:

[   33.393332] ENTER nb8800_stop
[   33.421731] nb8800_mdio_write: reg=9 val=0200	# by genphy_config_aneg()
[   33.444062] nb8800_mdio_write: reg=0 val=1200	# by genphy_restart_aneg()
[   33.466411] nb8800_mdio_write: reg=18 val=0000	# by at803x_config_intr() from phy_stop()

Those are from the phy_stop() call.

The following is the ridiculous dance required to stop rx dma in
nb8800_dma_stop():

[   33.488859] ++ETH++ gw8  0xf0026004 0x03		# ETH_MAC_CORE_RECEIVE_CONTROL

Disable address filter.

[   33.492805] ++ETH++ gw8  0xf0026044 0x04		# ETH_MAC_CORE_MAC_MODE  *WHY ??*

Enable loopback.

[   33.496748] ++ETH++ gw32 0xf002610c 0x9dcac000	\
[   33.501217] ++ETH++ gw32 0xf0026100 0x000002ff	|

Send frame.

[   33.505750] ++ETH++ gw32 0xf0026104 0x00000004	| IRQ?
[   33.510213] ++ETH++ gw32 0xf0026204 0x00000004	| 
[   33.514674] ++ETH++ gw32 0xf0026218 0x003cc4a4	|
[   33.519146] ++ETH++ gw32 0xf0026104 0x00000001	/

Handle resulting interrupt(s).

[   33.605719] ++ETH++ gw32 0xf002610c 0x9dcac000	\
[   33.610187] ++ETH++ gw32 0xf0026100 0x000002ff	|

Send frame.

[   33.614717] ++ETH++ gw32 0xf0026104 0x00000004	| ANOTHER IRQ?
[   33.619178] ++ETH++ gw32 0xf0026204 0x00000004	|
[   33.623637] ++ETH++ gw32 0xf0026218 0x003cc4a4	|
[   33.628106] ++ETH++ gw32 0xf0026104 0x00000001	/

Handle interrupt(s).

[   33.714689] ++ETH++ gw32 0xf002610c 0x9dcac000	|
[   33.719156] ++ETH++ gw32 0xf0026100 0x000002ff	| ANOTHER IRQ?

Send frame.

[   33.723686] ++ETH++ gw32 0xf0026104 0x00000004	|

Handle tx interrupt.

[   33.724625] ++ETH++ gw8  0xf0026004 0x23		# ETH_MAC_CORE_RECEIVE_CONTROL

Enable address filter.

[   33.724627] ++ETH++ gw8  0xf0026044 0x00		# ETH_MAC_CORE_MAC_MODE *AGAIN ??*

Disable loopback.

[   33.724640] ++ETH++ gw32 0xf002620c 0x9de36000
From nb8800_dma_reset().

[   33.724642] ++ETH++ gw8  0xf0026004 0x22		# ETH_MAC_CORE_RECEIVE_CONTROL AGAIN

Disable rx.

[   33.724644] ++ETH++ gw8  0xf0026000 0x1c		# ETH_MAC_CORE_TRANSMIT_CONTROL

Disable tx.

[   33.724690] nb8800_mdio_write: reg=18 val=0000	# by at803x_config_intr() from phy_disable_interrupts()
[   33.724913] nb8800_mdio_write: reg=0 val=1800	# by at803x_suspend() from phy_suspend()

[   33.819562] ++ETH++ gw32 0xf0026204 0x00000004
[   33.824021] ++ETH++ gw32 0xf0026218 0x003cc4a4

Handle rx interrupt, probably from the last loopback frame sent earlier.

[   33.828518] EXIT  nb8800_stop

Now I did notice one thing.  When the interrupts from the loopback
frames are handled, the rx interrupt is all but disabled for NAPI poll
mode.  Of course, NAPI isn't active, so the rx interrupt never gets
re-enabled.  We should probably do this in ndo_open just to be sure.

Try this patch:

diff --git a/drivers/net/ethernet/aurora/nb8800.c b/drivers/net/ethernet/aurora/nb8800.c
index e078d8da978c..f968468076b9 100644
--- a/drivers/net/ethernet/aurora/nb8800.c
+++ b/drivers/net/ethernet/aurora/nb8800.c

@@ -969,6 +969,7 @@ static int nb8800_open(struct net_device *dev)
 	if (err)
 		goto err_free_dma;
 
+	nb8800_writel(priv, NB8800_RX_ITR, priv->rx_itr_irq);
 	nb8800_mac_rx(dev, true);
 	nb8800_mac_tx(dev, true);
 

-- 
Måns Rullgård

On 15/06/2017 12:19, Måns Rullgård wrote:

Now I did notice one thing.  When the interrupts from the loopback
frames are handled, the rx interrupt is all but disabled for NAPI poll
mode.  Of course, NAPI isn't active, so the rx interrupt never gets
re-enabled.  We should probably do this in ndo_open just to be sure.

I have tried everything I could think of (whether reasonable,
or just random). Setting the link down always hoses the board's
network connectivity, 100% reproducible on Vantage 1218.

The single HW difference that I'm aware of is that the Atheros
PHY used to be clocked at 25 MHz (I think) on 1172 and 1191
boards, but it is clocked at 100 MHz on this 1218 board. 
I'll try tweaking this setting.

For the time being, my test script is:

ip addr add 172.27.64.77/18 brd 172.27.127.255 dev eth0
ip link set eth0 up
sleep 6  ## autoneg should be complete
ntpdate 172.27.64.1
sleep 1
date +%T && ping -c 3 172.27.64.1 > /tmp/ping_before
sleep 1
ip link set eth0 down
sleep 2
ip link set eth0 up
sleep 5
date +%T && ping -c 3 172.27.64.1 > /tmp/ping_after

# cat /tmp/ping_before

PING 172.27.64.1 (172.27.64.1): 56 data bytes
64 bytes from 172.27.64.1: seq=0 ttl=64 time=31.442 ms
64 bytes from 172.27.64.1: seq=1 ttl=64 time=31.359 ms
64 bytes from 172.27.64.1: seq=2 ttl=64 time=31.379 ms

--- 172.27.64.1 ping statistics ---

3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 31.359/31.393/31.442 ms

=> These RTT numbers are crazy high for a LAN connection
through a switch. (I get 0.250 ms from my desktop.)

=> What the server saw was:
15:23:30.400066 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41987, seq 0, length 64
15:23:30.400094 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41987, seq 0, length 64
15:23:31.431479 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41987, seq 1, length 64
15:23:31.431500 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41987, seq 1, length 64
15:23:32.462844 IP 172.27.64.77 > 172.27.64.1: ICMP echo request, id 41987, seq 2, length 64
15:23:32.462867 IP 172.27.64.1 > 172.27.64.77: ICMP echo reply, id 41987, seq 2, length 64


# cat /tmp/ping_after  
PING 172.27.64.1 (172.27.64.1): 56 data bytes

--- 172.27.64.1 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

=> What the server saw was:
15:23:41.180068 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:23:41.180081 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:23:42.194409 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:23:42.194419 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28
15:23:43.207700 ARP, Request who-has 172.27.64.1 tell 172.27.64.77, length 46
15:23:43.207710 ARP, Reply 172.27.64.1 is-at 00:15:17:24:e0:81, length 28



Below are the "raw" logs (of what I thought might be useful)

Regards.


[    0.898538] libphy: Fixed MDIO Bus: probed
[    0.902734] ENTER nb8800_probe
[    0.905863] ++ETH++ gw8  0xf0026424 0x00
[    0.919601] ++ETH++ gw8  0xf0026424 0x01
[    0.923550] ++ETH++ gw16 0xf0026420 0x0050
[    0.927749] libphy: nb8800-mii: probed
[    0.933736] ENTER nb8800_hw_init
[    0.936991] ++ETH++ gw8  0xf0026000 0x1c
[    0.940952] ++ETH++ gw8  0xf0026001 0x05
[    0.944897] ++ETH++ gw8  0xf0026004 0x22
[    0.948839] ++ETH++ gw8  0xf0026008 0x04
[    0.952792] ++ETH++ gw8  0xf0026014 0x0c
[    0.956740] ++ETH++ gw8  0xf0026051 0x00
[    0.960693] ++ETH++ gw8  0xf0026052 0xff
[    0.964637] ++ETH++ gw8  0xf0026054 0x40
[    0.968579] ++ETH++ gw32 0xf0026100 0x000002fe
[    0.973058] ++ETH++ gw32 0xf0026118 0x003cc4a4
[    0.977527] ++ETH++ gw32 0xf0026200 0x000002fe
[    0.981994] ++ETH++ gw32 0xf0026218 0x00004dc8
[    0.986473] ++ETH++ gw8  0xf002602e 0x00
[    0.990417] EXIT nb8800_hw_init
[    0.993576] ++ETH++ gw8  0xf0026400 0x01
[    0.997518] ++ETH++ gw32 0xf0026200 0x0000028e

230 more lines