bonding with 3c59x driver

Hi Steffen, all,

I am trying to do network bonding on top of 3com 3C905C ethernet 
adapters. I am facing the problem that network cable removal isn't 
detected. It is detected when using Realtek hardware with driver 8139too 
or VIA hardware with driver via-rhine, but no luck with driver 3c59x. 
I'm using bonding option miimon=100 for link detection.

Does anyone know if this is a hardware limitation, or a missing feature 
of the 3c59x driver, or a bug in that driver?

Thanks,
-- 
Jean Delvare
Suse L3

Jean Delvare [off-list ref] wrote:

I am trying to do network bonding on top of 3com 3C905C ethernet 
adapters. I am facing the problem that network cable removal isn't 
detected. It is detected when using Realtek hardware with driver 8139too 
or VIA hardware with driver via-rhine, but no luck with driver 3c59x. 
I'm using bonding option miimon=100 for link detection.

Does anyone know if this is a hardware limitation, or a missing feature 
of the 3c59x driver, or a bug in that driver?

	It just so happens that I was looking at 3c59x earlier today for
something else.  It appears to run its internal link state detection on
a 5 second timer, after which (in theory) bonding's miimon would notice
(provided that the driver notices and calls netif_carrier_off).

	If that's not working, did you try use_carrier=0 to probe the
mii register state directly?

	I have used 3c59x with bonding in the distant past, but I don't
recall exactly what tweaks were necessary; it may have required
use_carrier=0 to work at all.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Hi Jay,

Thanks for the fast answer.

On Monday 13 February 2012 11:13:11 pm Jay Vosburgh wrote:

Jean Delvare [off-list ref] wrote:

quoted

I am trying to do network bonding on top of 3com 3C905C ethernet
adapters. I am facing the problem that network cable removal isn't
detected. It is detected when using Realtek hardware with driver
8139too or VIA hardware with driver via-rhine, but no luck with
driver 3c59x. I'm using bonding option miimon=100 for link
detection.

Does anyone know if this is a hardware limitation, or a missing
feature of the 3c59x driver, or a bug in that driver?

	It just so happens that I was looking at 3c59x earlier today for
something else.  It appears to run its internal link state detection
 on a 5 second timer, after which (in theory) bonding's miimon would
 notice (provided that the driver notices and calls
 netif_carrier_off).

Thanks for the pointer. There is indeed a 5 second timer to detect 
transitions from link down to link up. However the timer period to 
detect transitions from link up to link down is 1 minute. Knowing that, 
I tested again and indeed the bonding module end up detecting the link 
going down, after up to 1 minute. That's not exactly convenient for 
bonding.

I admit I don't quite get the rationale for having different timer 
periods for detecting the link going up or down. Steffen, this was done 
by you in:

commit b4ff6450f5336c492d1e2f184d3b8186e0716b7a
Author: Steffen Klassert [off-list ref]
Date:   Sun Mar 26 01:37:40 2006 -0800

    [PATCH] 3c59x: decrease polling interval

No rationale explained in the commit message. Can you explain? I find 
the asymmetry confusing, I'd rather use 5 seconds (if not less) for both 
cases.

	If that's not working, did you try use_carrier=0 to probe the
mii register state directly?

I tried use_carrier=0 but it makes no difference.

	I have used 3c59x with bonding in the distant past, but I don't
recall exactly what tweaks were necessary; it may have required
use_carrier=0 to work at all.

Well I am still curious why the 3c59x driver has such slow polling when 
other drivers I have been testing are able to report almost instantly 
when I remove a network cable. Could it be that other network chips 
generate an interrupt on cable removal and the 3com chips do not?

-- 
Jean Delvare
Suse L3

On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:

Hi Jay,

Thanks for the fast answer.

On Monday 13 February 2012 11:13:11 pm Jay Vosburgh wrote:

quoted

Jean Delvare [off-list ref] wrote:

quoted

I am trying to do network bonding on top of 3com 3C905C ethernet
adapters. I am facing the problem that network cable removal isn't
detected. It is detected when using Realtek hardware with driver
8139too or VIA hardware with driver via-rhine, but no luck with
driver 3c59x. I'm using bonding option miimon=100 for link
detection.

Does anyone know if this is a hardware limitation, or a missing
feature of the 3c59x driver, or a bug in that driver?

	It just so happens that I was looking at 3c59x earlier today for
something else.  It appears to run its internal link state detection
 on a 5 second timer, after which (in theory) bonding's miimon would
 notice (provided that the driver notices and calls
 netif_carrier_off).

Thanks for the pointer. There is indeed a 5 second timer to detect 
transitions from link down to link up. However the timer period to 
detect transitions from link up to link down is 1 minute. Knowing that, 
I tested again and indeed the bonding module end up detecting the link 
going down, after up to 1 minute. That's not exactly convenient for 
bonding.

I admit I don't quite get the rationale for having different timer 
periods for detecting the link going up or down. Steffen, this was done 
by you in:

commit b4ff6450f5336c492d1e2f184d3b8186e0716b7a
Author: Steffen Klassert [off-list ref]
Date:   Sun Mar 26 01:37:40 2006 -0800

    [PATCH] 3c59x: decrease polling interval

No rationale explained in the commit message. Can you explain? I find 
the asymmetry confusing, I'd rather use 5 seconds (if not less) for both 
cases.

This patch is based on a discussion with Andrew Morton some years ago.
The 3c59x does polling for external environment changes which is quite
expensive. Firing a timer that disables the interrupts on a running
interface every 5 seconds is just not reasonable for checking for external
environment changes. So we decided to let it depend on the link status,
5 seconds if the link is down and 60 seconds if the link is up.

Well I am still curious why the 3c59x driver has such slow 
polling when 
other drivers I have been testing are able to report almost instantly 
when I remove a network cable. Could it be that other network chips 
generate an interrupt on cable removal and the 3com chips do not?

Not sure about those specific cards, but detecting the
link state might require software bit-banging of an MII
(or similar) interface.

Since this needs delays between all the edges it can
end up spinning the cpu for a noticable time, especially
if the resolution (and minimim delay) of the delay()
function used is significantly longer than the delays
actually required - which might only be a few 100ns.

Using a cpu spin for the delays is also slightly
problematical because the writes might get 'posted'
and delayed further - so the writes have to be forced
out to the device itself (eg by a readback) prior to
the delay() being requested.

If the driver is polling the link status it might
be valid to do one bit-bang transition on each timer
interrupt. This would separate the edges without
spinning the cpu - provided there is a fast method
for being called on every timer tick.

Another scheme I have used is to write each value
to the bit-bang register multiple times - using
the maximum speed of the IO writes to generate the
required delays.

	David

On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:

Well I am still curious why the 3c59x driver has such slow polling when 
other drivers I have been testing are able to report almost instantly 
when I remove a network cable. Could it be that other network chips 
generate an interrupt on cable removal and the 3com chips do not?

Yes, at least some of the supported chips do not generate an interrupt
on cable removal, so we have to check for this with a timer.

Le mardi 14 février 2012 à 12:06 +0100, Steffen Klassert a écrit :

On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:

quoted

Well I am still curious why the 3c59x driver has such slow polling when 
other drivers I have been testing are able to report almost instantly 
when I remove a network cable. Could it be that other network chips 
generate an interrupt on cable removal and the 3com chips do not?

Yes, at least some of the supported chips do not generate an interrupt
on cable removal, so we have to check for this with a timer.
--

We could have a 5 sec timer in case device is a slave.

diff --git a/drivers/net/ethernet/3com/3c59x.c b/drivers/net/ethernet/3com/3c59x.c
index 1282f04..e463d10 100644
--- a/drivers/net/ethernet/3com/3c59x.c
+++ b/drivers/net/ethernet/3com/3c59x.c

@@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
 		ok = 1;
 	}
 
-	if (!netif_carrier_ok(dev))
+	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
 		next_tick = 5*HZ;
 
 	if (vp->medialock)

Hi Eric, Steffen,

On Tuesday 14 February 2012 12:13:37 pm Eric Dumazet wrote:

Le mardi 14 février 2012 à 12:06 +0100, Steffen Klassert a écrit :

quoted

On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:

quoted

Well I am still curious why the 3c59x driver has such slow
polling when other drivers I have been testing are able to report
almost instantly when I remove a network cable. Could it be that
other network chips generate an interrupt on cable removal and
the 3com chips do not?

Yes, at least some of the supported chips do not generate an
interrupt on cable removal, so we have to check for this with a
timer.

Too bad. I confirm I can't see the interrupt count increase when I 
remove the network cable from my 3C905C (nor when I plug it back in), 
but I was hopping this was a matter of enabling this event as an 
interrupt source in the driver. If the hardware just can't do it then 
this is hopeless. Back in the days I used to consider the 3C905 as the 
best fast Ethernet cards around, apparently I was horribly wrong.

Thanks for explaining anyway.

Yes I think this makes a lot of sense. If you send this upstream you can 
add:

Acked-by: Jean Delvare <redacted>

-- 
Jean Delvare
Suse L3

On Tue, Feb 14, 2012 at 01:50:57PM +0100, Jean Delvare wrote:

quoted

We could have a 5 sec timer in case device is a slave.

diff --git a/drivers/net/ethernet/3com/3c59x.c
 b/drivers/net/ethernet/3com/3c59x.c index 1282f04..e463d10 100644
--- a/drivers/net/ethernet/3com/3c59x.c
+++ b/drivers/net/ethernet/3com/3c59x.c

@@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
 		ok = 1;
 	}

-	if (!netif_carrier_ok(dev))
+	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
 		next_tick = 5*HZ;

 	if (vp->medialock)

Yes I think this makes a lot of sense. If you send this upstream you can 
add:

Acked-by: Jean Delvare <redacted>

I'm fine with this too.

On 02/14/2012 03:13 AM, Eric Dumazet wrote:

Does the device being a slave in a bond change the overheads of the 
check being performed?

rick jones

On Tuesday 14 February 2012 07:27:05 pm Rick Jones wrote:

On 02/14/2012 03:13 AM, Eric Dumazet wrote:

quoted

Le mardi 14 février 2012 à 12:06 +0100, Steffen Klassert a écrit :

quoted

On Tue, Feb 14, 2012 at 11:22:10AM +0100, Jean Delvare wrote:

quoted

Well I am still curious why the 3c59x driver has such slow
polling when other drivers I have been testing are able to report
almost instantly when I remove a network cable. Could it be that
other network chips generate an interrupt on cable removal and
the 3com chips do not?

Yes, at least some of the supported chips do not generate an
interrupt on cable removal, so we have to check for this with a
timer. --

We could have a 5 sec timer in case device is a slave.

diff --git a/drivers/net/ethernet/3com/3c59x.c

b/drivers/net/ethernet/3com/3c59x.c index 1282f04..e463d10 100644

--- a/drivers/net/ethernet/3com/3c59x.c
+++ b/drivers/net/ethernet/3com/3c59x.c

@@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
  		ok = 1;
  	}

-	if (!netif_carrier_ok(dev))
+	if (dev->flags&  IFF_SLAVE || !netif_carrier_ok(dev))
  		next_tick = 5*HZ;

  	if (vp->medialock)

Does the device being a slave in a bond change the overheads of the
check being performed?

No, but it changes the interest in having faster polling. The overhead 
is the same but the interest/overhead trade-off is much higher.

Without the patch above, doing bonding on top of 3c59x is simply not 
possible, at least not in active-backup mode. For other modes it depends 
if you are only interested in load balancing or also on redundancy.

-- 
Jean Delvare
Suse L3

From: Steffen Klassert <redacted>
Date: Tue, 14 Feb 2012 14:34:18 +0100

On Tue, Feb 14, 2012 at 01:50:57PM +0100, Jean Delvare wrote:

quoted

quoted

We could have a 5 sec timer in case device is a slave.

diff --git a/drivers/net/ethernet/3com/3c59x.c
 b/drivers/net/ethernet/3com/3c59x.c index 1282f04..e463d10 100644
--- a/drivers/net/ethernet/3com/3c59x.c
+++ b/drivers/net/ethernet/3com/3c59x.c

@@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
 		ok = 1;
 	}

-	if (!netif_carrier_ok(dev))
+	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
 		next_tick = 5*HZ;

 	if (vp->medialock)

Yes I think this makes a lot of sense. If you send this upstream you can 
add:

Acked-by: Jean Delvare <redacted>

I'm fine with this too.

Eric, please submit this formally.

Jean Delvare reported bonding on top of 3c59x adapters was not detecting
network cable removal fast enough.

3c59x indeed uses a 60 seconds timer to check link status if carrier is
on, and 5 seconds if carrier is off.

This patch reduces timer period to 5 seconds if device is a bonding
slave.

Reported-by: Jean Delvare <redacted>
Acked-by: Jean Delvare <redacted>
Acked-by: Steffen Klassert <steffen.klassert@secunet.com>
Signed-off-by: Eric Dumazet <redacted>
---
 drivers/net/ethernet/3com/3c59x.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/3com/3c59x.c b/drivers/net/ethernet/3com/3c59x.c
index 1282f04..e463d10 100644
--- a/drivers/net/ethernet/3com/3c59x.c
+++ b/drivers/net/ethernet/3com/3c59x.c

@@ -1841,7 +1841,7 @@ vortex_timer(unsigned long data)
 		ok = 1;
 	}
 
-	if (!netif_carrier_ok(dev))
+	if (dev->flags & IFF_SLAVE || !netif_carrier_ok(dev))
 		next_tick = 5*HZ;
 
 	if (vp->medialock)

On Tue, 2012-02-14 at 21:27 +0100, Eric Dumazet wrote:

Jean Delvare reported bonding on top of 3c59x adapters was not detecting
network cable removal fast enough.

3c59x indeed uses a 60 seconds timer to check link status if carrier is
on, and 5 seconds if carrier is off.

This patch reduces timer period to 5 seconds if device is a bonding
slave.

Maybe for posterity give some rationale as to why we feel we can reduce
it to 5 seconds for slaves instead of reducing the timer period in
general?  If you weren't party to this discussion that won't be apparent
from the commit log.

Dan

Le mardi 14 février 2012 à 14:51 -0600, Dan Williams a écrit :

On Tue, 2012-02-14 at 21:27 +0100, Eric Dumazet wrote:

quoted

Jean Delvare reported bonding on top of 3c59x adapters was not detecting
network cable removal fast enough.

3c59x indeed uses a 60 seconds timer to check link status if carrier is
on, and 5 seconds if carrier is off.

This patch reduces timer period to 5 seconds if device is a bonding
slave.

Maybe for posterity give some rationale as to why we feel we can reduce
it to 5 seconds for slaves instead of reducing the timer period in
general?  If you weren't party to this discussion that won't be apparent
from the commit log.

Apparently, Andrew Morton considered firing a timer every 5 seconds was
too expensive. I am not sure we want to state this (probably old fact)
in the changelog. Anyway are these slow NICS still used in 2011 ? ;)

On 02/14/2012 01:27 PM, Jean Delvare wrote:

Without the patch above, doing bonding on top of 3c59x is simply not
possible, at least not in active-backup mode. For other modes it depends
if you are only interested in load balancing or also on redundancy.

Could you just use arp monitoring instead of mii?

Chris

-- 
Chris Friesen
Software Developer
GENBAND
chris.friesen@genband.com
www.genband.com

From: Eric Dumazet <redacted>
Date: Tue, 14 Feb 2012 22:02:56 +0100

Le mardi 14 février 2012 à 14:51 -0600, Dan Williams a écrit :

quoted

On Tue, 2012-02-14 at 21:27 +0100, Eric Dumazet wrote:

quoted

Jean Delvare reported bonding on top of 3c59x adapters was not detecting
network cable removal fast enough.

3c59x indeed uses a 60 seconds timer to check link status if carrier is
on, and 5 seconds if carrier is off.

This patch reduces timer period to 5 seconds if device is a bonding
slave.

Maybe for posterity give some rationale as to why we feel we can reduce
it to 5 seconds for slaves instead of reducing the timer period in
general?  If you weren't party to this discussion that won't be apparent
from the commit log.

Apparently, Andrew Morton considered firing a timer every 5 seconds was
too expensive. I am not sure we want to state this (probably old fact)
in the changelog. Anyway are these slow NICS still used in 2011 ? ;)

At least Jean Delvare is :-)

I'm going to apply this as-is, if you configure bonding on one of these
chips you expect it to work.  And working is more important than optimal.

On Tuesday 14 February 2012 10:06:54 pm Chris Friesen wrote:

On 02/14/2012 01:27 PM, Jean Delvare wrote:

quoted

Without the patch above, doing bonding on top of 3c59x is simply
not possible, at least not in active-backup mode. For other modes
it depends if you are only interested in load balancing or also on
redundancy.

Could you just use arp monitoring instead of mii?

If it was just for me, I could, and probably would. But when I am trying 
to reproduce a customer bug, I have to use exactly the same options the 
customer is using.

-- 
Jean Delvare
Suse L3