From: David Miller <davem@davemloft.net> Date: 2009-08-18 01:03:13
From: Tim Bird <redacted>
Date: Mon, 17 Aug 2009 15:35:01 -0700
Tim Bird wrote:
quoted
See the definitions of CONF_PRE_OPEN and CON_POST_OPEN
in net/ipv4/ipconfig.c
They are set to ridiculously long values. In my experience,
you can cut them down considerably with no dangerous side
effects (but I haven't asked the network guys about the
possible downsides).
It turns out that others have seen this delay. Simon
Arlott recently posted a patch to make the delay avoidable
at boot time from the kernel command line.
See http://patchwork.kernel.org/patch/31678/
"Rediculiously long" is a relative term.
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
So what's there now is actually a quite agressive setting.
And BTW, discussions about stuff like this belong on
netdev@vger.kernel.org, which has been added to the CC:
From: Tim Bird <redacted>
Date: Mon, 17 Aug 2009 15:35:01 -0700
quoted
Tim Bird wrote:
quoted
See the definitions of CONF_PRE_OPEN and CON_POST_OPEN
in net/ipv4/ipconfig.c
They are set to ridiculously long values. In my experience,
you can cut them down considerably with no dangerous side
effects (but I haven't asked the network guys about the
possible downsides).
It turns out that others have seen this delay. Simon
Arlott recently posted a patch to make the delay avoidable
at boot time from the kernel command line.
See http://patchwork.kernel.org/patch/31678/
"Rediculiously long" is a relative term.
No offense intended. I could have phrased this
better. The delays were a few orders of
magnitude longer than apparently needed, on my
embedded test systems with ethernet. I didn't
try eliminating them completely, as in the Arlott patch.
1.5 seconds is a long time for me. My bootup time budget for
the kernel ranges from 0.5 to 3.0 seconds, depending on the
product.
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
What types of delays are these timeouts supposed to
cover? Networking delays or hardware bring-up delays?
(Or both)? If for networking delays, is this for all
types of networks, or just some (e.g. ones that create
virtual circuits)?
I'm trying to get a sense for whether the card/switch
combinations that would take this long would be encountered
in the types of embedded devices I code for. (TVs, camcorders,
etc.)
So what's there now is actually a quite agressive setting.
And BTW, discussions about stuff like this belong on
netdev@vger.kernel.org, which has been added to the CC:
I was going to wait to see if this solved Robert's
problem, before widening the discussion. But I'm happy
to find out more about these delays now.
Thanks,
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
From: David Miller <davem@davemloft.net> Date: 2009-08-18 01:27:44
From: Tim Bird <redacted>
Date: Mon, 17 Aug 2009 18:24:26 -0700
David Miller wrote:
quoted
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
What types of delays are these timeouts supposed to
cover?
The problem is that if you don't first give at least some time for the
link to come up, the remaining time it takes the link to come up will
end up chewing into the actual bootp/dhcp protocol timeouts. And
that's what we're trying to avoid.
From: Tim Bird <redacted>
Date: Mon, 17 Aug 2009 18:24:26 -0700
quoted
David Miller wrote:
quoted
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
What types of delays are these timeouts supposed to
cover?
The problem is that if you don't first give at least some time for the
link to come up, the remaining time it takes the link to come up will
end up chewing into the actual bootp/dhcp protocol timeouts. And
that's what we're trying to avoid.
What link? I'm not that familiar with networking.
Assuming I'm using ethernet, what link needs to come up?
Is this something to do with power propagation to the
physical wire? Is there some MAC layer negotiation
between the card and the switch? Is it the time for
the switch to do speed detection?
And, can any of this be more accurately determined
or guessed-at with knowledge of the onboard hardware?
Or is it dependent on external conditions?
Where would be a good place to find out more about
startup delays for networking chips and/or protocols?
Our usual solution is to kick the can down the road
and let user-space initialize anything that takes a long
time, while we do other stuff like focus the camera or display
the TV picture. It would be good to learn more about
this.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================
From: David Miller <davem@davemloft.net> Date: 2009-08-18 01:56:05
From: Tim Bird <redacted>
Date: Mon, 17 Aug 2009 18:40:48 -0700
David Miller wrote:
quoted
From: Tim Bird <redacted>
Date: Mon, 17 Aug 2009 18:24:26 -0700
quoted
David Miller wrote:
quoted
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
What types of delays are these timeouts supposed to
cover?
The problem is that if you don't first give at least some time for the
link to come up, the remaining time it takes the link to come up will
end up chewing into the actual bootp/dhcp protocol timeouts. And
that's what we're trying to avoid.
What link? I'm not that familiar with networking.
The speed and duplex settings which are negotiated or forced between
the ethernet card and whatever is at the other end of the cable.
Assuming I'm using ethernet, what link needs to come up?
All modern ethernet cards do autonegotiation of link parameters
with whatever is at the other end of the ethernet cable. Cards
created ages ago which only support 10MB half-duplex typically
do not support autonegotiation at all.
This autonegotiation works like a protocol where the two link partners
go back and forth trying to figure out the best speed and duplex
settings to use. There are advertisements of link capabilities and
stuff like that.
It should happen almost instantaneously, but there are millions
upon millions of cruddy parts out there, and some of them take
a long time to go through this negotiation process.
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
Gotta love it when things adhere to specs...
the default on Cisco switches is to wait 30 seconds before fully enabling
the port so that it can listen for spanning tree broadcasts.
for windows systems this doesn't cause any problems (they take long enough
to boot), but for a tightly configured linux box it can be fully booted
long before the switch decides to enable the port (for that matter, I have
a bare-metal install process that takes about 90 seconds from hitting
power, from a CD. I could probably speed things up by switching to USB
boot media)
David Lang
On Tuesday 18 August 2009 03:27, David Miller wrote:
From: Tim Bird <redacted>
Date: Mon, 17 Aug 2009 18:24:26 -0700
quoted
David Miller wrote:
quoted
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
What types of delays are these timeouts supposed to
cover?
The problem is that if you don't first give at least some time for the
link to come up, the remaining time it takes the link to come up will
end up chewing into the actual bootp/dhcp protocol timeouts. And
that's what we're trying to avoid.
But in this case, they assign a static IP. They do not use DHCP.
So they pay this time penalty even if they wouldn't use networking
until sometime later (or never).
Since DHCP and any other networking activity like TCP connects
accomodate packet loss, things should work even without any delay
in kernel IP config code. The delay will be just shifted to the
moment when first DHCP/TCP/whatever negotiation happens.
If dropping delays altogether sounds too big a change,
then it makes sense at least to allow people to tweak it with
ipdelay=TIME_IN_MS
--
vda
On Mon, Aug 17, 2009 at 07:45:33PM -0700, david@lang.hm wrote:
On Mon, 17 Aug 2009, Rick Jones wrote:
quoted
David Miller wrote:
quoted
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
Gotta love it when things adhere to specs...
the default on Cisco switches is to wait 30 seconds before fully enabling
the port so that it can listen for spanning tree broadcasts.
And this causes a lot of trouble in high availability environments,
because the link is up but unusable. So if you're using it as a primary
bond link you can lose connectivity for that time. Fortunately, you can
configure the port in "switchport mode access", "portfast" mode to avoid
this annoying delay.
Willy
Since DHCP and any other networking activity like TCP connects
accomodate packet loss, things should work even without any delay
in kernel IP config code. The delay will be just shifted to the
moment when first DHCP/TCP/whatever negotiation happens.
Until the link is up, the packet scheduler just holds onto packets
and queued them up.
When the link comes up, this queue is released and the packets
sent out.
That's why it's beneficial to wait some time until the link
comes up before we start sending stuff out. Because otherwise
any timeouts used will be inaccurate.
This could code be spiffed up to wait for the link up indication
on devices it cares about. Feel free to code that up :-)
From: Tim Bird <redacted>
Date: Mon, 17 Aug 2009 18:24:26 -0700
quoted
David Miller wrote:
quoted
I have card/switch combinations that take up to 10 seconds to
negotiate a proper link.
What types of delays are these timeouts supposed to
cover?
The problem is that if you don't first give at least some time for the
link to come up, the remaining time it takes the link to come up will
end up chewing into the actual bootp/dhcp protocol timeouts. And
that's what we're trying to avoid.
What link? I'm not that familiar with networking.
Assuming I'm using ethernet, what link needs to come up?
When you plug an ethernet cable in, you may have noticed it takes a
short time before the signal light comes on. That's negotiation time.
Some are slower than others, but none of them do it instantly.
Is this something to do with power propagation to the
physical wire?
Not really.
Is there some MAC layer negotiation between the card and the switch?
Is it the time for the switch to do speed detection?
Yes and yes.
And, can any of this be more accurately determined
or guessed-at with knowledge of the onboard hardware?
Or is it dependent on external conditions?
It can be accurately determined with most cards (all modern ones)
because you get a notification when it's done, or you can poll the card.
That's why on the desktop it's able to detect when you plug in an
ethernet cable and start DHCP as soon as link negotiation is complete.
So the right thing to do, as David Miller suggested too, isn't a fixed
timeout. It should wait for link state UP and then start DHCP
immediately.
-- Jamie