[PATCH, untested] Support for PPPOE on SMP

Paul Mackerras says PPPoE relies on receiving packets in wire order,
and he has bug reports caused by packet reordering.

This is icky.  Example code below:
1) Extract core queuing part of netif_rx into __netif_rx.

2) If the protocol is requires serialization, packets are put on a
   global "serial" queue instead of the local queue.  (Which protocols
   currently hardcoded).

3) One cpu (boot cpu as it happens) drains this serial queue, so it
   stays ordered.

4) Fix bug in cpu_raise_softirq: need to wake softirqd if it's a
   different cpu.

Another option would simply be to stamp a serialization number into
the skb if the proto needs serialization, and drop packets if serial
number goes backwards.  But since this is actually happening to
people, that would suck, too.

I don't understand the unbalanced dev_put in net_rx_action(), BTW.

Cheers,
Rusty.

diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.5.72-bk2/kernel/softirq.c working-2.5.72-bk2-serial-protocols/kernel/softirq.c

--- linux-2.5.72-bk2/kernel/softirq.c	2003-06-25 17:17:19.000000000 +1000
+++ working-2.5.72-bk2-serial-protocols/kernel/softirq.c	2003-06-25 14:55:15.000000000 +1000

@@ -130,7 +130,7 @@ inline void cpu_raise_softirq(unsigned i
 	 * Otherwise we wake up ksoftirqd to make sure we
 	 * schedule the softirq soon.
 	 */
-	if (!in_interrupt())
+	if (!in_interrupt() || cpu != smp_processor_id())
 		wakeup_softirqd(cpu);
 }
 

diff -urpN --exclude TAGS -X /home/rusty/devel/kernel/kernel-patches/current-dontdiff --minimal linux-2.5.72-bk2/net/core/dev.c working-2.5.72-bk2-serial-protocols/net/core/dev.c

--- linux-2.5.72-bk2/net/core/dev.c	2003-06-20 11:53:36.000000000 +1000
+++ working-2.5.72-bk2-serial-protocols/net/core/dev.c	2003-06-25 17:11:36.000000000 +1000

@@ -1323,42 +1323,11 @@ static void sample_queue(unsigned long d
 }
 #endif
 
-
-/**
- *	netif_rx	-	post buffer to the network code
- *	@skb: buffer to post
- *
- *	This function receives a packet from a device driver and queues it for
- *	the upper (protocol) levels to process.  It always succeeds. The buffer
- *	may be dropped during processing for congestion control or by the
- *	protocol layers.
- *
- *	return values:
- *	NET_RX_SUCCESS	(no congestion)
- *	NET_RX_CN_LOW   (low congestion)
- *	NET_RX_CN_MOD   (moderate congestion)
- *	NET_RX_CN_HIGH  (high congestion)
- *	NET_RX_DROP     (packet was dropped)
- *
- */
-
-int netif_rx(struct sk_buff *skb)
+/* Called with IRQs disabled. */
+static inline int __netif_rx(int this_cpu,
+			     struct softnet_data *queue,
+			     struct sk_buff *skb)
 {
-	int this_cpu;
-	struct softnet_data *queue;
-	unsigned long flags;
-
-	if (!skb->stamp.tv_sec)
-		do_gettimeofday(&skb->stamp);
-
-	/*
-	 * The code is rearranged so that the path is the most
-	 * short when CPU is congested, but is still operating.
-	 */
-	local_irq_save(flags);
-	this_cpu = smp_processor_id();
-	queue = &softnet_data[this_cpu];
-
 	netdev_rx_stat[this_cpu].total++;
 	if (queue->input_pkt_queue.qlen <= netdev_max_backlog) {
 		if (queue->input_pkt_queue.qlen) {

@@ -1371,7 +1340,6 @@ enqueue:
 #ifndef OFFLINE_SAMPLE
 			get_sample_stats(this_cpu);
 #endif
-			local_irq_restore(flags);
 			return queue->cng_level;
 		}
 

@@ -1397,12 +1365,116 @@ enqueue:
 
 drop:
 	netdev_rx_stat[this_cpu].dropped++;
-	local_irq_restore(flags);
 
 	kfree_skb(skb);
 	return NET_RX_DROP;
 }
 
+#ifdef CONFIG_SMP
+/* Queue for serial protocols (eg PPPoe).  All handled by one CPU. */
+static spinlock_t serial_queue_lock = SPIN_LOCK_UNLOCKED;
+static struct softnet_data serial_queue;
+
+/* Which cpu does serial queue. */
+static int serial_cpu;
+
+static inline int net_proto_serialize(struct sk_buff *skb,
+				      int this_cpu, 
+				      int *ret)
+{
+	if (likely(skb->protocol != ETH_P_PPP_DISC 
+		   && skb->protocol != ETH_P_PPP_SES))
+		return 0;
+
+	spin_lock(&serial_queue_lock);
+	*ret = __netif_rx(this_cpu, &serial_queue, skb);
+	spin_unlock(&serial_queue_lock);
+	if (this_cpu != serial_cpu)
+		cpu_raise_softirq(serial_cpu, NET_RX_SOFTIRQ);
+	return 1;
+}
+
+static void init_queue(struct softnet_data *queue);
+
+static void init_serial(void)
+{
+	init_queue(&serial_queue);
+	serial_cpu = smp_processor_id();
+}
+
+static inline void drain_serial_queue(int this_cpu)
+{
+	if (this_cpu != serial_cpu)
+		return;
+
+	spin_lock(&serial_queue_lock);
+	while (!list_empty(&serial_queue.poll_list)) {
+		struct net_device *dev;
+
+		dev = list_entry(serial_queue.poll_list.next,
+				 struct net_device, poll_list);
+
+		list_del(&dev->poll_list);
+		list_add_tail(&dev->poll_list, &serial_queue.poll_list);
+	}
+	spin_unlock(&serial_queue_lock);
+}
+#else
+static inline int net_proto_serialize(struct sk_buff *skb,
+				      int this_cpu,
+				      int *ret)
+{
+	return 0;
+}
+
+static void init_serial(void)
+{
+}
+
+static inline void drain_serial_queue(int this_cpu)
+{
+}
+#endif /* CONFIG_SMP */
+
+/**
+ *	netif_rx	-	post buffer to the network code
+ *	@skb: buffer to post
+ *
+ *	This function receives a packet from a device driver and queues it for
+ *	the upper (protocol) levels to process.  It always succeeds. The buffer
+ *	may be dropped during processing for congestion control or by the
+ *	protocol layers.
+ *
+ *	return values:
+ *	NET_RX_SUCCESS	(no congestion)
+ *	NET_RX_CN_LOW   (low congestion)
+ *	NET_RX_CN_MOD   (moderate congestion)
+ *	NET_RX_CN_HIGH  (high congestion)
+ *	NET_RX_DROP     (packet was dropped)
+ *
+ */
+
+int netif_rx(struct sk_buff *skb)
+{
+	int ret, this_cpu;
+	unsigned long flags;
+
+	if (!skb->stamp.tv_sec)
+		do_gettimeofday(&skb->stamp);
+
+	/*
+	 * The code is rearranged so that the path is the most
+	 * short when CPU is congested, but is still operating.
+	 */
+	local_irq_save(flags);
+	this_cpu = smp_processor_id();
+
+	if (!net_proto_serialize(skb, this_cpu, &ret))
+		ret = __netif_rx(this_cpu, &softnet_data[this_cpu], skb);
+	local_irq_restore(flags);
+	return ret;
+}
+
 /* Deliver skb to an old protocol, which is not threaded well
    or which do not understand shared skbs.
  */

@@ -1705,6 +1777,8 @@ static void net_rx_action(struct softirq
 			local_irq_disable();
 		}
 	}
+
+	drain_serial_queue(this_cpu);
 out:
 	local_irq_enable();
 	preempt_enable();

@@ -2944,6 +3018,20 @@ int unregister_netdevice(struct net_devi
 }
 
 
+static void init_queue(struct softnet_data *queue)
+{
+	skb_queue_head_init(&queue->input_pkt_queue);
+	queue->throttle = 0;
+	queue->cng_level = 0;
+	queue->avg_blog = 10; /* arbitrary non-zero */
+	queue->completion_queue = NULL;
+	INIT_LIST_HEAD(&queue->poll_list);
+	set_bit(__LINK_STATE_START, &queue->backlog_dev.state);
+	queue->backlog_dev.weight = weight_p;
+	queue->backlog_dev.poll = process_backlog;
+	atomic_set(&queue->backlog_dev.refcnt, 1);
+}
+
 /*
  *	Initialize the DEV module. At boot time this walks the device list and
  *	unhooks any devices that fail to initialise (normally hardware not

@@ -2976,21 +3064,9 @@ static int __init net_dev_init(void)
 	 *	Initialise the packet receive queues.
 	 */
 
-	for (i = 0; i < NR_CPUS; i++) {
-		struct softnet_data *queue;
-
-		queue = &softnet_data[i];
-		skb_queue_head_init(&queue->input_pkt_queue);
-		queue->throttle = 0;
-		queue->cng_level = 0;
-		queue->avg_blog = 10; /* arbitrary non-zero */
-		queue->completion_queue = NULL;
-		INIT_LIST_HEAD(&queue->poll_list);
-		set_bit(__LINK_STATE_START, &queue->backlog_dev.state);
-		queue->backlog_dev.weight = weight_p;
-		queue->backlog_dev.poll = process_backlog;
-		atomic_set(&queue->backlog_dev.refcnt, 1);
-	}
+	for (i = 0; i < NR_CPUS; i++)
+		init_queue(&softnet_data[i]);
+	init_serial();
 
 #ifdef CONFIG_NET_PROFILE
 	net_profile_init();
--
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.

On Wed, 25 Jun 2003, Rusty Russell wrote:

Paul Mackerras says PPPoE relies on receiving packets in wire order,
and he has bug reports caused by packet reordering.

I dont know of any ordering dependencies with pppoe. Is this a bug
in the ppp code?

This is icky.

Yes it is ;->
The effects of your patch could be achieved in two ways:
a) tie the pppoe related ethernet card to a processor.
b) use a NAPI caopable ethernet card.

Now, if there is a real need to have a serialization queue (i dont see
one) you really dont need to tie to a processor. Just have a single queue
shared by all processors; every one grabs a lock to it.

cheers,
jamal

First some background for those new to this discussion (I was going post
the original discussion that strted this to this list, but the summary
here should get everyone up to speed).

A user has observed a race condition where the last packet of PPPoE
discovery arrives just before the first payload packet.  The discovery
packet carries the session id and pppd needs to take this session id and
create a PPPoE socket which will then pick up all packets matching the
given session id.  The race is between the arrival of the first payload
packet and  pppd's creation of the socket that is to receive PPPoE
payload.  If the packet wins the race, the payload packet is lost.  This
problem was noticed only because the ISP in this case configured their
systems to use a longer, non-standard (but legal) retransmit timeout
thus causing noticeable delays in PPP negotiation.

About the patch:  Do we have any guarantees that no drivers will break
this?  From the few drivers I've looked at, this will not be a problem
since they lock to ensure that we can't have races in submitting packets
to netif_rx.  My concern here would be that it appears that there is no
explicit requirement that this be so; we may be safe in this regard only
by accident.  (I can think of a device and driver design where this need
not be so.) 

I believe there are concerns with other protocols as well (SNA, spanning
tree - I'm just the messenger on this).  If this is so, then I have two
concerns:

  1.  Some protocols may have no in-kernel implementation, we'd have to
      ensure that raw sockets get packets in the right order (perhaps 
      even regardless of what packet type we hreceive).

  2.  There are two issues with PPPoE: there's the creation race
      described above which requires correct ordering of packets of two
      different packet types (discovery is 0x8863, payload is 0x8864),
      as well payload packets must be ordered to handle Paul's concerns 
      regarding compression.

The patch as is adequate to 2), but I'm concerned it would get ugly if
we need to do 1) (and in the process of doing 1) we may break 2) if we
can't synchronize between two different packet types).

I think we can fix the race condition I've described up top without such
core infrastructure changes (delay dropping unmatched payload packets,
give pppd a chance to make the socket).  This however doesn't solve the
other ordering problems.

-- 
Michal Ostrowski [off-list ref]

Perhaps instead of using a special queue that keeps packets ordered, we
add a tag to each skb as it comes off the card and let higher level
protocols use this to re-order things themselves?  (And add some option
for AF_PACKET sockets to optionally enforce this ordering in presenting
packets to apps, or not.)

This may require modifying all drivers, but it does provide for an
explicit mechanism that can be made mandatory for drivers, avoids
special casing, avoids dumping work onto a single CPU and leaves it up
to the higher-level code to figure out ordering, if it wants to.

-- 
Michal Ostrowski [off-list ref]

Have you tested the case where the ethernet card is tied to only
CPU in SMP? That guarantees ordering.
Ordering per protocol should really be that protocols problem to
solve. If you cant solve it you have a bug.

cheers,
jamal

On Wed, 25 Jun 2003, Michal Ostrowski wrote:

Perhaps instead of using a special queue that keeps packets ordered, we
add a tag to each skb as it comes off the card and let higher level
protocols use this to re-order things themselves?  (And add some option
for AF_PACKET sockets to optionally enforce this ordering in presenting
packets to apps, or not.)

This may require modifying all drivers, but it does provide for an
explicit mechanism that can be made mandatory for drivers, avoids
special casing, avoids dumping work onto a single CPU and leaves it up
to the higher-level code to figure out ordering, if it wants to.

--
Michal Ostrowski [off-list ref]

rusty@rustcorp.com.au said:

I don't understand the unbalanced dev_put in net_rx_action(), BTW.

It's tricky. There are two paths an skb can take into net_rx_action(),
napi and non-napi. The non-napi path uses dev_hold/dev_put on both
skb->dev and a virtual per-cpu struct net_device, the backlog_dev.

In a non-napi skb receive, the driver uses netif_rx() to hand the skb up
to the net core. netif_rx does a dev_hold on skb->dev, puts the skb on
the current cpu's softnet_data queue, and uses netif_rx_schedule to
schedule that softnet-data's ->backlog_dev to be polled.  In the
process, __netif_rx_schedule does a dev_hold(backlog_dev).

So the queue of ready net_devices processed by net_rx_action may contain
actual struct net_devices (napi) or the virtual ->backlog_dev
net_device. In the former case, net_rx_action's dev_put balances the
dev_hold done when the driver called __netif_rx_schedule(). In the
latter case, net_rx_action's dev_put balances the dev_hold of the
backlog_dev done when netif_rx called __netif_rx_schedule().

I hope that makes some kind of sense. It took a while to figure out, but
I saved my notes. :)

Jason

On 25 Jun 2003 09:21:02 -0400
Michal Ostrowski [off-list ref] wrote:

First some background for those new to this discussion (I was going post
the original discussion that strted this to this list, but the summary
here should get everyone up to speed).

A user has observed a race condition where the last packet of PPPoE
discovery arrives just before the first payload packet.  The discovery
packet carries the session id and pppd needs to take this session id and
create a PPPoE socket which will then pick up all packets matching the
given session id.  The race is between the arrival of the first payload
packet and  pppd's creation of the socket that is to receive PPPoE
payload.  If the packet wins the race, the payload packet is lost.  This
problem was noticed only because the ISP in this case configured their
systems to use a longer, non-standard (but legal) retransmit timeout
thus causing noticeable delays in PPP negotiation.

Also, you only need the ordering dependency till the session is setup, not
after it is established.  Imagine a large ISP with many PPPoE sessions; it
makes no sense to serialize traffic just for this session establishment case.

In the long run, the right answer probably is to push the session management
out of the daemon and into the kernel.  Today the PPPoE code in the kernel
is only half-brained, it needs pppd to survive.

On Wed, 25 Jun 2003, Stephen Hemminger wrote:

In the long run, the right answer probably is to push the session management
out of the daemon and into the kernel.  Today the PPPoE code in the kernel
is only half-brained, it needs pppd to survive.

I would think pppd is the half-brained portion ;->

Placing control protocols in the kernel is plain wrong.

cheers,
jamal

On Wed, 25 Jun 2003 12:22:35 -0400 (EDT)
Jamal Hadi [off-list ref] wrote:

On Wed, 25 Jun 2003, Stephen Hemminger wrote:

quoted

In the long run, the right answer probably is to push the session management
out of the daemon and into the kernel.  Today the PPPoE code in the kernel
is only half-brained, it needs pppd to survive.

I would think pppd is the half-brained portion ;->

Placing control protocols in the kernel is plain wrong.

What about arp, TCP, IP, routing protocols.  The problem is that state management
needs to be done in one place.

On Wed, 25 Jun 2003, Stephen Hemminger wrote:

On Wed, 25 Jun 2003 12:22:35 -0400 (EDT)
Jamal Hadi [off-list ref] wrote:

quoted

Placing control protocols in the kernel is plain wrong.

What about arp, TCP, IP, routing protocols.

ARP should really be ripped off the kernel. I mentioned to you once
the same in regards to STP and iirc you agreed.
I  wouldnt call TCP or IP control protocols.

The problem is that state management needs to be done in one place.

a protocol or implementation which wishes to do state maintanance
properly oughta be able to do the synchronization on its own.
Separation between policy and mechanism has been the strength of unix.
A clean separation between control and a data path is very important.
Control protocols tend to be very rich environments which are
constantly changing. Take STP, there are so many features that could be
added to STP that are much harder to add because it is in the kernel.

Maybe what needs to be looked at i sthe design of pppoe or ppp.
The patch from Rusty is just bandaid.


cheers,
jamal

Paul:  you made an assertion to me in an eariler e-mail that you were
concerned about packet ordering for the sake of vj and compression. 
IIRC the PPPoE spec prohibits compression, probably for this very
reason. Is there any other reason we'd be worried about re-ordering in
the PPP data stream?

On Wed, 2003-06-25 at 11:45, Jamal Hadi wrote:

Have you tested the case where the ethernet card is tied to only
CPU in SMP? That guarantees ordering.

Agreed, this does guarantee ordering.  But there are cases where I don't
have this guarantee and those are the issues Rusty's patch attempts to
solve.

Ordering per protocol should really be that protocols problem to
solve. If you cant solve it you have a bug.

The session initiation race I described earlier is brought about
independently by several problems:

1.  PPPoE negotiation is done in user space and thus there is a window
    between completion of this negotiation and the creation of the PPPoE
    socket during which a payload packet may arrive and be dropped (SMP
    and UP).

2.  Re-ordering by softIRQ handling on SMP may cause same problem.

There's also the question as to whether or not there are other protocols
(perhaps not implemented in the kernel, but relying on AF_PACKET) may be
affected by this (#2).

We can fix #1 without any patches to core networking code.  If the SMP
softIRQ re-ordering issues is handled, then we may have some better
options for fixing #1.  But note that even if #1 is fixed and #2 isn't,
then we're not any better off. 

-- 
Michal Ostrowski [off-list ref]

On Wed, 25 Jun 2003 13:07:46 -0400 (EDT)
Jamal Hadi [off-list ref] wrote:

On Wed, 25 Jun 2003, Stephen Hemminger wrote:

quoted

On Wed, 25 Jun 2003 12:22:35 -0400 (EDT)
Jamal Hadi [off-list ref] wrote:

quoted

Placing control protocols in the kernel is plain wrong.

What about arp, TCP, IP, routing protocols.

ARP should really be ripped off the kernel. I mentioned to you once
the same in regards to STP and iirc you agreed.
I  wouldnt call TCP or IP control protocols.

quoted

The problem is that state management needs to be done in one place.

a protocol or implementation which wishes to do state maintanance
properly oughta be able to do the synchronization on its own.
Separation between policy and mechanism has been the strength of unix.
A clean separation between control and a data path is very important.
Control protocols tend to be very rich environments which are
constantly changing. Take STP, there are so many features that could be
added to STP that are much harder to add because it is in the kernel.

Rather than take an architectural approach about what is right and wrong,
I take the practical point of view.  If the protocol is small, and the 
policy can be done in the kernel fine; if the implementation gets messy
and the right information is not there, then it belongs in user space.

For PPPoE, the session management needs to be in kernel space, with the policy
in user space.  What if the kernel, initialized the session when it saw
the discovery and notified the pppd,  session would not be established
until ppd accepted the connection.  This would be more like a socket
protocol without auto-accept like TCP. Any data for the session would
then stay queued until it was accepted or rejected.

Having special non-SMP receive logic is bogus; and probably won't work
anyway with preempt and other races.

There is already work in moving STP out of the kernel, but even that
has shown that the problem is how to have the proper management hooks
to do the job. That is why it hasn't been a simple slam dunk.

On Wed, 2003-06-25 at 13:40, Stephen Hemminger wrote:

For PPPoE, the session management needs to be in kernel space, with the policy
in user space.  What if the kernel, initialized the session when it saw
the discovery and notified the pppd,  session would not be established
until ppd accepted the connection.  This would be more like a socket
protocol without auto-accept like TCP. Any data for the session would
then stay queued until it was accepted or rejected.

Regardless of the solution take for the session-initiation race, any
solution would fall apart if SMP softIRQ's can reorder packets (that is
it would result in dropped packets).  Only once there is a solution for
this reordering problem does it make sense to consider the options for
handling this race.

PPPoE also doesn't exactly cleanly fit nicely into the standard
bind()/listen()/accept()/connect() mould.  I've been convinced that
negotiation/discovery belongs in pppd and so would like to avoid adding
connection detection logic into the kernel.

Finally, please keep in mind that with PPPoE when we do hit this problem
the effect is that PPP session establishment takes a bit longer since we
have to wait for an LCP timeout and retransmit.   I am much more curious
about how other protocols may be affected by packet reordering.

-- 
Michal Ostrowski [off-list ref]

Why don't you just queue the payload packets in a "resolution queue"
until the socket is created?  Just make the resolution queue packets
timeout using a value that will easily exceed any reasonable PPP
negotiation time.

All this ordered packet arrival shit is just beyond stupid.

On Wed, 2003-06-25 at 17:33, David S. Miller wrote:

Why don't you just queue the payload packets in a "resolution queue"
until the socket is created?  Just make the resolution queue packets
timeout using a value that will easily exceed any reasonable PPP
negotiation time.

All this ordered packet arrival shit is just beyond stupid.

Exactly this mechanism is what I had in mind.  

The open question remaining is if there are any protocols which can be
affected by packets being processed out of order.  Some people have
suggested that there are.  If not, then there's not much to discuss. Can
anyone comment on this decisively, either way?



-- 
Michal Ostrowski [off-list ref]

Michal Ostrowski writes:

Paul:  you made an assertion to me in an eariler e-mail that you were
concerned about packet ordering for the sake of vj and compression. 
IIRC the PPPoE spec prohibits compression, probably for this very
reason. Is there any other reason we'd be worried about re-ordering in
the PPP data stream?

Reordering would stop you doing multilink, for instance.  Generally,
PPP protocols assume ordering where it is helpful since most
point-to-point links don't reorder packets.  IMO the PPPoE protocol
itself should have included a sequence number, but we can't change
what's deployed.

James might be able to comment better than me on what will happen if
packets get reordered during the negotiation phase of a PPP
connection.  I think the worst is that some packets will have to be
retransmitted and thus the negotiation will take several seconds
longer than it needs to.

Paul.

Jamal Hadi writes:

a protocol or implementation which wishes to do state maintanance
properly oughta be able to do the synchronization on its own.
Separation between policy and mechanism has been the strength of unix.
A clean separation between control and a data path is very important.
Control protocols tend to be very rich environments which are
constantly changing. Take STP, there are so many features that could be
added to STP that are much harder to add because it is in the kernel.

Maybe what needs to be looked at i sthe design of pppoe or ppp.

OK, now that we have had our little flight of fancy about what things
will be like once we get to heaven, can we talk about this bastard
protocol called PPPoE? :)

Or are you going to go personally to each ISP in the world and tell
them they shouldn't use PPPoE? :)

In any case the problem isn't strictly with PPPoE, since ethernet
doesn't reorder packets on the wire.  The problem is that the lower
parts of the Linux network stack lose information.

Paul.

Paul Mackerras wrote:

Jamal Hadi writes:

quoted

a protocol or implementation which wishes to do state maintanance
properly oughta be able to do the synchronization on its own.
Separation between policy and mechanism has been the strength of unix.
A clean separation between control and a data path is very important.
Control protocols tend to be very rich environments which are
constantly changing. Take STP, there are so many features that could be
added to STP that are much harder to add because it is in the kernel.

Maybe what needs to be looked at i sthe design of pppoe or ppp.

OK, now that we have had our little flight of fancy about what things
will be like once we get to heaven, can we talk about this bastard
protocol called PPPoE? :)

Or are you going to go personally to each ISP in the world and tell
them they shouldn't use PPPoE? :)

In any case the problem isn't strictly with PPPoE, since ethernet
doesn't reorder packets on the wire.  The problem is that the lower
parts of the Linux network stack lose information.

Paul.

Nothing is guaranteed, but you may be right at least most of
the time.  Btw, if you want a proprietary tool that
will emulate an ethernet network that reorders packets, I write
such a thing and will give it to you.  It could help you
with testing perhaps.

Also, if you have a PCMCIA Zircom NIC, it seems to reorder packets
just for the hell of it (and no, I'm not using a dual-cpu laptop :))

I don't know of any other protocols that can't handle reordering,
since most of them seem to be designed to run over the real internet,
where reordering/drop/duplication is a part of life.

Ben

-- 
Ben Greear [off-list ref]       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

On Wed, 2003-06-25 at 18:17, Paul Mackerras wrote:

James might be able to comment better than me on what will happen if
packets get reordered during the negotiation phase of a PPP
connection.  I think the worst is that some packets will have to be
retransmitted and thus the negotiation will take several seconds
longer than it needs to.

This is exactly what we're dealing with the current "bug"; the worst
case effect is a delay.  I don't think heroic measures are called for
the sake of this PPPoE issue alone.

-- 
Michal Ostrowski [off-list ref]

   From: Michal Ostrowski [off-list ref]
   Date: 25 Jun 2003 18:06:54 -0400

   Exactly this mechanism is what I had in mind.  
   
Great.

   The open question remaining is if there are any protocols which can be
   affected by packets being processed out of order.  Some people have
   suggested that there are.  If not, then there's not much to discuss. Can
   anyone comment on this decisively, either way?

TCP, as one example, is able to cope very well.  It is even able to
distinguish reordering from true packet loss.