From: David Miller <davem@davemloft.net> Date: 2009-06-15 09:07:35
Does the flow director implementation in IXGBE make sure to preserve
packet ordering in some way?
You can't just change the RX flow destination queue whenever you like.
For example, if a transmit happens on cpu 1, then we start to receive
some packets for that flow on that cpu, will a sudden set of transmits
on another cpu make the flow director quickly switch the RX
destination?
That would be very bad.
Without some extreme care that will result in potential packet
reordering.
There needs to be some safey net in place in your silicon to wait for
a significant "quiet time" in the RX stream before you can revector
the RX flow destination setting. Enough time to let the stack process
all that packets for that flow received on the currently bound flow
CPU.
To be honest, I'm not optimistic about this situation :-/
From: Waskiewicz Jr, Peter P <hidden> Date: 2009-06-16 06:42:29
On Mon, 15 Jun 2009, David Miller wrote:
Does the flow director implementation in IXGBE make sure to preserve
packet ordering in some way?
You can't just change the RX flow destination queue whenever you like.
For example, if a transmit happens on cpu 1, then we start to receive
some packets for that flow on that cpu, will a sudden set of transmits
on another cpu make the flow director quickly switch the RX
destination?
Not by design, no. The mode of Flow Director that is currently in the
driver is the hashing mode. 82599 also supports a perfect match mode,
which isn't supported yet, since there's no good interface to program the
5-tuple filters (I've been watching the ethtool work from the Sun guys for
niu, but haven't had time to help).
Back to the hashing mode. The idea is that a particular flow, say on CPU
1, will be hashed. There are two layers of hashing; the first
will land the flow into a bucket, the second will land it into the
filter lists. The two hash values for the Tx packet are then written
to the filter table along with the Rx queue. There are two ways that the
Rx queue for that flow could be moved: 1) The flow itself moves from CPU 1
to a different CPU. This is the most likely case that will move the Rx
flow. 2) We have a double collision on the hashes. This really isn't
very likely that a different flow will collide on both hashed values.
The reason we have two levels of hashing is to try and avoid this possible
situation of a collision accidentally moving a different flow.
That would be very bad.
Agreed.
Without some extreme care that will result in potential packet
reordering.
There needs to be some safey net in place in your silicon to wait for
a significant "quiet time" in the RX stream before you can revector
the RX flow destination setting. Enough time to let the stack process
all that packets for that flow received on the currently bound flow
CPU.
The current Rx DMA's that have been posted must be completed before our
hardware will apply the changes to the Rx filters.
To be honest, I'm not optimistic about this situation :-/
The way the internal hashing works in the software and hardware, I think
we're pretty safe with getting moved unexpectedely.
Cheers,
-PJ Waskiewicz
From: David Miller <davem@davemloft.net> Date: 2009-06-16 09:31:29
From: "Waskiewicz Jr, Peter P" <redacted>
Date: Mon, 15 Jun 2009 23:42:16 -0700 (Pacific Daylight Time)
The way the internal hashing works in the software and hardware, I think
we're pretty safe with getting moved unexpectedely.
The device RX being empty doesn't mean anything.
Those packets, even after you give them all to netif_receive_skb(),
can be held up in all number of queues, or even scheduled off to be
processed by ksoftirqd after a full context switch.
In the meantime you cannot switch the RX queue mapping without
introducing reordering.
So my original fears and doubts still stand :-)
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-07-05 04:19:55
David Miller [off-list ref] wrote:
In the meantime you cannot switch the RX queue mapping without
introducing reordering.
So my original fears and doubts still stand :-)
Methinks this whole idea is doomed. The fact is as long as there
are more sockets than queues, then you can have two sockets pull
the same queue in different directions and there's nothing you can
do about that.
That's my bit of uncertainty for you :)
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
In the meantime you cannot switch the RX queue mapping without
introducing reordering.
So my original fears and doubts still stand :-)
Methinks this whole idea is doomed. The fact is as long as there
are more sockets than queues, then you can have two sockets pull
the same queue in different directions and there's nothing you can
do about that.
That's my bit of uncertainty for you :)
Flows map to queues, queues don't move. Rather the mapping between
flows and queues changes dynamically.
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-07-06 01:10:53
On Sun, Jul 05, 2009 at 03:53:54PM -0700, David Miller wrote:
Flows map to queues, queues don't move. Rather the mapping between
flows and queues changes dynamically.
Oh that makes more sense.
In that case I don't see the big need in moving the flows/sockets
after establishment. After all process migration should be rare,
so if it does occur then it's OK for the existing sockets to stay
on their original queue.
I'm curious about what the limit on the number of flows is though.
Because when you go above that limit performance will magically
suck :)
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
In that case I don't see the big need in moving the flows/sockets
after establishment. After all process migration should be rare,
so if it does occur then it's OK for the existing sockets to stay
on their original queue.
I personally assume that a master process that accepts connections
and then hands them off down to worker threads is quite common.
I'm curious about what the limit on the number of flows is though.
Because when you go above that limit performance will magically
suck :)
Yes, of course that's an important aspect of the implementation.
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-07-06 02:55:13
On Sun, Jul 05, 2009 at 06:59:56PM -0700, David Miller wrote:
I personally assume that a master process that accepts connections
and then hands them off down to worker threads is quite common.
In this scenario I think it might be better to determine which
worker thread to hand the socket to based on the current RX queue
assignment. This way you won't have to worry about what limit
the hardware has in terms of how many flows it can assign.
Of course if you can't modify the app then you'd have to rely on
hardware assignment.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt