# Re: \[RFC PATCH\] net: Introduce a socket option to enable picking tx queue based on rx queue.

- From: Samudrala, Sridhar \<sridhar.samudrala@intel.com\>
- Date: 2017-09-12 22:31:46 
- List: netdev
- Page: https://ratatoskr.run/netdev/2017/09/11065957
- Thread: https://ratatoskr.run/netdev/2017/08/11064975/t.md

```text


On 9/12/2017 8:47 AM, Eric Dumazet wrote:
> On Mon, 2017-09-11 at 23:27 -0700, Samudrala, Sridhar wrote:
>> On 9/11/2017 8:53 PM, Eric Dumazet wrote:
>>> On Mon, 2017-09-11 at 20:12 -0700, Tom Herbert wrote:
>>>
>>>> Two ints in sock_common for this purpose is quite expensive and the
>>>> use case for this is limited-- even if a RX->TX queue mapping were
>>>> introduced to eliminate the queue pair assumption this still won't
>>>> help if the receive and transmit interfaces are different for the
>>>> connection. I think we really need to see some very compelling results
>>>> to be able to justify this.
>> Will try to collect and post some perf data with symmetric queue
>> configuration.
>>
>>> Yes, this is unreasonable cost.
>>>
>>> XPS should really cover the case already.
>>>    
>> Eric,
>>
>> Can you clarify how XPS covers the RX-> TX queue mapping case?
>> Is it possible to configure XPS to select TX queue based on the RX queue
>> of a flow?
>> IIUC, it is based on the CPU of the thread doing the transmit OR based
>> on skb->priority to TC mapping?
>> It may be possible to get this effect if the the threads are pinned to a
>> core, but if the app threads are
>> freely moving, i am not sure how XPS can be configured to select the TX
>> queue based on the RX queue of a flow.
> If application is freely moving, how NIC can properly select the RX
> queue so that packets are coming to the appropriate queue ?
The RX queue is selected via RSS and we don't want to move the flow based on
where the thread is running.
>
> This is called aRFS, and it does not scale to millions of flows.
> We tried in the past, and this went nowhere really, since the setup cost
> is prohibitive and DDOS vulnerable.
>
> XPS will follow the thread, since selection is done on current cpu.
>
> The problem is RX side. If application is free to migrate, then special
> support (aRFS) is needed from the hardware.
This may be true if most of the rx processing is happening in the interrupt context.
But with busy polling,  i think we don't need aRFS as a thread should be able to poll
any queue irrespective of where it is running.
>
> At least for passive connections, we already have all the support in the
> kernel so that you can have one thread per NIC queue, dealing with
> sockets that have incoming packets all received on one NIC RX queue.
> (And of course all TX packets will use the symmetric TX queue)
>
> SO_REUSEPORT plus appropriate BPF filter can achieve that.
>
> Say you have 32 queues, 32 cpus.
>
> Simply use 32 listeners, 32 threads (or 32 pools of threads)
Yes. This will work if each thread is pinned to a core associated with the RX interrupt.
It may not be possible to pin the threads to a core.
Instead we want to associate a thread to a queue and do all the RX and TX completion
of a queue in the same thread context via busy polling.

Thanks
Sridhar
```