Hi netdev folks,
It looks like we call dev_forward_skb() in veth_xmit(), which calls
netif_rx() eventually.
While netif_rx() will enqueue the skb to the CPU RX backlog before the
actual processing takes place. So, this actually means a TX skb has to
wait some un-related RX skbs to finish. And this will happen twice for
a single ping, because the veth device always works as a pair?
IMHO this might lead to some latency issue under certain workload,
can we change the call to dev_forward_skb() to something like this?
if (likely(__dev_forward_skb(rcv, skb) == NET_RX_SUCCESS)) {
local_bh_disable();
netif_receive_skb(skb);
local_bh_enable();
Could you please shed some light on this change? And please feel free
to correct my if my understanding is wrong.
Thanks,
- Xiangning
From: Eric Dumazet <hidden> Date: 2017-01-23 19:07:44
On Mon, 2017-01-23 at 10:46 -0800, Xiangning Yu wrote:
Hi netdev folks,
It looks like we call dev_forward_skb() in veth_xmit(), which calls
netif_rx() eventually.
While netif_rx() will enqueue the skb to the CPU RX backlog before the
actual processing takes place. So, this actually means a TX skb has to
wait some un-related RX skbs to finish. And this will happen twice for
a single ping, because the veth device always works as a pair?
IMHO this might lead to some latency issue under certain workload,
can we change the call to dev_forward_skb() to something like this?
if (likely(__dev_forward_skb(rcv, skb) == NET_RX_SUCCESS)) {
local_bh_disable();
netif_receive_skb(skb);
local_bh_enable();
Could you please shed some light on this change? And please feel free
to correct my if my understanding is wrong.
How veth would have different latency requirement than loopback device ?
Calling netif_receive_skb() is considered too dangerous here (or from
any ndo_start_xmit()) because of possible kernel stack exhaustion.
On Mon, Jan 23, 2017 at 11:07 AM, Eric Dumazet [off-list ref] wrote:
On Mon, 2017-01-23 at 10:46 -0800, Xiangning Yu wrote:
quoted
Hi netdev folks,
It looks like we call dev_forward_skb() in veth_xmit(), which calls
netif_rx() eventually.
While netif_rx() will enqueue the skb to the CPU RX backlog before the
actual processing takes place. So, this actually means a TX skb has to
wait some un-related RX skbs to finish. And this will happen twice for
a single ping, because the veth device always works as a pair?
IMHO this might lead to some latency issue under certain workload,
can we change the call to dev_forward_skb() to something like this?
if (likely(__dev_forward_skb(rcv, skb) == NET_RX_SUCCESS)) {
local_bh_disable();
netif_receive_skb(skb);
local_bh_enable();
Could you please shed some light on this change? And please feel free
to correct my if my understanding is wrong.
How veth would have different latency requirement than loopback device ?
The traffic from those veth device will reach external network, and
normally those are RPC type traffic.
Calling netif_receive_skb() is considered too dangerous here (or from
any ndo_start_xmit()) because of possible kernel stack exhaustion.
I agree, stack space is a concern, especially if the traffic is
loopback-ed to another namespace.
Thanks,
- Xiangning
From: Cong Wang <hidden> Date: 2017-01-23 20:56:43
On Mon, Jan 23, 2017 at 10:46 AM, Xiangning Yu [off-list ref] wrote:
Hi netdev folks,
It looks like we call dev_forward_skb() in veth_xmit(), which calls
netif_rx() eventually.
While netif_rx() will enqueue the skb to the CPU RX backlog before the
actual processing takes place. So, this actually means a TX skb has to
wait some un-related RX skbs to finish. And this will happen twice for
a single ping, because the veth device always works as a pair?
For me it is more like for the completeness of network stack of each
netns. The /proc net.core.netdev_max_backlog etc. are per netns, which
means each netns, as an independent network stack, should respect it
too.
Since you care about latency, why not tune net.core.dev_weight for your
own netns?
On Mon, Jan 23, 2017 at 12:56 PM, Cong Wang [off-list ref] wrote:
On Mon, Jan 23, 2017 at 10:46 AM, Xiangning Yu [off-list ref] wrote:
quoted
Hi netdev folks,
It looks like we call dev_forward_skb() in veth_xmit(), which calls
netif_rx() eventually.
While netif_rx() will enqueue the skb to the CPU RX backlog before the
actual processing takes place. So, this actually means a TX skb has to
wait some un-related RX skbs to finish. And this will happen twice for
a single ping, because the veth device always works as a pair?
For me it is more like for the completeness of network stack of each
netns. The /proc net.core.netdev_max_backlog etc. are per netns, which
means each netns, as an independent network stack, should respect it
too.
Since you care about latency, why not tune net.core.dev_weight for your
own netns?
I haven't tried that yet, thank you for the hint! Though normally one
of the veth device will be in the global namespace.
Thanks,
- Xiangning
From: Eric Dumazet <hidden> Date: 2017-01-23 22:05:47
On Mon, 2017-01-23 at 13:46 -0800, Xiangning Yu wrote:
On Mon, Jan 23, 2017 at 12:56 PM, Cong Wang [off-list ref] wrote:
quoted
On Mon, Jan 23, 2017 at 10:46 AM, Xiangning Yu [off-list ref] wrote:
quoted
Hi netdev folks,
It looks like we call dev_forward_skb() in veth_xmit(), which calls
netif_rx() eventually.
While netif_rx() will enqueue the skb to the CPU RX backlog before the
actual processing takes place. So, this actually means a TX skb has to
wait some un-related RX skbs to finish. And this will happen twice for
a single ping, because the veth device always works as a pair?
For me it is more like for the completeness of network stack of each
netns. The /proc net.core.netdev_max_backlog etc. are per netns, which
means each netns, as an independent network stack, should respect it
too.
Since you care about latency, why not tune net.core.dev_weight for your
own netns?
I haven't tried that yet, thank you for the hint! Though normally one
of the veth device will be in the global namespace.
Well, per cpu backlog are not per net ns, but per cpu.
So Cong suggestion is not going to work.
From: Cong Wang <hidden> Date: 2017-01-23 22:42:43
On Mon, Jan 23, 2017 at 2:05 PM, Eric Dumazet [off-list ref] wrote:
On Mon, 2017-01-23 at 13:46 -0800, Xiangning Yu wrote:
quoted
On Mon, Jan 23, 2017 at 12:56 PM, Cong Wang [off-list ref] wrote:
quoted
On Mon, Jan 23, 2017 at 10:46 AM, Xiangning Yu [off-list ref] wrote:
quoted
Hi netdev folks,
It looks like we call dev_forward_skb() in veth_xmit(), which calls
netif_rx() eventually.
While netif_rx() will enqueue the skb to the CPU RX backlog before the
actual processing takes place. So, this actually means a TX skb has to
wait some un-related RX skbs to finish. And this will happen twice for
a single ping, because the veth device always works as a pair?
For me it is more like for the completeness of network stack of each
netns. The /proc net.core.netdev_max_backlog etc. are per netns, which
means each netns, as an independent network stack, should respect it
too.
Since you care about latency, why not tune net.core.dev_weight for your
own netns?
I haven't tried that yet, thank you for the hint! Though normally one
of the veth device will be in the global namespace.
Well, per cpu backlog are not per net ns, but per cpu.
Right, they all point to a same weight_p. But weight_p itself
is not per cpu, softnet_data is. However, if a container uses
cpuset and netns for isolations, per cpu will turn into per netns.
The point of network stack completeness still stands.