ixgbe TX function selection

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function 
even if (dev->data->scattered_rx == 1). That seems a bit dangerous, as 
you can turn that on inadvertently when you don't set max_rx_pkt_len and 
buffer size in certain ways. I've learnt it in the hard way, as my 
segmented packets were leaking memory on the TX path, which doesn't 
cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the 
'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a 
solid way as other interfaces still can have SG turned on.

Regards,

Zoltan

Hi Zoltan,

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: dev@dpdk.org
Subject: [dpdk-dev] ixgbe TX function selection

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function even if

Thanks for let us know the problem. But I don't catch your point. Do you really mean TX here? After a quick look at the code, I don’t find the SG/non-SG functions for TX. Do I miss something?

(dev->data->scattered_rx == 1). That seems a bit dangerous, as you can turn
that on inadvertently when you don't set max_rx_pkt_len and buffer size in
certain ways. I've learnt it in the hard way, as my segmented packets were
leaking memory on the TX path, which doesn't cries if you send out segmented
packets.

Which one will cause problem? SG or non-SG packets? And where does the memory leak happen?

How should this case be treated? Assert on the non-SG TX side for the 'next'
pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other
interfaces still can have SG turned on.

Regards,

Zoltan

On 04/03/16 01:47, Lu, Wenzhuo wrote:

Hi Zoltan,

quoted

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: dev@dpdk.org
Subject: [dpdk-dev] ixgbe TX function selection

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function even if

Thanks for let us know the problem. But I don't catch your point. Do you really mean TX here? After a quick look at the code, I don’t find the SG/non-SG functions for TX. Do I miss something?

The simple code path doesn't handle multisegmented packets. 
ixgbe_txq_vec_setup() and ixgbe_xmit_pkts_simple() doesn't even check 
the next pointer of the mbuf, just put the first one on the descriptor 
ring, and when TX completion happens, the memory is leaked because it 
just sets ->next to NULL, and calls rte_mempool_put[_bulk]
ixgbe_xmit_pkts() puts all the segments on the descriptor ring, 
therefore when the descriptors are released they are released as well.
This is what these functions supposed to do, but my point is it's very 
easy to send a multisegmented packet to the simple code path.

quoted

(dev->data->scattered_rx == 1). That seems a bit dangerous, as you can turn
that on inadvertently when you don't set max_rx_pkt_len and buffer size in
certain ways. I've learnt it in the hard way, as my segmented packets were
leaking memory on the TX path, which doesn't cries if you send out segmented
packets.

Which one will cause problem? SG or non-SG packets? And where does the memory leak happen?

quoted

How should this case be treated? Assert on the non-SG TX side for the 'next'
pointer? Or turning on SG if RX has it? It doesn't seem to be a solid way as other
interfaces still can have SG turned on.

Regards,

Zoltan

Hi, Zoltan

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: dev@dpdk.org
Subject: [dpdk-dev] ixgbe TX function selection

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function
even if (dev->data->scattered_rx == 1). That seems a bit dangerous, as
you can turn that on inadvertently when you don't set max_rx_pkt_len and
buffer size in certain ways. I've learnt it in the hard way, as my
segmented packets were leaking memory on the TX path, which doesn't
cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the
'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a
solid way as other interfaces still can have SG turned on.

If you look into the ixgbe_set_tx_function, you will find tx function
selection is decided by the tx_flags on queue configure, which is
passed by rte_eth_txconf. So even you set dev->data->scattered_rx to 1,
if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple is
still selected as tx function. So, you'd better to set tx_flags=0, and have a try.

Regards,

Zoltan

On 10/03/16 07:51, Wu, Jingjing wrote:

Hi, Zoltan

quoted

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: dev@dpdk.org
Subject: [dpdk-dev] ixgbe TX function selection

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function
even if (dev->data->scattered_rx == 1). That seems a bit dangerous, as
you can turn that on inadvertently when you don't set max_rx_pkt_len and
buffer size in certain ways. I've learnt it in the hard way, as my
segmented packets were leaking memory on the TX path, which doesn't
cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the
'next' pointer? Or turning on SG if RX has it? It doesn't seem to be a
solid way as other interfaces still can have SG turned on.

If you look into the ixgbe_set_tx_function, you will find tx function
selection is decided by the tx_flags on queue configure, which is
passed by rte_eth_txconf. So even you set dev->data->scattered_rx to 1,
if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple is
still selected as tx function. So, you'd better to set tx_flags=0, and have a try.

You mean getting default_txconf from rte_eth_dev_info_get() and 
explicitly turn ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with 
zeros doesn't work very well) That's a way to solve it for me, but I'm 
rather talking about using defaults which doesn't cause memory leak 
quite easily.

quoted

Regards,

Zoltan

Hi Zoltan,

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Friday, March 18, 2016 1:11 AM
To: Wu, Jingjing; dev@dpdk.org
Subject: Re: [dpdk-dev] ixgbe TX function selection



On 10/03/16 07:51, Wu, Jingjing wrote:

quoted

Hi, Zoltan

quoted

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: dev@dpdk.org
Subject: [dpdk-dev] ixgbe TX function selection

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function
even if (dev->data->scattered_rx == 1). That seems a bit dangerous,
as you can turn that on inadvertently when you don't set
max_rx_pkt_len and buffer size in certain ways. I've learnt it in the
hard way, as my segmented packets were leaking memory on the TX path,
which doesn't cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the
'next' pointer? Or turning on SG if RX has it? It doesn't seem to be
a solid way as other interfaces still can have SG turned on.

If you look into the ixgbe_set_tx_function, you will find tx function
selection is decided by the tx_flags on queue configure, which is
passed by rte_eth_txconf. So even you set dev->data->scattered_rx to
1, if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple
is still selected as tx function. So, you'd better to set tx_flags=0, and have a try.

You mean getting default_txconf from rte_eth_dev_info_get() and explicitly turn
ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with zeros doesn't work
very well) That's a way to solve it for me, but I'm rather talking about using
defaults which doesn't cause memory leak quite easily.

Yes, ETH_TXQ_FLAGS_NOMULTSEGS only can be set to 1 when you know all your packets will not be segmented.
I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will be segmented or not.
You don't need to set tx_flags to 0, only the ETH_TXQ_FLAGS_NOMULTSEGS bit should be 0, the other bits can be 1 if needed.

quoted

quoted

Regards,

Zoltan

On 18/03/16 00:45, Lu, Wenzhuo wrote:

Hi Zoltan,

quoted

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Friday, March 18, 2016 1:11 AM
To: Wu, Jingjing; dev@dpdk.org
Subject: Re: [dpdk-dev] ixgbe TX function selection



On 10/03/16 07:51, Wu, Jingjing wrote:

quoted

Hi, Zoltan

quoted

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: dev@dpdk.org
Subject: [dpdk-dev] ixgbe TX function selection

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function
even if (dev->data->scattered_rx == 1). That seems a bit dangerous,
as you can turn that on inadvertently when you don't set
max_rx_pkt_len and buffer size in certain ways. I've learnt it in the
hard way, as my segmented packets were leaking memory on the TX path,
which doesn't cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the
'next' pointer? Or turning on SG if RX has it? It doesn't seem to be
a solid way as other interfaces still can have SG turned on.

If you look into the ixgbe_set_tx_function, you will find tx function
selection is decided by the tx_flags on queue configure, which is
passed by rte_eth_txconf. So even you set dev->data->scattered_rx to
1, if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple
is still selected as tx function. So, you'd better to set tx_flags=0, and have a try.

You mean getting default_txconf from rte_eth_dev_info_get() and explicitly turn
ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with zeros doesn't work
very well) That's a way to solve it for me, but I'm rather talking about using
defaults which doesn't cause memory leak quite easily.

Yes, ETH_TXQ_FLAGS_NOMULTSEGS only can be set to 1 when you know all your packets will not be segmented.
I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will be segmented or not.
You don't need to set tx_flags to 0, only the ETH_TXQ_FLAGS_NOMULTSEGS bit should be 0, the other bits can be 1 if needed.

So can we agree that the default settings should set 
ETH_TXQ_FLAGS_NOMULTSEGS to 0?

quoted

quoted

quoted

Regards,

Zoltan

Hi Zoltan,

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Friday, March 18, 2016 1:33 PM
To: Lu, Wenzhuo; Wu, Jingjing; dev@dpdk.org
Subject: Re: [dpdk-dev] ixgbe TX function selection



On 18/03/16 00:45, Lu, Wenzhuo wrote:

quoted

Hi Zoltan,

quoted

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Friday, March 18, 2016 1:11 AM
To: Wu, Jingjing; dev@dpdk.org
Subject: Re: [dpdk-dev] ixgbe TX function selection



On 10/03/16 07:51, Wu, Jingjing wrote:

quoted

Hi, Zoltan

quoted

-----Original Message-----
From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Zoltan Kiss
Sent: Wednesday, March 2, 2016 3:19 AM
To: dev@dpdk.org
Subject: [dpdk-dev] ixgbe TX function selection

Hi,

I've noticed that ixgbe_set_tx_function() selects the non-SG function
even if (dev->data->scattered_rx == 1). That seems a bit dangerous,
as you can turn that on inadvertently when you don't set
max_rx_pkt_len and buffer size in certain ways. I've learnt it in the
hard way, as my segmented packets were leaking memory on the TX path,
which doesn't cries if you send out segmented packets.
How should this case be treated? Assert on the non-SG TX side for the
'next' pointer? Or turning on SG if RX has it? It doesn't seem to be
a solid way as other interfaces still can have SG turned on.

If you look into the ixgbe_set_tx_function, you will find tx function
selection is decided by the tx_flags on queue configure, which is
passed by rte_eth_txconf. So even you set dev->data->scattered_rx to
1, if the tx_flags is ETH_TXQ_FLAGS_NOMULTSEGS, ixgbe_xmit_pkts_simple
is still selected as tx function. So, you'd better to set tx_flags=0, and have a try.

You mean getting default_txconf from rte_eth_dev_info_get() and explicitly turn
ETH_TXQ_FLAGS_NOMULTSEGS to 0? (filling tx_flags with zeros doesn't work
very well) That's a way to solve it for me, but I'm rather talking about using
defaults which doesn't cause memory leak quite easily.

Yes, ETH_TXQ_FLAGS_NOMULTSEGS only can be set to 1 when you know all your packets will not be segmented.
I think that means normally we should use full function path for TX, for we have no knowledge about if the packets will be

segmented or not.

quoted

You don't need to set tx_flags to 0, only the ETH_TXQ_FLAGS_NOMULTSEGS bit should be 0, the other bits can be 1 if needed.

So can we agree that the default settings should set
ETH_TXQ_FLAGS_NOMULTSEGS to 0?

I'd prefer to keep things as they are right now.
There always will be arguments and supporters for both alternatives: 
Should the fastest or the most comprehensive path be the default one.
Again default txq_flags can vary from one PMD to another. 
So, I think  the right behaviour for the app would be not to rely on default value
but set it up manually to the desired value. 
Konstantin

quoted

quoted

quoted

quoted

Regards,

Zoltan