Re: [dpdk-dev] [EXT] Re: [PATCH 1/3] mbuf: add Tx offloads for packet marking

On Wed, Jun 03, 2020 at 10:28:44AM +0200, Olivier Matz wrote:

Hi,

On Tue, Jun 02, 2020 at 07:55:37PM +0530, Nithin Dabilpuram wrote:

quoted

On Tue, Jun 02, 2020 at 10:53:08AM +0000, Ananyev, Konstantin wrote:

quoted

Hi Jerin,

quoted

quoted

quoted

quoted

I also share Olivier's concern about consuming 3 bits in ol_flags for that feature.
Can it probably be squeezed somehow?
Let say we reserve one flag that this information is present or not, and
re-use one of rx-only fields for store additional information (packet_type, or so).
Or might be some other approach.

We are fine with this approach where we define one bit in Tx offloads for pkt
marking and and 3 bits reused from Rx offload flags area.

For example:

@@ -186,10 +186,16 @@ extern "C" {

 /* add new RX flags here, don't forget to update PKT_FIRST_FREE */

+/* Reused Rx offload bits for Tx offloads */
+#define PKT_X_TX_MARK_VLAN_DEI         (1ULL << 0)
+#define PKT_X_TX_MARK_IP_DSCP          (1ULL << 1)
+#define PKT_X_TX_MARK_IP_ECN           (1ULL << 2)
+
 #define PKT_FIRST_FREE (1ULL << 23)
-#define PKT_LAST_FREE (1ULL << 40)
+#define PKT_LAST_FREE (1ULL << 39)

 /* add new TX flags here, don't forget to update PKT_LAST_FREE  */
+#define PKT_TX_MARK_EN         (1ULL << 40)

Is this fine ?

Any thoughts on this approach which uses only 1 bit in Tx flags out of 18
and reuse unused Rx flag bits ?

My thought was not about re-defining the flags (I think it is better to keep them intact),
but adding a union for one of rx-only fields (packet_type/rss/timestamp).

Ok. Adding a union field at packet_type field is also fine like below. 

@@ -187,9 +187,10 @@ extern "C" {
 /* add new RX flags here, don't forget to update PKT_FIRST_FREE */
 
 #define PKT_FIRST_FREE (1ULL << 23)
-#define PKT_LAST_FREE (1ULL << 40)
+#define PKT_LAST_FREE (1ULL << 39)
 
 /* add new TX flags here, don't forget to update PKT_LAST_FREE  */
+#define PKT_TX_MARK_EN		(1ULL << 40)
 
 /**
  * Outer UDP checksum offload flag. This flag is used for enabling

@@ -461,6 +462,14 @@ enum {
 #endif
 };
 
+/* Tx packet marking flags in rte_mbuf::tx_mark.
+ * Valid only when PKT_TX_MARK_EN is set in
+ * rte_mbuf::ol_flags.
+ */
+#define TX_MARK_VLAN_DEI	(1ULL << 0)
+#define TX_MARK_IP_DSCP	(1ULL << 1)
+#define TX_MARK_IP_ECN		(1ULL << 2)
+
 /**
  * The generic rte_mbuf, containing a packet mbuf.
  */

@@ -543,6 +552,10 @@ struct rte_mbuf {
 			};
 			uint32_t inner_l4_type:4; /**< Inner L4 type. */
 		};
+		struct {
+			uint32_t reserved:29;
+			uint32_t tx_mark:3;
+		};
 	};

Please correct me if this is not what you mean.

I'm not a big fan of reusing Rx fields or flags for Tx.
It's not obvious for an application than adding a tx_mark will overwrite
the packet_type. I understand that the risk is limited because packet_type
is Rx and the marks are Tx, but there is still one.

I'm also not a big fan but just wanted to take this approach so that,
it can both conserve space and also help fast path.
Reusing Rx area is however not a new thing as is already followed for 
mbuf->txadapter field.

Apart from documentation issue, Is there any other issue or future 
ramification with using Rx field's for Tx ?
If it is only about documentation, then we can add more documentation to make things clear.

To summarize the different proposed approaches (please correct me if I'm wrong):

a- add 3 Tx mbuf flags
   (-) consumes limited resource

b- add 3 dynamic flags
   (-) slower

- Tx burst Vector implementation can't be done for this tx offload as
  offset keeps changing.

c- add 1 Tx flag and union with Rx field
   (-) exclusive with Rx field
   (-) still consumes one flag

My preference is still b-, for these reasons:

- There are many different DPDK use cases, and resources in mbuf is tight.
  Recent contributions (rte_flow and ice driver) already made use of dynamic
  fields/flags.

- Since RTE_FLOW metadata is 32-bit field, it is a clear candidate for
dynamic flags. 
- ICE PMD's dynamic field is however a vendor specific field and only for
ICE PMD users.

In this case, it is just 1 bit out of 18 free bits available in ol_flags.

- When I implemented the dynamic fields/flags feature, I did a test which
  showed that the cost of having a dynamic offset was few cycles (on my test
  platform, it was~3 cycles for reading a field and ~2 cycles for writing a
  field).

I think this cost is of the case where the address where the dyn_offset is 
stored is already in cache as it needs to be read first.

Regards,
Olivier

quoted

quoted

quoted

+ Techboard

There is a related thread going on
https://urldefense.proofpoint.com/v2/url?u=http-3A__mails.dpdk.org_archives_dev_2020-2DMay_168810.html&d=DwIGaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=FZ_tPCbgFOh18zwRPO9H0yDx8VW38vuapifdDfc8SFQ&m=nyV4Rud03HW6DbWMpyvOCulQNkagmfo0wKtrwQ7zmmg&s=VuktoUb_xoLsHKdB9mV87x67cP9tXk3DqVXptt9nF_s&e= 

If there is no consensus on email, then I would like to add this item
to the next TB meeting.

Ok, I'll add that to tomorrow meeting agenda.
Konstantin