sctp: set ooo_okay properly for Transmit Packet Steering

Unlike tcp_transmit_skb,
sctp_packet_transmit does not set ooo_okay explicitly,
causing unwanted Tx queue switching when multiqueue is in use;
Tx queue switching may cause out-of-order packets.
Change sctp_packet_transmit to allow Tx queue switching only for
the first in flight packet, to avoid unwanted Tx queue switching.

Signed-off-by: Wally Zhao <redacted>
---
 net/sctp/output.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/sctp/output.c b/net/sctp/output.c
index dbda7e7..5ff75cc 100644
--- a/net/sctp/output.c
+++ b/net/sctp/output.c

@@ -626,6 +626,10 @@ int sctp_packet_transmit(struct sctp_packet *packet, gfp_t gfp)
 	/* neighbour should be confirmed on successful transmission or
 	 * positive error
 	 */
+
+	/* allow switch tx queue only for the first in flight pkt */
+	head->ooo_okay = asoc->outqueue.outstanding_bytes == 0;
+
 	if (tp->af_specific->sctp_xmit(head, tp) >= 0 &&
 	    tp->dst_pending_confirm)
 		tp->dst_pending_confirm = 0;

-- 
1.8.3.1

On Wed, Oct 30, 2019 at 12:07:17PM -0400, Wally Zhao wrote:

Unlike tcp_transmit_skb,
sctp_packet_transmit does not set ooo_okay explicitly,
causing unwanted Tx queue switching when multiqueue is in use;

It is initialized to 0 by __alloc_skb() via:
        memset(skb, 0, offsetof(struct sk_buff, tail));
and never set to 1 by anyone for SCTP.

The patch description seems off. I don't see how the unwanted Tx queue
switching can happen. IOW, it's not fixing it OOO packets, but
improving it by allowing switching on the first packet. Am I missing
something?

Considering we are talking about NIC queues here, we would have a
better result with tp->flight_size instead. As in, we can switch
queues if, for this transport, the queue is empty.

+
 	if (tp->af_specific->sctp_xmit(head, tp) >= 0 &&
 	    tp->dst_pending_confirm)
 		tp->dst_pending_confirm = 0;
-- 
1.8.3.1

On 10/30/19, Marcelo Ricardo Leitner [off-list ref] wrote:

On Wed, Oct 30, 2019 at 12:07:17PM -0400, Wally Zhao wrote:

quoted

Unlike tcp_transmit_skb,
sctp_packet_transmit does not set ooo_okay explicitly,
causing unwanted Tx queue switching when multiqueue is in use;

It is initialized to 0 by __alloc_skb() via:
        memset(skb, 0, offsetof(struct sk_buff, tail));
and never set to 1 by anyone for SCTP.

The patch description seems off. I don't see how the unwanted Tx queue
switching can happen. IOW, it's not fixing it OOO packets, but
improving it by allowing switching on the first packet. Am I missing
something?

Thanks for pointing this out. You are right. This ooo_okay is default to false.

I was observing some Tx queue switching before when testing with
iperf3 (modified to be able to set window size, for higher throughput
with long RTT), so I thought ooo_okay was set to true somewhere else
after allocation. Just now I did the test again, it turns out that
iperf3 made a re-connect silently which caused the Tx queue change.

As for the improving purpose of this patch, that is not that critical
from my side, and the patch description is not correct for this
purpose. So I will give up this patch attempt. Thank you again for
your time on this.

quoted

Tx queue switching may cause out-of-order packets.
Change sctp_packet_transmit to allow Tx queue switching only for
the first in flight packet, to avoid unwanted Tx queue switching.

Signed-off-by: Wally Zhao <redacted>
---
 net/sctp/output.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/sctp/output.c b/net/sctp/output.c
index dbda7e7..5ff75cc 100644
--- a/net/sctp/output.c
+++ b/net/sctp/output.c

@@ -626,6 +626,10 @@ int sctp_packet_transmit(struct sctp_packet *packet,

gfp_t gfp)
 	/* neighbour should be confirmed on successful transmission or
 	 * positive error
 	 */
+
+	/* allow switch tx queue only for the first in flight pkt */
+	head->ooo_okay = asoc->outqueue.outstanding_bytes == 0;

Considering we are talking about NIC queues here, we would have a
better result with tp->flight_size instead. As in, we can switch
queues if, for this transport, the queue is empty.

quoted

+
 	if (tp->af_specific->sctp_xmit(head, tp) >= 0 &&
 	    tp->dst_pending_confirm)
 		tp->dst_pending_confirm = 0;
--
1.8.3.1

On Wed, Oct 30, 2019 at 11:54:45PM +0800, Wei Zhao wrote:

On 10/30/19, Marcelo Ricardo Leitner [off-list ref] wrote:

quoted

On Wed, Oct 30, 2019 at 12:07:17PM -0400, Wally Zhao wrote:

quoted

Unlike tcp_transmit_skb,
sctp_packet_transmit does not set ooo_okay explicitly,
causing unwanted Tx queue switching when multiqueue is in use;

It is initialized to 0 by __alloc_skb() via:
        memset(skb, 0, offsetof(struct sk_buff, tail));
and never set to 1 by anyone for SCTP.

The patch description seems off. I don't see how the unwanted Tx queue
switching can happen. IOW, it's not fixing it OOO packets, but
improving it by allowing switching on the first packet. Am I missing
something?

Thanks for pointing this out. You are right. This ooo_okay is default to false.

I was observing some Tx queue switching before when testing with
iperf3 (modified to be able to set window size, for higher throughput
with long RTT), so I thought ooo_okay was set to true somewhere else
after allocation. Just now I did the test again, it turns out that
iperf3 made a re-connect silently which caused the Tx queue change.

Ah, okay.

As for the improving purpose of this patch, that is not that critical
from my side, and the patch description is not correct for this
purpose. So I will give up this patch attempt. Thank you again for
your time on this.

As you wish. If you don't have the time for it, ok, but the
improvement is welcomed. With a more accurate description and using
tp->flight_size instead, it should be good.

Thanks,
  Marcelo

quoted

quoted

Tx queue switching may cause out-of-order packets.
Change sctp_packet_transmit to allow Tx queue switching only for
the first in flight packet, to avoid unwanted Tx queue switching.

Signed-off-by: Wally Zhao <redacted>
---
 net/sctp/output.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/sctp/output.c b/net/sctp/output.c
index dbda7e7..5ff75cc 100644
--- a/net/sctp/output.c
+++ b/net/sctp/output.c

@@ -626,6 +626,10 @@ int sctp_packet_transmit(struct sctp_packet *packet,

gfp_t gfp)
 	/* neighbour should be confirmed on successful transmission or
 	 * positive error
 	 */
+
+	/* allow switch tx queue only for the first in flight pkt */
+	head->ooo_okay = asoc->outqueue.outstanding_bytes == 0;

Considering we are talking about NIC queues here, we would have a
better result with tp->flight_size instead. As in, we can switch
queues if, for this transport, the queue is empty.

quoted

+
 	if (tp->af_specific->sctp_xmit(head, tp) >= 0 &&
 	    tp->dst_pending_confirm)
 		tp->dst_pending_confirm = 0;
--
1.8.3.1

On 10/30/19 9:07 AM, Wally Zhao wrote:

Unlike tcp_transmit_skb,
sctp_packet_transmit does not set ooo_okay explicitly,
causing unwanted Tx queue switching when multiqueue is in use;
Tx queue switching may cause out-of-order packets.
Change sctp_packet_transmit to allow Tx queue switching only for
the first in flight packet, to avoid unwanted Tx queue switching.

While the patch seems fine, the changelog is quite confusing.

When skb->ooo_olay is 0 (which is the default for freshly allocated skbs),
the core networking stack will stick to whatever TX queue was chosen
at the time the dst_entry was attached to the (connected) socket.

This means no reorder can happen at all by default.

By setting ooo_okay carefully (as you did in your patch), you allow
core networking stack to _switch_ to another TX queue based on
current CPU  (XPS selection)

So even without your fix, SCTP should not experience out-of-order packets.

On Thu, Oct 31, 2019 at 3:03 AM Eric Dumazet [off-list ref] wrote:

On 10/30/19 9:07 AM, Wally Zhao wrote:

quoted

Unlike tcp_transmit_skb,
sctp_packet_transmit does not set ooo_okay explicitly,
causing unwanted Tx queue switching when multiqueue is in use;
Tx queue switching may cause out-of-order packets.
Change sctp_packet_transmit to allow Tx queue switching only for
the first in flight packet, to avoid unwanted Tx queue switching.

While the patch seems fine, the changelog is quite confusing.

When skb->ooo_olay is 0 (which is the default for freshly allocated skbs),
the core networking stack will stick to whatever TX queue was chosen
at the time the dst_entry was attached to the (connected) socket.

This means no reorder can happen at all by default.

By setting ooo_okay carefully (as you did in your patch), you allow
core networking stack to _switch_ to another TX queue based on
current CPU  (XPS selection)

So even without your fix, SCTP should not experience out-of-order packets.

Yes, you are right, as Marcelo also pointed out.
The changelog was given based on incorrect observation of a test
result, as I replied to Marcelo.
Since ooo_okay is default to 0, this is good enough; no need for any
patch from my side.
Thank you for your time on this.

quoted

Signed-off-by: Wally Zhao <redacted>
---
 net/sctp/output.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/net/sctp/output.c b/net/sctp/output.c
index dbda7e7..5ff75cc 100644
--- a/net/sctp/output.c
+++ b/net/sctp/output.c

@@ -626,6 +626,10 @@ int sctp_packet_transmit(struct sctp_packet *packet, gfp_t gfp)
      /* neighbour should be confirmed on successful transmission or
       * positive error
       */
+
+     /* allow switch tx queue only for the first in flight pkt */
+     head->ooo_okay = asoc->outqueue.outstanding_bytes == 0;
+
      if (tp->af_specific->sctp_xmit(head, tp) >= 0 &&
          tp->dst_pending_confirm)
              tp->dst_pending_confirm = 0;

FYI, we noticed the following commit (built with gcc-7):

commit: 327fecdaf39ab7163e6d189e19abdadf4555b002 ("[PATCH] sctp: set ooo_okay properly for Transmit Packet Steering")
url: https://github.com/0day-ci/linux/commits/Wally-Zhao/sctp-set-ooo_okay-properly-for-Transmit-Packet-Steering/20191101-171354


in testcase: boot

on test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 8G

caused below changes (please refer to attached dmesg/kmsg for entire log/backtrace):


+-------------------------------------------------------+------------+------------+
|                                                       | 52340b82cf | 327fecdaf3 |
+-------------------------------------------------------+------------+------------+
| boot_successes                                        | 32         | 48         |
| boot_failures                                         | 3          | 28         |
| BUG:kernel_hang_in_test_stage                         | 3          | 14         |
| BUG:kernel_NULL_pointer_dereference,address           | 0          | 14         |
| Oops:#[##]                                            | 0          | 14         |
| RIP:sctp_packet_transmit                              | 0          | 14         |
| Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 0          | 13         |
+-------------------------------------------------------+------------+------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <redacted>


[   35.312661] BUG: kernel NULL pointer dereference, address: 00000000000005d8
[   35.316225] #PF: supervisor read access in kernel mode
[   35.319178] #PF: error_code(0x0000) - not-present page
[   35.322078] PGD 800000021b569067 P4D 800000021b569067 PUD 21b688067 PMD 0 
[   35.325629] Oops: 0000 [#1] SMP PTI
[   35.327965] CPU: 0 PID: 3148 Comm: trinity-c5 Not tainted 5.4.0-rc3-01107-g327fecdaf39ab #12
[   35.332863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
[   35.337932] RIP: 0010:sctp_packet_transmit+0x767/0x822
[   35.340818] Code: 8a 45 40 d0 e8 83 e0 08 88 c2 8a 43 78 83 e0 f7 09 d0 88 43 78 41 f6 45 24 08 74 04 80 4b 7a 20 48 8b 04 24 4c 89 ee 48 89 df <83> b8 d8 05 00 00 00 0f 94 c0 c1 e0 07 88 c2 8a 43 78 83 e0 7f 09
[   35.350270] RSP: 0000:ffffc90000003ac0 EFLAGS: 00010246

93 more lines

On Mon, Nov 04, 2019 at 04:46:35PM +0800, kernel test robot wrote:

[   35.312661] BUG: kernel NULL pointer dereference, address: 00000000000005d8
[   35.316225] #PF: supervisor read access in kernel mode
[   35.319178] #PF: error_code(0x0000) - not-present page
[   35.322078] PGD 800000021b569067 P4D 800000021b569067 PUD 21b688067 PMD 0 
[   35.325629] Oops: 0000 [#1] SMP PTI
[   35.327965] CPU: 0 PID: 3148 Comm: trinity-c5 Not tainted 5.4.0-rc3-01107-g327fecdaf39ab #12
[   35.332863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
[   35.337932] RIP: 0010:sctp_packet_transmit+0x767/0x822

Right, as asoc can be NULL by then. (per the check on it a few lines
before the change here).

  Marcelo

On 11/4/19, Marcelo Ricardo Leitner [off-list ref] wrote:

On Mon, Nov 04, 2019 at 04:46:35PM +0800, kernel test robot wrote:

quoted

[   35.312661] BUG: kernel NULL pointer dereference, address:
00000000000005d8
[   35.316225] #PF: supervisor read access in kernel mode
[   35.319178] #PF: error_code(0x0000) - not-present page
[   35.322078] PGD 800000021b569067 P4D 800000021b569067 PUD 21b688067 PMD
0
[   35.325629] Oops: 0000 [#1] SMP PTI
[   35.327965] CPU: 0 PID: 3148 Comm: trinity-c5 Not tainted
5.4.0-rc3-01107-g327fecdaf39ab #12
[   35.332863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.10.2-1 04/01/2014
[   35.337932] RIP: 0010:sctp_packet_transmit+0x767/0x822

Right, as asoc can be NULL by then. (per the check on it a few lines
before the change here).

Yes, apologize for missing the NULL check (Actually I realized some
further check is need to correctly identify the first in flight
packet, as outstanding_bytes has already been increased by this first
in flight packet itself before getting into sctp_packet_transmit).

Anyway, I think I do not need further action, as the patch is anyway
not going to be merged, the 0day robot picks up the patch from the
mail list directly instead of git repo, right?

Thanks a lot,
Wally

  Marcelo

On Mon, Nov 04, 2019 at 10:14:00PM +0800, Wei Zhao wrote:

On 11/4/19, Marcelo Ricardo Leitner [off-list ref] wrote:

quoted

On Mon, Nov 04, 2019 at 04:46:35PM +0800, kernel test robot wrote:

quoted

[   35.312661] BUG: kernel NULL pointer dereference, address:
00000000000005d8
[   35.316225] #PF: supervisor read access in kernel mode
[   35.319178] #PF: error_code(0x0000) - not-present page
[   35.322078] PGD 800000021b569067 P4D 800000021b569067 PUD 21b688067 PMD
0
[   35.325629] Oops: 0000 [#1] SMP PTI
[   35.327965] CPU: 0 PID: 3148 Comm: trinity-c5 Not tainted
5.4.0-rc3-01107-g327fecdaf39ab #12
[   35.332863] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
1.10.2-1 04/01/2014
[   35.337932] RIP: 0010:sctp_packet_transmit+0x767/0x822

Right, as asoc can be NULL by then. (per the check on it a few lines
before the change here).

Yes, apologize for missing the NULL check (Actually I realized some
further check is need to correctly identify the first in flight
packet, as outstanding_bytes has already been increased by this first
in flight packet itself before getting into sctp_packet_transmit).

Anyway, I think I do not need further action, as the patch is anyway
not going to be merged, the 0day robot picks up the patch from the
mail list directly instead of git repo, right?

That's my understanding as well. I double checked and the patch wasn't
applied by Dave, so we're good.

Thanks,
Marcelo