-----邮件原件-----
发件人: Feifei Wang
发送时间: 2021年1月14日 14:24
收件人: Pavan Nikhilesh Bhagavatula [off-list ref]; Harry
van Haaren [off-list ref]; Nikhil Rao
[off-list ref]; Pavan Nikhilesh
[off-list ref]
抄送: dev@dpdk.org; jerinj@marvell.com; nd [off-list ref];
stable@dpdk.org; Ruifeng Wang [off-list ref]; Honnappa
Nagarahalli [off-list ref]; nd [off-list ref]
主题: 回复: [EXT] [PATCH v3] examples/eventdev: refactor ethdev port stop

quoted

-----邮件原件-----
发件人: Pavan Nikhilesh Bhagavatula [off-list ref]
发送时间: 2021年1月5日 18:09
收件人: Feifei Wang [off-list ref]; Harry van Haaren
[off-list ref]; Nikhil Rao [off-list ref]; Pavan
Nikhilesh [off-list ref]
抄送: dev@dpdk.org; jerinj@marvell.com; nd [off-list ref];
stable@dpdk.org; Ruifeng Wang [off-list ref]; Honnappa
Nagarahalli [off-list ref]
主题: RE: [EXT] [PATCH v3] examples/eventdev: refactor ethdev port stop

Hi, Pavan

quoted

Hi Feifei,

quoted

Move eth stop code from "signal_handler" function to the end of "main"
function. There are two reasons for this:

First, this improves code maintenance and makes code look simple and
clear. Based on this change, after receiving the interrupt signal,
"fdata->done" is set as 1. Then the main thread will wait all worker
lcores to jump out of the loop. Finally, the main thread will stop
and then close eth dev port.

Second, for older version, the main thread first stops eth dev port
and then waits the end of worker lcore. This may cause errors because
it may stop the eth dev port which worker lcores are using. This
moving change can fix this by waiting all worker threads to exit and
then stop the eth dev port.

Apologies for the delayed reply,

In case of event dev the workers don't interact with eth device
directly, Instead eth device "injects" packets into event device and
event device is responsible for scheduling them to the workers.

If the producer is not stopped i.e. in this case eth device then the
worker threads might never exit and the main core would wait
indefinitely for workers to exit. This will be predominantly seen in
cases where there are only a few flows and large number of workers
causing a lot of intra thread dependency.

For the case that the event device scheduling packets to the workers.
Though the producer is not stopped (eth device), when the main core
receive the interrupt signal, the "fdata->done"
will be set as 1. Then all the workers load the value 1 of "fdata->done" and
jump out of the loop to finish their thread.

And I also test the case you said above with this patch, when I send interrupt signal(ctrl + c), IXIA
is running and eth device is also working, then the program can exit normally.

HW:
Traffic generator: IXIA 
Nics: ixgbe 82599ES 10-Gigabit
Architecture: aarch64
CPU: Cortex-A72

With this patch:
______________________________________________________________________________________________
Command: sudo ./dpdk-eventdev_pipeline -l 0-2,8-11 --vdev=event_sw0 -- -r2 -t2 -e4 -w F00 -s4 -n0 -c32 -W1000 -D
.........
Config:
        ports: 2
        workers: 4
        packets: 9223372036854775807
        Queue-prio: 0
        qid0 type: atomic
        Cores available: 7
        Cores used: 6
        Eventdev 0: event_sw
  Stages:
        Stage 0, Type Atomic    Priority = 128
        Stage 1, Type Atomic    Priority = 128
        Stage 2, Type Atomic    Priority = 128
        Stage 3, Type Atomic    Priority = 128

Port 0 modified RSS hash function based on hardware support,requested:0x3afbc configured:0x38d34
Port 0 MAC: 90 e2 ba 56 ed 6c
Port 1 modified RSS hash function based on hardware support,requested:0x3afbc configured:0x38d34
Port 1 MAC: 90 e2 ba 56 ed 6d
[dump_core_info()] lcore 1 executing NIC Rx
[dump_core_info()] lcore 1 executing NIC Tx
[dump_core_info()] lcore 2 executing scheduler
[dump_core_info()] lcore 8 executing worker, using eventdev port 0
[dump_core_info()] lcore 9 executing worker, using eventdev port 1
[dump_core_info()] lcore 10 executing worker, using eventdev port 2
[dump_core_info()] lcore 11 executing worker, using eventdev port 3


Command: ctrl + c
.........
  worker 1 thread done. RX=0 TX=0
  worker 2 thread done. RX=0 TX=0
  worker 11 thread done. RX=1844023 TX=1844023
  worker 9 thread done. RX=1844888 TX=1844888
  worker 8 thread done. RX=1845490 TX=1845490
  worker 10 thread done. RX=1844903 TX=1844903

Port Workload distribution:
worker 0 :      25.0 % (1845474 pkts)
worker 1 :      25.0 % (1844872 pkts)
worker 2 :      25.0 % (1844887 pkts)
worker 3 :      25.0 % (1844007 pkts)
Invalid port_id=0
Invalid port_id=1
______________________________________________________________________________________________

Best Regards
Feifei

quoted

Regards,
Pavan.

quoted

In the meanwhile, remove wmb in signal_handler.

This is because when the main lcore receive the stop signal, it
stores
1 into fdata->done. And then the worker lcores load "fdata->done" and
jump out of the loop to stop running. Nothing should be stored after
updating
fdata->done, so the wmb is unnecessary.

Fixes: 085edac2ca38 ("examples/eventdev_pipeline: support Tx
adapter")
Cc: pbhagavatula@marvell.com
Cc: stable@dpdk.org

Suggested-by: Ruifeng Wang <redacted>
Signed-off-by: Feifei Wang <redacted>
Reviewed-by: Ruifeng Wang <redacted>
Reviewed-by: Honnappa Nagarahalli
[off-list ref]
Acked-by: Harry van Haaren <redacted>
---

v2:
1. Delete unused variable to fix build error 2. Reduce commit message
length to fix coding style issues

v3:
1. Title improvement (Van Haaren)

examples/eventdev_pipeline/main.c | 16 ++++------------
1 file changed, 4 insertions(+), 12 deletions(-)

diff --git a/examples/eventdev_pipeline/main.c

b/examples/eventdev_pipeline/main.c
index 823f8b51c..fdbaf667b 100644

--- a/examples/eventdev_pipeline/main.c
+++ b/examples/eventdev_pipeline/main.c

@@ -280,7 +280,6 @@ static void

signal_handler(int signum)
{
	static uint8_t once;
-	uint16_t portid;

	if (fdata->done)
		rte_exit(1, "Exiting on signal %d\n", signum); @@ -291,17

+290,6 @@

quoted

signal_handler(int signum)
			rte_event_dev_dump(0, stdout);
		once = 1;
		fdata->done = 1;
-		rte_smp_wmb();
-
-		RTE_ETH_FOREACH_DEV(portid) {
-			rte_event_eth_rx_adapter_stop(portid);
-			rte_event_eth_tx_adapter_stop(portid);
-			if (rte_eth_dev_stop(portid) < 0)
-				printf("Failed to stop port %u", portid);
-		}
-
-		rte_eal_mp_wait_lcore();
-
	}
	if (signum == SIGTSTP)
		rte_event_dev_dump(0, stdout);

@@ -465,6 +453,10 @@ main(int argc, char **argv)

	}

	RTE_ETH_FOREACH_DEV(portid) {
+		rte_event_eth_rx_adapter_stop(portid);
+		rte_event_eth_tx_adapter_stop(portid);
+		if (rte_eth_dev_stop(portid) < 0)
+			printf("Failed to stop port %u", portid);
		rte_eth_dev_close(portid);
	}

--
2.25.1