Re: [PATCH v4 1/7] service cores: header and implementation

<lots of snips to make responses consumable!>

From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
To: Van Haaren, Harry <redacted>
Cc: dev@dpdk.org; thomas@monjalon.net; Wiles, Keith <redacted>; Richardson,
Bruce [off-list ref]
Subject: Re: [PATCH v4 1/7] service cores: header and implementation

<snip>

Remove above info from the git commit.

Done

Fix the below mentioned documentation warning.

+/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:338:
warning: argument 'enabled' of command @param is not found in the
argument list of rte_service_set_stats_enable(int enable)
+/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:346:
warning: The following parameters of rte_service_set_stats_enable(int
enable) are not documented:
+  parameter 'enable'
+/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:349:
warning: argument 'The' of command @param is not found in the argument
list of rte_service_lcore_list(uint32_t array[], uint32_t n)
+/export/dpdk.org/lib/librte_eal/common/include/rte_service.h:367:
warning: The following parameters of rte_service_lcore_list(uint32_t
array[], uint32_t n) are not documented:
+  parameter 'n'

Done

command to reproduce:
./devtools/test-build.sh -j8 x86_64-native-linuxapp-gcc+shared x86_64-native-linuxapp-
gcc+debug

Thanks - noted.

quoted

+
+/* the internal values of a service core */
+struct core_state {

Change to lcore_state.

Done.

quoted

+
+void rte_service_set_stats_enable(int enabled)

IMO, It should be per service  i.e
rte_service_set_stats_enable(const struct rte_service_spec *spec, int enable)

Improved service library to handle statistics collection on a per service basis.

quoted

+			/* check if this is the only core mapped, else use
+			 * atomic to serialize cores mapped to this service
+			 */
+			uint32_t *lock = (uint32_t *)&s->execute_lock;
+			if ((s->spec.capabilities & RTE_SERVICE_CAP_MT_SAFE) ||
+					(s->num_mapped_cores == 1 ||
+					rte_atomic32_cmpset(lock, 0, 1))) {
+				void *userdata = s->spec.callback_userdata;
+
+				if (cs->collect_statistics) {
+					uint64_t start = rte_rdtsc();
+					s->spec.callback(userdata);
+					uint64_t end = rte_rdtsc();
+					s->cycles_spent += end - start;
+					cs->calls_per_service[i]++;
+					s->calls++;
+				} else
+					s->spec.callback(userdata);
+
+				if ((s->spec.capabilities &
+						RTE_SERVICE_CAP_MT_SAFE) == 0 &&
+						s->num_mapped_cores > 1)

How about computing the non rte_atomic32_cmpset() mode value first and
using in both place i.e here and in the top "if" loop

	const int need_cmpset = (s->spec.capabilities & RTE_SERVICE_CAP_MT_SAFE)...
	if (need_cmpset || rte_atomic32_cmpset(lock, 0, 1))
	..
	if (need_cmpset)
		rte_atomic32_clear()..

Yes good idea. Indeed I wasn't happy with that, this is a good fix.
The checks to detect if we need_cmpset are a little complex, but it's the only solution I see.

I've added another unit test to verify both MT safe and MT unsafe callback operations.

quoted

+int32_t
+rte_service_set_default_mapping(void)
+{
+	/* create a default mapping from cores to services, then start the
+	 * services to make them transparent to unaware applications.
+	 */
+	uint32_t i;
+	int ret;
+	uint32_t count = rte_service_get_count();
+
+	int32_t lcore_iter = 0;
+	uint32_t ids[RTE_MAX_LCORE];
+	int32_t lcore_count = rte_service_lcore_list(ids, RTE_MAX_LCORE);
+
+	for (i = 0; i < count; i++) {
+		struct rte_service_spec *s = rte_service_get_by_id(i);
+		if (!s)
+			return -EINVAL;
+
+		/* if no lcores available as services cores, don't setup map.
+		 * This means app logic must add cores, and setup mappings
+		 */
+		if (lcore_count > 0) {

quoted

+			/* do 1:1 core mapping here, with each service getting
+			 * assigned a single core by default. Adding multiple
+			 * services should multiplex to a single core, or 1:1
+			 * if services == cores
+			 */
+			ret = rte_service_enable_on_lcore(s, ids[lcore_iter]);
+			if (ret)
+				return -ENODEV;
+		}
+
+		lcore_iter++;
+		if (lcore_iter >= lcore_count)
+			lcore_iter = 0;
+
+		ret = rte_service_start(s);

IMO, we don't need to start the service if lcore_count == 0. How about
moving the "if (lcore_count > 0)" check on top of for the loop and exist
from the function if lcore_count == 0.

Good point, added   if() check for lcore_count at start of function, and return if no service cores are available.

quoted

+
+int32_t
+rte_service_lcore_add(uint32_t lcore)
+{
+	if (lcore >= RTE_MAX_LCORE)
+		return -EINVAL;
+	if (cores_state[lcore].is_service_core)
+		return -EALREADY;
+
+	set_lcore_state(lcore, ROLE_SERVICE);
+
+	/* ensure that after adding a core the mask and state are defaults */
+	cores_state[lcore].service_mask = 0;
+	cores_state[lcore].runstate = RUNSTATE_STOPPED;

If worker core can call rte_service_lcore_add() then add rte_smp_wmb()
here. Applies to rte_service_lcore_del() as well.

Added barriers to both add() and del().