Re: [dpdk-dev] [PATCH v13 09/10] eal: add EAL argument for setting thread priority

On Thu, 19 Aug 2021 14:30:19 -0700
Narcisa Ana Maria Vasile [off-list ref] wrote:

On Thu, Aug 19, 2021 at 10:06:06AM +0100, Bruce Richardson wrote:

quoted

On Wed, Aug 18, 2021 at 02:28:33PM -0700, Stephen Hemminger wrote:  

quoted

On Tue,  3 Aug 2021 12:01:30 -0700 Narcisa Ana Maria Vasile
[off-list ref] wrote:
  

quoted

+static int +eal_parse_thread_priority(const char *arg) +{ +
struct internal_config *internal_conf = +
eal_get_internal_configuration(); +	enum rte_thread_priority priority;
+ +	if (!strncmp("normal", arg, sizeof("normal"))) +
priority = RTE_THREAD_PRIORITY_NORMAL; +	else if
(!strncmp("realtime", arg, sizeof("realtime"))) +		priority =
RTE_THREAD_PRIORITY_REALTIME_CRITICAL; +	else +		return -1;
+ +	internal_conf->thread_priority = priority; +	return 0; +} +  

In my experience using real time priority is dangerous and risks
starvation and deadlock. The problem is that DPDK applications are
typically 100% CPU poll mode with no system calls; but the kernel has a
number of worker threads that can be required on those CPUs.

The typical failure is a disk completion interrupt happens on a CPU that
is being used by DPDK lcore thread. With RT priority, the kernel thread
to process that I/O completion never runs because the RT user thread has
higher priority than the kernel I/O completion thread.

It maybe possible to workaround this with lots of hand crafting through
sysfs to reassign threads and irq's. Also, later kernels with full RT
might also help.

Before putting this in as part of DPDK in EAL, a full set of testing and
documentation of how to configure these kind of applications and systems
is needed.
 

I would tend to agree caution here, based on my experience of having locked
up a number of systems in the past when testing running DPDK apps with RT
priority!  

Thank you for the comments! I've added this option since it was requested by
multiple users. I understand RT priority causes issues on Linux platforms.
On Windows we want to be able to use REALTIME priority in certain scenarios.

Would it be acceptable to replace this option with a "HIGH_PRIORITY" one
and keep it realtime on Windows and choose a higher (but non-realtime) option on Linux?
However, there are 2 issues here:
 * We will have different behaviors between the 2 platforms.
 * Not sure if I can set a normal but higher priority on Linux. SCHED_OTHER only allows
   one value and Linux "nice" values don't help. If anyone knows of a way to accomplish
   this on Linux, please do advise.
Alternatively, we can have this option for Windows only.

In the meantime, I've removed this patch from this patchset in v14 as the cmdline option is not
being enabled yet, as DmitryK noted.

I think on Linux it should produce a big a*** warning message.