As this discussion has broad implications for DPDK, is it a good candidate for a techboard meeting topic? 

-----Original Message-----
From: Burakov, Anatoly
Sent: Monday, July 16, 2018 10:09 AM
To: Eads, Gage <redacted>; dev@dpdk.org
Subject: Re: DPDK and forked processes

On 16-Jul-18 4:00 PM, Eads, Gage wrote:

quoted

Hi all,

Does DPDK support forking secondary processes after executing
rte_eal_init()? The l2fwd_fork example and at least one application
(OpenEM: https://sourceforge.net/projects/eventmachine/) use this
model, and they do so by fixing up the EAL internals (e.g. manually
changing process_type from primary to secondary) at the start of the
child process. This feels like a hack, and I can’t find any
documentation describing this model.

Moreover, this approach doesn’t appear to be compatible with recent
EAL changes. For instance, the multi-process communication creates a
couple handler threads (“rte_mp_handle” and “rte_mp_async”) during EAL
initialization. The child processes won’t inherit these threads, and
so won’t be able to participate in multi-process comms. This means the
reworked memory subsystem and upcoming device hotplug support
(http://mails.dpdk.org/archives/dev/2018-July/107704.html) won’t work
with this fork-after-init model.

This is just one example – there may be other features/subsystems that
won’t work. As far as I can tell there is no official stance (though
the l2fwd_fork example implies it’s supported, IMO); I think either
DPDK should either drop the example and not support this model, or
support it and either document its limitations or resolve them. This
model could be an interesting way to run multi-process DPDK on an
ASLR-enabled system, but supporting this wouldn’t be trivial.

Thanks,

Gage

I think it's a very bad idea to use such a model in recent versions of DPDK. As you
have correctly pointed out, IPC will not work in such a scenario, and given how
our memory subsystem relies on IPC, this is a recipe for memory corruption and
divergent memory maps (since technically both initial and forked processes
believe they are primary).

Even hacking rte_config to make DPDK think it's a secondary process will not
work, because the initialization has already completed, but all of the threads
(IPC, interrupt, etc.) are gone and correct IPC socket was not created, which
means the process becomes invisible to the primary for all intents and purposes.

We _could_ introduce some kind of "official DPDK fork" function that would fork
the process and then restart interrupt, IPC etc. stuff on an already running
instance of DPDK, but that seems like a workaround for a problem that shouldn't
exist in the first place, because such usage is fundamentally incompatible with
DPDK as it stands now.

--
Thanks,
Anatoly