Re: [PATCH 0/2] dpdk: Allow for dynamic enablement of some isolated features

On Fri, Aug 01, 2014 at 02:08:22PM -0700, Bruce Richardson wrote:

On Fri, Aug 01, 2014 at 04:43:52PM -0400, Neil Horman wrote:

quoted

On Fri, Aug 01, 2014 at 12:22:22PM -0700, Bruce Richardson wrote:

quoted

On Fri, Aug 01, 2014 at 11:06:29AM -0400, Neil Horman wrote:

quoted

On Thu, Jul 31, 2014 at 01:25:06PM -0700, Bruce Richardson wrote:

quoted

On Thu, Jul 31, 2014 at 04:10:18PM -0400, Neil Horman wrote:

quoted

On Thu, Jul 31, 2014 at 11:36:32AM -0700, Bruce Richardson wrote:

quoted

Thu, Jul 31, 2014 at 02:10:32PM -0400, Neil Horman wrote:

quoted

On Thu, Jul 31, 2014 at 10:32:28AM -0400, Neil Horman wrote:

quoted

On Thu, Jul 31, 2014 at 03:26:45PM +0200, Thomas Monjalon wrote:

quoted

2014-07-31 09:13, Neil Horman:

quoted

On Wed, Jul 30, 2014 at 02:09:20PM -0700, Bruce Richardson wrote:

quoted

On Wed, Jul 30, 2014 at 03:28:44PM -0400, Neil Horman wrote:

quoted

On Wed, Jul 30, 2014 at 11:59:03AM -0700, Bruce Richardson wrote:

quoted

On Tue, Jul 29, 2014 at 04:24:24PM -0400, Neil Horman wrote:

quoted

Hey all-

With regards to the general approach for runtime detection of software
functions, I wonder if something like this can be handled by the
packaging system? Is it possible to ship out a set of shared libs
compiled up for different instruction sets, and then at rpm install
time, symlink the appropriate library? This would push the whole issue
of detection of code paths outside of code, work across all our
libraries and ensure each user got the best performance they could get
form a binary?
Has something like this been done before? The building of all the
libraries could be scripted easy enough, just do multiple builds using
different EXTRA_CFLAGS each time, and move and rename the .so's after
each run.

Sorry, I missed this in my last reply.

In answer to your question, the short version is that such a thing is roughly
possible from a packaging standpoint, but completely unworkable from a
distribution standpoint.  We could certainly build the dpdk multiple times and
rename all the shared objects to some variant name representative of the
optimzations we build in for certain cpu flags, but then we woudl be shipping X
versions of the dpdk, and any appilcation (say OVS that made use of the dpdk
would need to provide a version linked against each variant to be useful when
making a product, and each end user would need to manually select (or run a
script to select) which variant is most optimized for the system at hand.  Its
just not a reasonable way to package a library.

Sorry, perhaps I was not clear, having the user have to select the
appropriate library was not what I was suggesting. Instead, I was
suggesting that the rpm install "librte_pmd_ixgbe.so.generic",
"librte_pmd_ixgbe.so.sse42" and "librte_pmd_ixgbe.so.avx". Then the rpm
post-install script would look at the cpuflags in cpuinfo and then
symlink librte_pmd_ixgbe.so to the best-match version. That way the user
only has to link against "librte_pmd_ixgbe.so" and depending on the
system its run on, the loader will automatically resolve the symbols
from the appropriate instruction-set specific .so file.

This is an absolute packaging nightmare, it will potentially break all sorts of
corner cases, and support processes.  To cite a few examples:

1) Upgrade support - What if the minimum cpu requirements for dpdk are advanced
at some point in the future?  The above strategy has no way to know that a given
update has more advanced requirements than a previous update, and when the
update is installed, the previously linked library for the old base will
dissappear, leaving broken applications behind.

Firstly, I didn't know we could actually specify minimum cpu
requirements for packaging, that is something that could be useful :-)

You misread my comment :).  I didn't say we could specify minimum cpu
requirements at packaging (you can't, beyond general arch), I said "what if the
dpdk's cpu requriements were raised?".  Completely different thing.  Currently
teh default, lowest common denominator system that dpdk appears to build for is
core2 (as listed in the old default config).  What if at some point you raise
those requirements and decide that SSE4.2 really is required to achieve maximum
performance. Using the above strategy any system that doesn't meet the new
requirements will silently break on such an update.  Thats not acceptable.

Core2 was the first Intel set of chips that had x86_64 instruction set
before (Core microarchitecture), so that's why it's listed as the
minimum - it's the same thing as generic x86_64 support. :-)

Yes, and that matches with the base cpu that Fedora supports for x86_64.

quoted

quoted

Secondly, what is the normal case for handling something like this,
where an upgrade has enhanced requirements compared to the previous
version? Presumably you either need to prevent the upgrade from
happening or else accept a broken app. Can the same mechanism not also
be used to prevent upgrades using a multi-lib scheme?

The case for handling something like this is: Don't do it.  When you package
something for Fedora (or any distro), you provide an implicit guaratee that it
will run (or fail gracefully) on all supported systems.  You can add support for
systems as you go forward, but you can't deprecate support for systems within a
major release.  That is to say, if something runs on F20 now, its got to keep
running on F20 for the lifetime of F20.  If it stops running, thats a
regression, the user opens a bug and you fix it.

The DPDK is way off the reservation in regards to this.  Application packages,
as a general rule don't build with specific cpu features in mind, because
performance, while important isn't on the same scale as what you're trying to do
in the dpdk.  A process getting scheduled off the cpu while we handle an
interrupt wipes out any speedup gains made by any micro-optimizations, so theres
no point in doing so.  The DPDK is different, I understand that, but the
drawback is that it (the DPDK) needs to make optimizations that really aren't
considered particularly important to the rest of user space.  I'm trying to
opportunistically make the DPDK as fast as possible, but I need to do it in a
single binary, that works on a lowest common demoninator system.

quoted

quoted

2) Debugging - Its going to be near impossible to support an application built
with a package put together this way, because you'll never be sure as to which
version of the library was running when the crash occured.  You can figure it
out for certain, but for support/development people to need to remember to
figure this out is going to be a major turn off for them, and the result will be
that they simply won't use the dpdk.  Its Anathema to the expectations of linux
user space.

Sorry, I just don't see this as being any harder to support than
multiple code paths for the same functionality. In fact, it will surely make
debugging easier, since you only have the one code path, just compiled
up in different ways.

Well, then by all means, become a Fedora packager, an you can take over the DPDK
maintenece there :).  Until then, you'll just have to trust me.  If you have
multiple optional code paths (especialy if they're limited to isolated features) 
its manageable.  But regardless of how you look at it, building the same
source multiple times with different cpu support means completely different
binaries.  The assembly and optimization are just plain different.  They may be
close, but they're not the same, and they need to be QA-ed independently.  With
a single build and optional code paths, all the common code is executed no
matter what system you're running on, and its always the same. Multiple builds
with different instruction support means that code that is identical at a source
level may well be significantly different at a binary level, and thats not
something I can sanely manage in a general purpose environment.

quoted

quoted

3) QA - Building multiple versions of a library means needing to QA multiple
versions of a library.  If you have to have 4 builds to support different levels
of optimization, you've created a 4x increase in the amount of testing you need
to do to ensure consistent behavior.  You need to be aware of how many different
builds are available in the single rpm at all times, and find systems on which
to QA which will ensure that all of the builds get tested (as they are in fact,
unique builds).  While you may not hit all code paths in a single build, you
will at least test all the common paths.

Again, the exact same QA conditions will also apply to an approach using
multiple code paths bundled into the same library. Given a choice
between one code path with multiple compiles, vs multiple code paths
each compiled only once, the multiple code paths option leaves far
greater scope for bugs, and when bugs do occur means that you always
have to find out what specific hardware it was being run on. Using the
exact same code multiply compiled, the vast, vast majority of bugs are
going to occur across all platforms and systems so you should rarely
need to ask what the specific platform being used is.

No, they won't (see above).  Enabling insructions will enable the compiler to
emit and optimize common paths differently, so identical source code will lead
to different binary code.  I need to have a single binary so that I know what
I'm working with when someone opens a bug.  I don't have that using a multiple
binary approach.  At least with multiple runtime paths (especially/specifically
with the run time paths we've been discussing, the igbe rx vector path and the
acl library, which are isolated), I know that, if I get a bug report and the
backtrace ends in either location, I know I'm sepecifically dealing with that
code.  With your multiple binary approach, if I get a crash in, say
rte_eal_init, I need to figure out if this crash happened in the sse3 compiled
binary, the ss4.2 compiled binary, the avx binary, the avx512 binary, or the
core2 binary.  You can say thats easy, but its easy to say that when you're not
the one that has to support it.

quoted

quoted

The bottom line is that Distribution packaging is all about consistency and
commonality.  If you install something for an arch on multiple systems, its the
same thing on each system, and it works in the same way, all the time.  This
strategy breaks that.  Thats why we do run time checks for things.

If you want to have the best tuned code running for each instruction
set, then commonality and consistency goes out the window anyway,

So, this is perhaps where communication is breaking down.  I don't want to have the
best tuned code running for each instruction set.  What I want is for the dpdk
to run on a lowest common denominator platform, and be able to opportunistically
take advantage of accelerated code paths that require advanced cpu featrues.


Lets take the ixgbe code as an example. Note I didn't add any code paths there,
at all (in fact I didn't add any anywhere).  The ixgbe rx_burst method gets set
according to compile time configuration.  You can pick the bulk_alloc rx method,
or the vectorized rx method at compile time (or some others I think, but thats
not relevant). As it happened the vectorized rx path option had an implicit
dependency on SSE4.2.  Instead of requiring that all cpus that run the dpdk have
SSE4.2, I instead chose to move that compile time decision to a run tmie
decision, by building only the vectorized path with sse4.2 and only using it if
we see that the cpu supports sse4.2 at run time.  No new paths created, no new
support requirements, you're still supporting the same options upstream, the
only difference is I was able to include them both in a single binary.  Thats
better for our end users because the single binary still works everywhere.
Thats better for our QA group because For whatever set of tests they perform,
they only need an sse4.2 enabled system to test the one isolated path for that
vector rx code.  The rest of their tests can be conducted once, on any system,
because the binary is exactly the same.  If we compile multiple binaries,
testing on one system doesn't mean we've tested all the code.

quoted

because two different machines calling the same function are going to
execute different sets of instructions. The decision then becomes:

But thats not at all what I wanted.  I want two different machines calling the
same function to execute the same instructions 99.9999% of the time.  The only
time I want to diverge from that is in isolated paths where we can take
advantage of a feature that we otherwise could not (i.e. the ixgbe and acl
code).  I look at it like the alternatives code in linux.  There are these
isolated areas where you have limited bits of code that at run time are
re-written to use available cpu features.  99.9% of the code is identical, but
in these little spots its ok to diverge from simmilarity because they'er isolated,
easily identifiable

quoted

a) whether you need multiple sets of instructions - if no then you pay
with lack of performance
b) how you get those multiple sets of instructions
c) how you validate those multiple sets of instructions.

As is clear by now :-), my preference by far is to have multiple sets of
instructions come from a single code base, as less code means less
maintenance, and above all, fewer bugs. If that can't be done, then we
need to look carefully at each code path being added and do a
cost-benefit analysis on it.

Yes, its quite clear :), I think its equally clear that I need a single binary,
and would like to opportunisitcally enhance it where possible without losing the
fact that its a single binary.

I suppose its all somewhat moot at this point though,  The reduction to sse3 for
ixgbe seems agreeable to everyone, and it lets me preserve single binary builds
there.  I'm currently working on the ACL library, as you noted thats a tougher
nut to crack.  I think I'll have it done early next week (though i'm sure my
translation of the instruction set reference to C will need some through testing
:)).  I'll post it when its ready.

Agreed, lets get everything working to a common baseline anyway. In
terms of the number of RX and TX functions, you mentioned, I'd hope in future 
we could cut the number of them down a bit as we make the vector versions more
generally applicable, but that's a whole discussion for another day.

As long as all paths can continue to work with a generic x86_64 compilation, I'm
good with that.  I'd still like to see some options be run time, rather than
compile time selectable, but we can deal with that much later.

Best
Neil

/Bruce