Re: Further fun with ABI tracking

On Tue, Feb 14, 2017 at 11:52 AM, Christian Ehrhardt
[off-list ref] wrote:

Hi,
when moving to DPDK 16.11 Debian/Ubuntu packaging of DPDK has hit a new
twist on the (it seems reoccurring) topic of DPDK ABI tracking.

I have found, ... well I don't want to call it solution ..., let's say a
crutch to get around it for the moment. But I wanted to use the example I
had to share a few thoughts on it and to kick off a wider discussion.


*## In library cross-dependencies plus partial ABI bumps ##*

Since the day moving away from the combined shared library we had several
improvements on tracking the ABI versions. These days [1] we have LIBABIVER
per library and it gets bumped to reflect it is breaking with former
versions e.g. removing symbols.

Now in the 16.11 release the ABIs for cryptodev, eal and ethdev got bumped
by [2] and [3].

OTOH please remember that in general two versions of a shared library in
the usual sense are meant to be able to stay alongside on a system without
hurting each other. I picked a random one on my system.
Package              Library
libisc-export160: /lib/x86_64-linux-gnu/libisc-export.so.160
libisc-export160: /lib/x86_64-linux-gnu/libisc-export.so.160.0.0
libisc-export95: /lib/x86_64-linux-gnu/libisc-export.so.95
libisc-export95: /lib/x86_64-linux-gnu/libisc-export.so.95.5.0
Some link against the new, some against the old library - all fine.
Usually most programs can just be rebuilt against the new library and after
some time the old one can be dropped. That mechanism gives downstream
distributions a way to handle transitions and consumers of libraries which
might not all be ready for the same version every time.
And since the per lib versioning with LIBABIVER and and the version maps we
are good - in fact we qualify for all common cases on [4].

Now in DPDK of those libraries that got an ABI bump eal and ethdev are part
of those which most of us consider "core libraries" and most other libs and
pmds link to them.
And here DPDK continues to be special, due to that inter-dependency with
old and new libraries installed on the same system the following happens on
openvswitch built for an older version of dpdk:
ovs-vswitchd-dpdk
    librte_eal.so.2 => /usr/lib/x86_64-linux-gnu/librte_eal.so.2
    librte_pdump.so.1 => /usr/lib/x86_64-linux-gnu/librte_pdump.so.1
        librte_eal.so.3 => /usr/lib/x86_64-linux-gnu/librte_eal.so.3

You can see that Openvswitch itself depends on the "old" librte_eal.so.2.
But because  librte_pdump.so.1 did not get an ABI bump it got upgraded to
the newer version from DPDK 16.11.
But since the "new" pdump got built with the new DPDK 16.11 it depends on
the "new" librte_eal.so.3.
And having both in the same executable space at the same time causes
segfaults and pain.

As I said for now I have passed the issue with a crutch that I'm not proud
of and I'd like to avoid in the future. For that I'm reaching out to you
with several suggestions to discuss.


*## Thoughts ##*
None of these seems like a perfect solution to me yet, but clearly good to
start discussions on them.

Options that were in discussion so far and that we might adopt next cycle
(some of these are upstream changes, some downstream, some require both to
change - but any of them should have an ack upstream so that we are
agreeing how to proceed with those cases).

1. Downstreams to insert Major version into soname
Distributions could insert the DPDK major version (like 16.11) into the
soname and package names. A common example of this is libboost [5].
That would perfectly allow 16.07.<LIBABIVER> to coexist with
16.11.<LIBABIVER> even if for a given library LIBABIVER did not change.
Yet it would mean that anything depending on the old library will have to
be recompiled to pick up the new code, even if it depends on an ABI that is
still present in the new release.
Also - not a technical reason - but it is clearly more work to force update
all dependencies and clean out old packages for every release.

Actually this isn't exactly what I proposed during the summit. Just
keep it simple and fix the ABI version of all libraries at 16.11.0.
This is a proven approach and has been used for years with different
libraries. You could easily do this independently of us upstream
fixing the ABI problems.

2. ABI Ranges

ABI is either backwards compatible (same major) or not. A range
doesn't solve the problem.

3. A lot of conflicts

This doesn't allow us to have multiple version of the library
available at runtime. So in the end it doesn't solve the problem for
the distro either.

4. ABI bump is infecting

5. back to single ABI

This is very similar to approach 1. It just uses up a lot more ABI versions.

6. More
I'm sure there are more approaches to this, feel free to come up with more.

The problem is that we do not detect and fix the ABI changes that
"shine-through" the dependencies of our libraries. We need to work on
them and fix them one by one. Long-term we need to invest into keeping
the API/ABI stable and adding backward compatible symbols as well as
making structures opaque.

I'm sure my five suggestions alone will make the thread messy, Maybe we do
this in two rounds, sorting out the insane and identifying the preferred
ones to then in a second run focus on discussing and maybe implementing the
details of what we like.


[1]: http://dpdk.org/browse/dpdk/tree/doc/guides/contributing/versioning.rst
[2]: http://dpdk.org/browse/dpdk/commit/?id=d7e61ad3ae36
[3]: http://dpdk.org/browse/dpdk/commit/?id=6ba1affa54108
[4]: https://wiki.debian.org/TransitionBestPractices
[5]: https://packages.debian.org/sid/libboost1.62-dev
[6]:
https://www.debian.org/doc/debian-policy/ch-relationships.html#s-conflicts
[7]: https://wiki.ubuntu.com/ProposedMigration

P.S. I beg a pardon for the wall of text

--
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd