Re: [RFCv2 00/15] RFCv2: Consolidated userspace RDMA library repo

On 8/26/2016 6:27 PM, Jason Gunthorpe wrote:

On Fri, Aug 26, 2016 at 11:42:06AM -0400, Jarod Wilson wrote:

quoted

quoted

Would you help making a spec file that is close to what you
could use in the distro? I don't really care if it is stored in the
package or stored inside RH, but we need to get one made.

Sure. It would probably be something to route through Fedora first though,
and I don't own all the packages this would encompass in Fedora, but could
certainly round a few people up and help out.

Sounds good, here are my thoughts let me know what you think:

1) Compiler flags/etc. I haven't audited all the CentOS rpms, but at
   least verbs overrides the compiler flag to set -fno-strict-aliasing
   Upstream should be distributing the package in a way that works, so
   downstream should not need do this. We need to decide if we want/need to
   force no-strict-aliasing or not, or use some kind of compiler
   test or whatever.

The most recent upstream libibverbs includes -fno-strict-aliasing now so
it can be dropped from the extra stuff in the Fedora spec file.

2) Static libraries. Do we want to continue to provide these? Looking
   at it, I'm skeptical that libibverbs works at all as a static, does
   anyone know? Particularly, how does it work? Should we get rid of
   it or do more work to try and make it work sanely. (eg a
   libibverbs.a that includes all the providers or something)

I'm ready to drop static libraries.

3) Version numbers. What version numbers do we give to the sub
   packages? I guess this is very distro specific.

You can not give different versions to sub-packages.  Or, allow me to be
more precise:  You can't use the Version: tag on sub-packages, it resets
the version of the entire package overall when you do.  The last
Version: tag in the spec file ends up being the version applied to
everything.  This impacts the %{version} macro usage in the spec file.
The only way to put versions on sub-packages is to make it part of the
sub-package's name.

But, really, once the whole schmeal is packaged together, you can't
distribute just an update to a driver, and you can't build just that
updated driver, so I would recommend we just package it all up in the
primary package.

For instance, in order to make certain things work well with the split
package setup, I had to use a fake pseudo require/provide pair in the
drivers and in libibverbs so that installing libibverbs would
automatically pull in the driver libraries too.  If we just package the
driver libraries with the libibverbs library, that's all solved.

4) Package split. I'm suggesting
    libibcm1 -devel -static
    libibumad3 -devel -static
    libibverbs1 -devel -static
    librdmacm1 -devel -static
    rdma-utils (all the other random binaries and stuff)
    ibverbs-plugins (all the providers)

I've done this sort of package split in a single spec file.  Once.  I
undid it and never went back.  I don't suggest going down this road.
Lots of things in spec files that are supposed to "just work" break when
you do this.  The %{version} macro I mentioned above is just one of them.

If you instead do things as multiple spec files, then that would work.
It would be a little funky in terms of our build system to have multiple
packages that use the same source tarball, but it would work.
Essentially, our internal packages are defined on a per spec file basis,
so with multiple spec files, we would be forced to have multiple
distinct package in every way except that they use the same tarball.  It
creates some interesting convolutions to think about how upstream
consolidates and we de-consolidate it right back to what it was ;-)

   So we would get rid of the individual packages for the providers.
   There would be no -devel for the providers and the static providers
   would be bundled with libibverbs1-static

The providers just need to be with the base libibverbs library.  One
doesn't work without the other, the separation serves no purpose.

   IMHO this is more inline with common distro practice for packaging
   plugins. (most providers are about 20k-30k)

Normally I agree...when you have separate plugins and you only install
the ones you need.  But, we only have a few drivers and installing just
the ones you need is more hassle than it's worth.

   I don't want to dictate what distros should do, but this is best
   done with support from the build system upstream, so some guidance
   on what to do is needed to finish the cmake stuff. (see #6)

I'm inclined to wrap it all up into a single package. librdmacm,
libibcm, libibumad, libibverbs, providers, all of it.  I would also be
inclined to contribute the rdma scripts we have as part of the package
too.  I think some of them can use some cleanups if they are going to
become an upstream item instead of a RH specific item, but otherwise
they work well.  If I did that, I would also welcome appropriate similar
scripts for OSes that use different tools than RH OSes.

5) Valgrind - I've noticed CentOS forces valgrind on, and Debian
   leaves it as the upstream default (off)
   IMHO Upstream should shift to use valgrind by default if available
   and encourage all downstreams to compile it with valgrind headers
   present.

Agree.  Changing the default on this when present is good.

6) Actually splitting the file list. cmake has a INSTALL COMPONENT
   facility to help with this, but it needs to be setup upstream.
   Other than the include files and man pages this is straightforward
   to do from the spec file

Not an issue if we package it all up together.

7) NDEBUG - As upstream do we want this set or left unset for a
   release build. It eliminates all asserts and a few other things.
   I'm not sure what is being done today, but the %cmake rpmbuild
   default forces it on. (IMHO, upstream should decide this, not
   downstream?)

I think I would set this.  But I'm flexible.  Basically, I know end
users want Valgrind.  We've had multiple requests to enable it.  They
don't want to install custom packages to get Valgrind support.  But they
need Valgrind to help with debugging their own applications.  So here,
if the NDEBUG being set turns off asserts for internal operations, then
we can set it.  If it turns off asserts for user provided data or user
usage, then we probably ought to leave the asserts enabled.  It all
boils down to whether or not the code eliminated and cycles saved are
internal debugging or a debugging aid for the end user's application.
If it helps them, it needs to stay enabled.  If it verifies our own
operation, it doesn't.

8) COPYING file and other doc stuff. There are 4 different licenses
   being used in the source. Everything is GPLv2 however. I think this
   only impacts the providers.
9) libtool .la files. I preserved them, but if distros don't want them
   I'd love to ditch them.

Drop them.

I'd like to get thing in a state where it is not painful for the
distros to transition - to ease things along.

My approach has been to faithfully preserve exactly what we do today,
and the list above is basicly my list of things that seem like they
need attention.

Jason

-- 
Doug Ledford [off-list ref]
    GPG Key ID: 0E572FDD