Re: [RFCv2 00/15] RFCv2: Consolidated userspace RDMA library repo

On Sun, Aug 28, 2016 at 12:36:27PM -0600, Jason Gunthorpe wrote:

On Sun, Aug 28, 2016 at 04:28:04PM +0300, Leon Romanovsky wrote:

quoted

On Fri, Aug 26, 2016 at 11:17:52PM -0400, Doug Ledford wrote:

quoted

On 8/25/2016 4:13 PM, Jason Gunthorpe wrote:

quoted

We've never been able to run the rdma stack from the build trees,

It's possible...you just have to do every step in the right order ;-)  I
had a spreadsheet that I tracked packages, versions, build states,
install states, and dependencies.  Made things a lot easier.

But I get your point.

Sorry, but I missed your point. On my development machine, I'm
running all relevant RDMA stack directly from their git trees
without any excel file. It worked flawlessly without any
obstacles, did I do anything wrong?

Maybe, because it is fairly hard to do correctly.

How do you make sure that eg librdmacm or a provider is built against
the git checkout of verbs, headers, shlib and all?

How do you rebuild the right trees if a change is made to verbs?

How do you have verbs load the provider driver from the git directory build?

When I said 'from the build trees' I mean without a 'make install',
just run 'build/bin/ibvdev_info' for instance and it guarenteed uses
the build products exclusively, including all the libraries and
providers.

Or set LD_LIBRARY_PATH=`pwd`/build/lib and have everything
use the new build.

This is the goal, and the consolidated repo can do everything except
the providers right now.

I'm doing that by running number of one liners [1], it builds me
development setup in shared folder, so my base image is continuing
to be clean.

It is not final version, I didn't upload it yet.

Libraries:
	@echo "Build libibverbs"
	@cd $(LIBIBVERBS_SRC)/; ./autogen.sh; ./configure --prefix=$(KVM_SHARED) CFLAGS=-I$(KVM_SHARED)/include LDFLAGS=-L$(KVM_SHARED)/lib CPPFLAGS=-I$(KVM_SHARED)/include; $(MAKE); $(MAKE) install
	@echo "Build libmlx5"
	@cd $(LIBMLX5_SRC)/; ./autogen.sh; ./configure --prefix=$(KVM_SHARED) CFLAGS=-I$(KVM_SHARED)/include LDFLAGS=-L$(KVM_SHARED)/lib CPPFLAGS=-I$(KVM_SHARED)/include; $(MAKE); $(MAKE) install

Kernel headers:
	@echo "Install kernel headers"
	@make -C $(KERNEL_SRC) headers_install INSTALL_HDR_PATH=$(KVM_SHARED)

Doug is also adressing a larger issue with packaging and getting
everything to build correctly as the distro - he talked about it here:

https://www.spinics.net/lists/linux-rdma/msg37855.html

I read that thread and think that it is related to unoptimized build
process of one distro who for any reason decided do not use repo tool
[2] for managing complex build dependencies between different git trees,
but used excel file and manual download of source tarballs.

We used such tool for our 100+ git trees project and it worked like a charm.
There is worth to invest time and revise internal process of releasing
RDMA stack in that specific distro and no need to invent new beast
(rdma-plumbers).

[1] https://github.com/rleon/dev-scripts/blob/master/Makefile#L89
[2] https://source.android.com/source/developing.html

Jason