Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

On 29/08/2021 18:40, Yann E. MORIN wrote:

Thomas, All,

On 2021-08-29 14:51 +0200, Yann E. MORIN spake thusly:
[--SNIP--]

quoted

Finally, as a side note: there is still a case of file overwrite that we
do not detect, even with this series: if two packages that are not part
of the same dependency chain (i.e. they do not depend one on the other),
and they both install the same file, then the file-overwrite will only
happen when we eventually assemble the global target/ and host/ from the
individual PPD target/ and host/ and we still have to add a detection
for that case (which is not a pre-requisite before we apply the current
series, of course).

And of course, this will also cause an overwrite when two such packages
are also dependencies of a third patckage, and so when the PPD of that
third package is prepared, we would have to detect file overwrite.

Note: let's assume the following hypotetical dependency graph (where the
notation A --<-- B means that B depends on A):

      .-<-- A --<-.                 S: skeleton
     /             \                F: target-finalize
    S --<-- B --<-- C
     \               \
      `-<-- D --<-----<-- F

If A and B both install the same file, then we do not have a sane PPD
to build C.

If D and any package in the dependency chain of C (i.e. S, A, B, or C
itself) install the same file, then we can only detect it in
target-finalize.

So, let's take a step back, and have a wider look. I wonder if the end
of a package installation step is the best location to detected file
overwrite...

Indeed, let's assume that we do file overwrite detection:

 1. in PPD preparation (when we rsync the dependencies, in the macro
    per-package-rsync): we can also detect that a package overwrites one
    or more files of another package, report those files, *and* prevent
    building a package with unsound dependencies;

 2. in {target,host}-finalize: we can detect that a package overwrites
    one or more files provided by another package, and report those
    files.

So, the above (still theoretical implementation) is all we would need,
in the end, and we would not need to do the detection at the end of a
package installation step of each package, because it would not provide
any information that we couldn't have with the above.

 Fabrice's patches have shown that the overwritten file detection is too
aggressive in the sense to that it *will* cause (many?) failures in the current
situation.

 So, taking a step further back, what really is the problem we're trying to
solve here? Why can't we overwrite files from other packages?

 I think there are three distinct problems. Let's take Yann's example from above.

1. A file written in A and overwritten in C is only a problem because the
finalize step doesn't take into account dependencies when combining the trees.
We could solve that by taking into the account the dependencies.

2. A file written by both A and B is a problem for package C, because it is not
predictable which version it will see. This problem can be solved by adding a
dependency between A and B.

3. A file written by S and overwritten by A - this, I think, is the big problem.
At first sight it's like case 1 - by taking into account the dependencies, the
finalize stage will see the version from A. However, the problem is for stuff
installed into staging, where B sees the version from S while C sees the version
from A or from S, depending on order of A/B installation. This has two problems:
3a. C's version depends on the order of A/B installation;
3b. B sees a different version of the file than A - this could be a problem for
e.g. header files that affect compilation.

 The original scheme we discussed three or four years ago already took into
account situations 1 and 2, I think, but not yet situation 3. Note that back
then, the overwritten file detection was assumed to happen only in finalize and
accept overwrites in case there's a dependency.

 The curses and crypt issues that Fabrice detected fall in category 3. And
they're extra nasty, because one of the packages involved is the toolchain that
everybody depends on, so you can't just add a dependency to break it.

 I haven't given sufficient thought to it yet, but one idea is to allow packages
to add a post-install hook to another package. That way, ncurses could remove
curses.h from the toolchain, and libxcrypt could remove crypt.h. Still tricky
though because it would mean that any package that uses crypt.h would need to
optionally depend on libxcrypt - so that particular case would probably have to
be solved by making libxcrypt be part of the toolchain... Or maybe making sure
it doesn't overwrite crypt.h and teach its dependencies to include the
libxcrypt-specific crypt.h.

 One thing is certain: it's complicated :-)

 Regards,
 Arnout

However, a benefit of also doing it at the end of the installation step,
of a package, is that we can cut short on the build time. However, I am
not sure that benefit is that important or major...

So, I'd like we think a bit harder before we apply this patch...

Regards,
Yann E. MORIN.

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot