Re: [PATCH 0/8] Makefile: make command-list.h 2-5x as fast with -jN

On Wed, Oct 20 2021, Taylor Blau wrote:

On Thu, Oct 21, 2021 at 02:48:24AM +0200, Ævar Arnfjörð Bjarmason wrote:

quoted

quoted

quoted

Per Eric's Sunshine's upthread comments an awk and Perl implementation
were both considered before[1].

Ah sorry, I thought it was just a perl one that had been the
show-stopper. I hadn't noticed the awk one. However, the point of my
patch was to use perl if available, and fall back otherwise. Maybe
that's too ugly, but it does address the concern with Eric's
implementation.

I think carrying two implementations is worse than just having the one
slightly slower one.

I have no opinion on whether or not assuming that awk or Perl exists and
can be relied upon during the build is reasonable or not. It seems like
the former might be a slightly safer assumption than the latter, but in
all honesty it seems like both are always likely to be around.

In any case, I think the point was that we could improve upon Peff's
patch by just having a single implementation done in awk. And when I
wrote that I definitely was in the mindset of being able to rely on awk
during compilation.

quoted

quoted

quoted

I.e. I think if you e.g. touch Documentation/git-a*.txt with this series
with/without this awk version the difference in runtime is within the
error bars. I.e. making the loop faster isn't necessary. It's better to
get to a point where make can save you from doing all/most of the work
by checking modification times, rather than making an O(n) loop faster.

FWIW, I don't agree with this paragraph at all. Parallelizing or reusing
partial results is IMHO inferior to just making things faster.

I agree with you in the general case, but for something that's consumed
by a make dependency graph I find it easier to debug things if
e.g. changing git-add.txt results in a change to git-add.gen, which is
then cat'd together.

IOW if we had a sufficiently fast C compiler I think I'd still prefer
make's existing rules over some equivalent of:

    cat *.c | super-fast-cc

Since similar to how the *.sp files depend on the the *.o files now,
declaring the dependency graph allows you to easily add more built
things.

This seems like an unfair comparison to me. I might be more sympathetic
if we were generating a more complicated artifact by running
generate-cmdlist.sh, but its inputs and outputs seem very well defined
(and non-complicated) to me.

They are? a foo.o to foo.o input is relatively uncomplicated, and you
can discover the exact dependencies with 3rd party tools, like the GCC
and Clang switches we use generate the .depends dirs[1].

Whereas with the custom shellscripts that have for-loops of their own
like generate-cmdlist.sh what it depends on exactly is relatively opaque
to you until you read the shellscript.

I guess it's a matter of taste, but if you run this with/without this
series:

    touch Documentation/git-a*.txt; time make -j1 command-list.h --debug=b V=1

You'll see that before we'd spot that e.g. git-add.txt changed, but
we'll run one target in response to that at the end.

So it's just like what you'd get when you make %.o from %.c to produce a
"program" that links all those %.o together at the end.

So I do think it's a fair comparison, if anything it's unfair to this
series, because as noted you can discover these dependency independently
with GCC etc. for C code. But for a custom *.txt format with an ad-hoc
*.sh to parse it there's no such aid available.

1. $ cat .depend/help.o.d 
help.o: help.c cache.h git-compat-util.h compat/bswap.h wildmatch.h \
 banned.h strbuf.h hashmap.h hash.h repository.h path.h sha1dc_git.h \
 sha1collisiondetection/lib/sha1.h sha256/block/sha256.h list.h advice.h \
 gettext.h convert.h string-list.h trace.h trace2.h pack-revindex.h \
 oid-array.h mem-pool.h config.h builtin.h commit.h object.h tree.h \
 decorate.h gpg-interface.h pretty.h commit-slab.h commit-slab-decl.h \
 commit-slab-impl.h exec-cmd.h run-command.h thread-utils.h strvec.h \
 levenshtein.h help.h command-list.h column.h version.h refs.h \
 parse-options.h prompt.h
[...]