Re: [PATCH 0/8] Makefile: make command-list.h 2-5x as fast with -jN

On Wed, Oct 20, 2021 at 04:35:38PM -0400, Jeff King wrote:

On Wed, Oct 20, 2021 at 08:39:51PM +0200, Ævar Arnfjörð Bjarmason wrote:

quoted

This series is based off an off-hand comment I made about making the
cmdlist.sh faster, in the meantime much of the same methods are
already cooking in "next" for the "lint-docs" target.

See 7/8 for the main performance numbers, along the way I stole some
patches from Johannes Sixt who'd worked on optimizing the script
before, which compliment this new method of generating this file by
piggy-backing more on GNU make for managing a dependency graph for us.

I still think this is a much more complicated and error-prone approach
than just making the script faster. I know we can't rely on perl, but
could we use it optimistically?

I'll take credit for this terrible idea of using Perl when available.

But I don't think we even need to, since we could just rely on Awk. That
has all the benefits you described while still avoiding the circular
dependency on libgit.a. But the killer feature is that we don't have to
rely on two implementations, the lesser-used of which is likely to
bitrot over time.

The resulting awk is a little ugly, because of the nested-ness. I'm also
no awk-spert, but I think that something like the below gets the job
done.

It also has the benefit of being slightly faster than the equivalent
Perl implementation, for whatever those extra ~9 ms are worth ;).

Benchmark #1: sh generate-cmdlist.sh command-list.txt
  Time (mean ± σ):      25.3 ms ±   5.3 ms    [User: 31.1 ms, System: 8.3 ms]
  Range (min … max):    15.5 ms …  31.7 ms    95 runs

Benchmark #2: sh generate-cmdlist.sh.old command-list.txt
  Time (mean ± σ):      34.9 ms ±   9.8 ms    [User: 41.0 ms, System: 6.9 ms]
  Range (min … max):    22.4 ms …  54.8 ms    64 runs

Summary
  'sh generate-cmdlist.sh command-list.txt' ran
    1.38 ± 0.49 times faster than 'sh generate-cmdlist.sh.old command-list.txt'

---

diff --git a/generate-cmdlist.sh b/generate-cmdlist.sh
index a1ab2b1f07..39338ef1cc 100755
--- a/generate-cmdlist.sh
+++ b/generate-cmdlist.sh

@@ -64,12 +64,19 @@ print_command_list () {
 	echo "static struct cmdname_help command_list[] = {"

 	command_list "$1" |
-	while read cmd rest
-	do
-		printf "	{ \"$cmd\", $(get_synopsis $cmd), 0"
-		printf " | CAT_%s" $(echo "$rest" | get_category_line)
-		echo " },"
-	done
+	awk '{
+		f="Documentation/" $1 ".txt"
+		while((getline line<f) > 0) {
+			if (match(line, "^" $1 " - ")) {
+				syn=substr(line, RLENGTH+1)
+				printf "\t{ \"%s\", N_(\"%s\"), 0", $1, syn
+				for (i=2; i<=NF; i++) {
+					printf " | CAT_%s", $i
+				}
+				print " },"
+			}
+		}
+	}'
 	echo "};"
 }