Re: [PATCH] cmd-list.perl: fix identifying man sections

On Sat, 24 Sept 2022 at 00:07, Jeff King [off-list ref] wrote:

I wondered if we might have caught this in a more automatic way. The
output of:

  cd Documentation
  ./doc-diff cc75e556a9^ cc75e556a9

makes the problem apparent, but I don't fault reviewers for not running
it. I rarely remember to do so. And in general you need a human looking
at doc-diff output to know if it was the intended change or not.

I wondered if it might be worth running

  ./doc-diff v2.37.0 v2.38.0-rc1

near a release to scan over all of the changes. But the diff is over
8000 lines, and I admit my eyes glazed over before I got to the
problematic hunks (even though I knew I was looking for them!).

I know how you felt when you looked at that doc-diff... I was lucky
enough to first look at

  ./doc-diff v2.37.0 v2.38.0-rc0

so that when I then looked at

  ./doc-diff v2.38.0-rc0 v2.38.0-rc1

this stood out quite well.

So I dunno. I think doc-diff is a potentially useful tool, but I'm not
sure how to point the human attention at the right spot to find a bug.

I try to do some such doc-diffing every now and then, and in particular
around rc time. It has caught a few buglets, usually nothing major.

I've also done

  ./doc-diff origin/next origin/seen

at times to catch such things a lot earlier, but it's not often that I
find the time to do so / think about doing it.

Maybe "given enough eyeballs, all bugs are shallow" is our best bet
here. After all, it did find this bug before the release. :)

Yeah, and luck never hurts -- I think it's fair to assume that I would
have missed this bug if it hadn't come in after rc0.

Martin