On Thu, Jun 25, 2026 at 08:27:35PM -0700, Michael Montalbo wrote:

I think that is the trigger for issues we've been seeing. I spent
some time investigating the Apache side over the last week and maybe
found a mod_http2 bug, which I filed upstream with a potential fix:

  bug:  https://bz.apache.org/bugzilla/show_bug.cgi?id=70131
  fix:  https://github.com/mmontalbo/httpd/pull/2

Thanks both of you for digging into this. I'm not familiar enough with
Apache's code to pass confident judgement, but your findings certainly
convinced me that this is just an apache bug.

Given there could be a potential reliability issue with an upstream
dependency like Apache, I was considering what mitigation strategies
might help:
[...]

Depending on how widespread the Apache bug is, another option might just
be: do nothing and wait for it to get fixed.

Trying to make the wedged state fail fast and loudly is mostly just
punting on the problem. We'd still see spurious failures. We've so far
resisted the urge to do any automatic flaky-test retries, preferring
instead to just try to root out the flakes. I'm a little hesitant to
start now, because I think our strategy has mostly been good so far, and
I've seen some horrible counter-examples where flakes and retries become
a routine drag on development (and I'm afraid that accommodating flakes
might make them more common).

  - Make slow tests faster by optimizing the test itself and/or
    the test runner configuration (e.g., job number matching
    cores) so wedges become less likely.

It sounds like the bad state is triggered when Apache hits a timeout,
and we hit that timeout because the system is slow or busy. We could try
to make things less slow, but would it work equally well to increase
that timeout?

-Peff