Replying one more time, because I certainly don't want to create any
hard feeling here.

On Mon, Sep 11, 2023 at 03:30:55PM +0200, Michel Dänzer wrote:

quoted

quoted

quoted

quoted

By keeping those sets of expectations, we've been able to keep Mesa pretty
clear of regressions, whilst having a very clear set of things that should
be fixed to point to. It would be great if those set of things were zero,
but it just isn't. Having that is far better than the two alternatives:
either not testing at all (obviously bad), or having the test always be red
so it's always ignored (might as well just not test).

Isn't that what happens with flaky tests anyway?

For a small minority of tests. Daniel was referring to whole test suites.

quoted

Even more so since we have 0 context when updating that list.

The commit log can provide whatever context is needed.

Sure, I've yet to see that though.

There's in 6.6-rc1 around 240 reported flaky tests. None of them have
any context. That new series hads a few dozens too, without any context
either. And there's no mention about that being a plan, or a patch
adding a new policy for all tests going forward.

That does sound bad, would need to be raised in review.

quoted

Any concern I raised were met with a giant "it worked on Mesa" handwave

Lessons learned from years of experience with big real-world CI
systems like this are hardly "handwaving".

Your (and others) experience certainly isn't. It is valuable, welcome,
and very much appreciated.

However, my questions and concerns being ignored time and time again
about things like what is the process is going to be like, what is going
to be tested, who is going to be maintaining that test list, how that
interacts with stable, how we can possibly audit the flaky tests list,
etc. have felt like they were being handwaived away.

I'm not saying that because I disagree, I still do on some, but that's
fine to some extent. However, most of these issues are not so much an
infrastructure issue, but a community issue. And I don't even expect a
perfect solution right now, unlike what you seem to think. But I do
expect some kind of plan instead of just ignoring that problem.

Like, I had to ask the MT8173 question 3 times in order to get an
answer, and I'm still not sure what is going to be done to address that
particular issue.

So, I'm sorry, but I certainly feel like it here.

Maxime