Re: Coverity, was Re: What's cooking in git.git (Oct 2021, #02; Wed, 6)

Hi Peff,

On Tue, 16 Aug 2022, Jeff King wrote:

On Tue, Aug 16, 2022 at 11:05:48AM +0200, Johannes Schindelin wrote:

quoted

quoted

It sounds like Taylor is volunteering to set up the Coverity side for
git.git, and I can help him with getting those COVERITY_* variables into
the GitHub environment.

Given the challenges with Coverity (false positives, lack of support on
Synopsys' side, severely limited access to the reports), and given the
renewed efforts by OSTIF that focus not on Coverity but on CodeQL, I am
in favor of abandoning the idea to integrate Coverity in our GitHub
workflow.

Regarding CodeQL, I am still uncertain what level of integration we will
end up with, and the contacts I am working with are currently all on
vacation, but I am confident that we will have an easier time going
forward with static analysis using CodeQL instead of Coverity.

OK. I haven't been that impressed with CodeQL for C so far, but it may
be getting better.

If your lack of being impressed stems from CodeQL's default suite catching
very, very few issues, I agree with you.

The reason why I am more hopeful about CodeQL than about Coverity is that
I now have a contact to work with (although they are currently on
vacation, so I get to work on other things for now, including `merge-ort`
and trying to integrate `mimalloc` in Git for Windows). And that is one
more contact who is willing to work with me than I ever had on the
Coverity side of things.

And apparently CodeQL's default settings optimize for reducing false
positives in an attempt to avoid scaring potential users away.

But I have credible assurances that CodeQL has many more checks in store
that simply need to be enabled in order to catch way more issues, at the
expense of risking more false positives.

I certainly would be happier with a system that made it easier to
display and share reports.

Coverity does have a lot of false positives, but I've at least been able
to pick useful fixes out of them (especially because it is good about
saying "here are 5 new things to look at"). I've been continuing to
build my private branch with it, so we'll see if it turns up anything
useful. I do agree that inflicting it on ordinary users may be
counter-productive (I often have to stare really hard to understand why
its false positives are false, and that is not something I would wish
on, say, a GGG user).

Don't get me wrong, I do not plan on dropping the Coverity builds of the
Git for Windows project's `main` branch at
https://dev.azure.com/git-for-windows/git/_build?definitionId=35.

As you probably recall, I specifically looked through the Coverity report
in advance of v2.37.0 and tried to address some of the issues in
https://lore.kernel.org/git/pull.1264.git.1655336146.gitgitgadget@gmail.com/ (local),
with mixed results (mainly because of some time constraints on my side,
combined with your willingness to help fix some issues with my patches, as
well as René's, Taylor's and Junio's excellent assistance).

However, the sheer amount of false positives, with intentional issues
being a close second (in test helpers, we are not all that strict about
releasing memory, for example), makes it a tough sell to ask more than
just a few very dedicated contributors to have a look at the reports.

With CodeQL, I am optimistic that we can get it to a point where the
burden can be carried by a larger group of people, with the help of
enthusiastic CodeQL experts, which also means that the reports have a
bigger chance at making Git safer.

Ciao,
Dscho