Fatal error from a tool results in "Success with 0 issues found."

[dmitshur]

Note: Conception-go is a large unsupported repository with low quality Go code. It's not a goal to have it work well; I'm just using it as an opportunity to spot an issue that can happen in another useful Go repository.

See https://gci.gopherci.io/analysis/1156:

However, one of the tools failed to complete running:

$ staticcheck ./... 33.577s
fatal error: runtime: out of memory

It looks like a failure (non-zero exit code) from a tool isn't being correctly recognized and reported. This is bad, because it can give a false sense of confidence ("GopherCI told me were no issues in this commit; it turned out there were some issues but they weren't reported due to some problem").

[bradleyfalzon]

Reporting “Success” isn’t correct here, although, non zero exit codes are normal for static analysis tools. I’ll need to see if there’s a cleaner way or I will need to special case some of these messages.

[dmitshur]

non zero exit codes are normal for static analysis tools

Do you mean it's normal for static analysis tools to give non-zero exit code and report zero problems?

[bradleyfalzon]

Do you mean it's normal for static analysis tools to give non-zero exit code and report zero problems?

Ah no, I misunderstood your point. But I then agree, a non-zero exit status with no issues detected should then be considered an unknown state to be raised to the user.

Edit: well, actually, it's possible at least one issue was raised by the tool and successfully parsed by us - but then the tool could exceed memory limits.