JUnit-Output Format?

[strowi]

Hi,

im currently testing anchore-cli in ci. What about support for junit-compatible output-format? Junit is supported by a wide range of ci-tools, including Gitlab and Jenkins..

regards,
Roman

[zhill]

Hi @strowi , are you asking about adding tests and their output formats or an output format for the CLI itself that is junit-compatible? It seems like the latter, but I want to be sure I'm not mis-understanding. Could you provide a very short example of what you'd like to see as a use-case?

[strowi]

sorry not including an example. In our case i was talking about an default junit report, which can be use with gitlab-ci (doc: https://docs.gitlab.com/ee/ci/junit_test_reports.html).

[alfredodeza]

@strowi what type of information would you be wanting to extract? The output from anchore-cli can be very rich, which the spec might not really like. Do you think you can try with this sample JUnit XML?

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite errors="0" failures="1" hostname="localhost" name="anchore-cli" skipped="0" tests="1" time="2.305" timestamp="2020-01-29T15:58:20.319307">
    <testcase classname="CVE-2016-5636" file="/usr/lib64/python2.7/lib-dynload/Python" name="Python-2.7.5" time="0.001">
        <properties>
            <property name="severity" value="critical"/>
            <property name="fix" value="None"/>
            <property name="url" value="https://nvd.nist.gov/vuln/detail/CVE-2016-5636"/>
            <property name="severity" value="critical"/>
        </properties>
    </testcase>
  </testsuite>
</testsuites>

[strowi]

@alfredodeza thx for taking the time.;) That is almost working...

I had to add <failure>..</failure> within the testcase to actually show a ui-warning (otherwise it would show "no changes").
And i think the properties are ignored completely..?

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite errors="0" failures="1" hostname="localhost" name="anchore-cli" skipped="0" tests="1" time="2.305" timestamp="2020-01-29T15:58:20.319307">
    <testcase classname="CVE-2016-5636" file="/usr/lib64/python2.7/lib-dynload/Python" name="Python-2.7.5" time="0.001">
        <failure>Low: CVE-2016-5636 - https://nvd.nist.gov/vuln/detail/CVE-2016-5636 </failure>
        <properties>
            <property name="severity" value="critical"/>
            <property name="fix" value="None"/>
            <property name="url" value="https://nvd.nist.gov/vuln/detail/CVE-2016-5636"/>
        </properties>
    </testcase>
  </testsuite>
</testsuites>

Then (if any changes between the source and target branch occured) it will show up like this:

And in the detailed view you could show just any info needed.

And the actual job-failure would result from the anchorecli evalute - exit status...

[alfredodeza]

Ahhh interesting, so having a bunch of <property> tags might not be needed! This might be a bit simpler to get done. Thanks for reporting back!

[jeff-cook]

This would be a great addition to the inline tool.

[alfredodeza]

@strowi I am assuming that you are looking to get the Junit output from anchore-cli image vuln <image name> right? Or is this for another command?

[jeff-cook]

I would like to see it as an option for the following.

anchore-cli image get docker.io/library/debian:latest
anchore-cli image vuln docker.io/library/debian:latest os
anchore-cli evaluate check docker.io/library/debian:latest --detail
anchore-cli image content docker.io/library/debian:latest os

[strowi]

@alfredodeza sorry fot the delay, tought i already replied.;) Yes that is what i'm after.

[jeff-cook]

The GitLab UI will not show the properties of the JUnit report. So if that is the target there is no need. Include everything in the message.

It is nice they support JUint, but it is a limited set.

[nightfurys]

@zhill this looks like a support for new feature - new output format. Can we get your guidance on prioritizing for the backlog

[thoeynck]

+1 for this! As JUnit test results integrate very well with GitlabCI in terms of visibility and ease-of-use, a Anchore vulnarability report in the form of JUnit xml files would be wonderful!

[Church13]

@thoeynck this is more of a stopgap measure, but I wrote a (quick and dirty) python script to do just this (integration with CI): https://gist.github.com/Church13/75e41a3e33d67d705518adb82da4e2e9. Feel free to use it if you'd like!

It makes use of the anchore-cli --json flag to output results in JSON format, which is then parsed by my script and converted to JUnit XML. As of right now it is only compatible with the evaluate cli arg. Here's an example:

anchore-cli image add $STAGED_IMAGE --dockerfile=./Dockerfile --force || exit 1
anchore-cli image wait $STAGED_IMAGE || exit 1
anchore-cli image vuln $STAGED_IMAGE all
anchore-cli --json evaluate check $STAGED_IMAGE --detail > scan_results.json
python ./anchore-junit.py --json scan_results.json --output scan_results.xml

@nightfurys @zhill would also like to lend my vote toward it being a new feature, the JSON output is more parseable than straight CLI output but it is still somewhat messy to work with. I might look into implementing myself via a PR if that would be welcome/work has not been done internally toward this effort yet.