Skip to content

evalite export does not fail if threshold is not reached #370

@Fox32

Description

@Fox32

When running evalite with export to generate a report that I can upload in CI, I noticed that using a threshold doesn't work as I would expect it. The threshold is correctly outputted as failed, but the command does not fail with exit code. Therefore I can't detect in the CI that something is off. It works without using export, but I would like to have both so that I have something to analyze if it fails.

As this is a breaking change, is that something that should go into v1?

PS: I'm using 1.0.0-beta.15

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions