[Feature request] Report todos as a separate category

[richardcocks]

As a Test Driven Developer
I'd like to see failures due to todo reported differently
So that I can see my progress from todo to passing and more easily see if I've caused regressions or "proper" failures

Motivation:

If I have a library, I can scaffold the interface with todo in the implementation, and then write tests against the library in a todo state.

As I implement functions, it can be hard to see "legitimate" failures in the noise of failures due to todo.

I'd like any failure due to a todo reported separately, to aid test driven development.

I have an apparently working version in my fork, but I don't completely trust it yet, the core of my logic is:

  case gleam_panic.from_dynamic(error) {
    Ok(gleam_panic.GleamPanic(kind: gleam_panic.Todo, ..) as e) -> {
      let src = option.from_result(read_file(e.file))
      let message = format_gleam_error(e, module, function, src)
      io.print("\n" <> message)
      State(..state, todos: state.todos + 1)
    }

with State(passed: Int, failed: Int, skipped: Int, todos: Int) to track todo count.

[lpil]

Sounds great! Let's do this!

[richardcocks]

Yay! I'm glad you're on board, I already have a blog-post title in mind, "todo driven development".

[lpil]

Would you like to make a PR? :)

[richardcocks]

Gladly; should I put this behind a command line switch, and what's the preferred name for that? --report-todos or something less verbose?

[lpil]

Nah let's have it always work that way 👍 Seems useful for everyone. Only thing we need to double-check is that they make it report the tests as failed at the end.

[richardcocks]

I should better explain my feature request first then, because it's the message at the end that I wanted to change, so that passing tests look like this:

Tests runs with failures due to todos look like this:

And with a mix of todos and failures ( other panics or bad asserts ):

The tests themselves still raise todos and failures with details:

The exit code is still a failure even if there are any non-passing, so for CI systems, todos will still be picked up as a failure.

[lpil]

Having them in the todo category sounds good, but that summary list at the end would be a different feature to be discussed independently.

[richardcocks]

Sorry, I forgot that was also in my fork, I've now made a branch without that included, so this is how the feature should look:

Just todos:

Mix of failures and todos:

Just failures:

[lpil]

Sounds good! Those images seem to be dead though. A github problem maybe...

Would you like to make a PR?

[richardcocks]

Sorry for the delay in producing a PR, I've started doing testing and I came across an issue with a todo inside OTP. I'm still ironing out the details.

Hopefully I'll have it all neatened up soon and a PR by the end of this week.