Running multiple THEN's per test case

[U007D]

I have a test suite set up as follows:

#[test]
fn tests() {
    TestRunner::new()
        .set_module_path(module_path!())
        .set_attributes(TestRunnerAttributes.disable_final_stats | TestRunnerAttributes.minimize_output)
        .set_time_unit(TestRunnerTimeUnits.microseconds)
        .run_tests(vec![
            TestCase::new("App::run()", "yields arch width", Box::new(|_logger: &mut Logger| -> TestCaseStatus {
                // GIVEN an app
                let mock_width = 42;
                let expected_result = Ok::<String, Error>(format!("Hello, {}-bit world!", mock_width));
                let mock = MockArch::new(mock_width);
                let sut = App::new(&mock);

                // WHEN the app is run
                let result = sut.run();

                // THEN the result should contain the expected architecture width
                match result == expected_result {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
            })),
            TestCase::new("App::run()", "calls Info::width() once", Box::new(|_logger: &mut Logger| -> TestCaseStatus {
                // GIVEN an app
                let mock_width = 42;
                let mock = MockArch::new(mock_width);
                let sut = App::new(&mock);

                // WHEN the app is run
                let _ = sut.run();

                // THEN the app should have called Info::width() exactly once
                match mock.width_times_called.get() == 1 {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
            })),
        ]);
}

Instead of repeating nearly the entire test case, I would prefer to simply add an additional THEN clause to the end of the first test case, like so:

fn tests() {
    TestRunner::new()
        .set_module_path(module_path!())
        .set_attributes(TestRunnerAttributes.disable_final_stats | TestRunnerAttributes.minimize_output)
        .set_time_unit(TestRunnerTimeUnits.microseconds)
        .run_tests(vec![
            TestCase::new("App::run()", "yields arch width", Box::new(|_logger: &mut Logger| -> TestCaseStatus {
                // GIVEN an app
                let mock_width = 42;
                let expected_result = Ok::<String, Error>(format!("Hello, {}-bit world!", mock_width));
                let mock = MockArch::new(mock_width);
                let sut = App::new(&mock);

                // WHEN the app is run
                let result = sut.run();

                // THEN the result should contain the expected architecture width
                match result == expected_result {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }

                // AND_THEN the app should have called Info::width() exactly once
                match mock.width_times_called.get() == 1 {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
            })),
        ]);
}

Obviously, this won't compile given the current signature. The multi-test test case becomes more important as the complexity of the tests increase, by keeping the amount of duplicated code to a minimum.

Your version is at 0.9, so I thought I would bring this up before you stabilize your API at 1.0, just in case it ends up being a breaking change.

Trying not to break the API, here is one idea that might work:

fn tests() {
    TestRunner::new()
        .set_module_path(module_path!())
        .set_attributes(TestRunnerAttributes.disable_final_stats | TestRunnerAttributes.minimize_output)
        .set_time_unit(TestRunnerTimeUnits.microseconds)
        .run_tests(vec![
            ResultTestCase::new("App::run()", "yields arch width", Box::new(|_logger: &mut Logger| -> Result<(), TestCaseStatus::FAILED>  {
                // GIVEN an app
                let mock_width = 42;
                let expected_result = Ok::<String, Error>(format!("Hello, {}-bit world!", mock_width));
                let mock = MockArch::new(mock_width);
                let sut = App::new(&mock);

                // WHEN the app is run
                let result = sut.run();

                // THEN the result should contain the expected architecture width
                test_case_assert_eq(match result, expected_result)?

                // AND_THEN the app should have called Info::width() exactly once
                test_case_assert_eq(mock.width_times_called.get(), 1)?
            })),
        ]);
}

The benefits are that the test fails at the exact line where the test fails. This means that a developer can read the error message and will know the precise issue without having to enter debug.
Contrasted with:

...
                let test_case_status = TestCaseStatus::FAILED;
                // THEN the result should contain the expected architecture width
                test_case_status = match result == expected_result {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
                // AND_THEN the app should have called Info::width() exactly once
                test_case_status = match mock.width_times_called.get() == 1 {
                    true => TestCaseStatus::PASSED,
                    false => TestCaseStatus::FAILED,
                }
                // AND_THEN ...
                ...

                // AND_THEN ...
                ...

                test_case_status
            })),

where a) state is required to be maintained by the developer, and b) in the event of a failure, the specific sub-test which failed is lost, necessitating c) a debug session.

Anyway, this is not urgent or anything--this is just a thought I wanted to share with you. Please let me know if you have thoughts on other ways to achieve this.

[U007D]

Hi, @Alkass, just wondering what you think of this idea? Also, is there any chance you could push 0.9.8 with the configurable time units to crates.io?

All the best,
U007D

[Alkass]

@U007D
I apologize for setting you on the back burner, my friend. I've been dealing with a crazy schedule at work over the last few weeks. I just went through the issue you submitted, and I have a suggestion I'd like to propose. How about an on_success and on_failure test cases? So every test case has a another test case it calls if the current test cases passes, and another test case if the test case fails. This turns your test cases into a tree structure, and you get code sharing as a bonus here!

Does this sound like something that'd serve your purpose?

Fadi

[U007D]

No problem at all, @Alkass!

In my proposal, adding an additional THEN clause is exactly one line of code (and, of course, there is full code reuse of the GIVEN and WHEN clause code).

I'm not sure what your proposal would involve for a common scenario such as the one I've outlined. Could you show an example like I've done above?

Eliminating as much boilerplate as possible is definitely a design goal. It could also be that our two proposals serve two different scenarios, and may both add value to polish. What do you think?