Unable to use approvaltest with mutation testing [mutmut]

[udelledo]

I configured in my project ApprovalTest for unit testing, and mutmut for mutation testing.

Unfortunately they don't seem to work well together.

I originally reported the issue in the mutmut repo but was mentioned that there might be a too simplified check in the assertion of multiple verify execution.

Is there any available workaround that would let me still use the two libraries together?

[nitsanavni]

What's happening when trying to use them together?

[nitsanavni]

In general, the way I use Approvals + Mutation Testing, is that for the sake of running the mutation tests I supress the usage of approvals reporters - I only want the test to fail, but don't want a diff tool to open on each failure.

[udelledo]

I get the error about multiple verify being executed without the proper configuration

error message

❯ mutmut run
⠹ Generating mutants
    done in 790ms
⠇ Running stats     
    done
⠧ Running clean testsF
============================================================================================================ FAILURES =============================================================================================================
___________________________________________________________________________________________ TestTeamCityConfig.test_save_configuration ____________________________________________________________________________________________

self = <tests.ezconfig_test.TestTeamCityConfig object at 0x7f20f970a370>, args = (), kwargs = {}, config_file = <tempfile._TemporaryFileWrapper object at 0x7f20f9638160>

    @patch("getpass.getpass", lambda x: "my_test_token")
    @patch("builtins.input", lambda *args: "")
    @patch("requests.Session.get", lambda url, x: MagicMock())
    def test_save_configuration(self, *args, **kwargs):
        with tempfile.NamedTemporaryFile() as config_file:
            sut = TeamCityConfig(config_file.name)
            sut.populate_from_user_input()
            sut.configure_token()
>           verify(
                "\n".join(
                    [f"{field.name}={sut.get(field.name)}" for field in sut.fieldList]
                ),
                options=Options().with_scrubber(DateScrubber(r"\d{8}T\d{6}").scrub),
            )
E           approvaltests.approval_exception.ApprovalException: We noticed that you called verify more than once in the same test. 
E           This is the second call to verify:
E               approved_file: /webdev/teamcity-lib/tests/approved_files/TestTeamCityConfig.test_save_configuration.approved.txt
E           
E           By default, ApprovalTests only allows one verify() call per test.
E           To find out more, visit: 
E           https://github.com/approvals/ApprovalTests.Python/blob/main/docs/how_to/multiple_approvals_per_test.md
E           
E           # Possible Fixes
E           1. Separate your test into two tests
E           2. In your verify call, add `options=NamerFactory.with_parameters("your_paramater")`
E           3. In your test, call `approvals.settings().allow_multiple_verify_calls_for_this_method()`

How do you remove the reporters only for mutation testing?

[nitsanavni]

Right, I was able to reproduce this too here - https://github.com/nitsanavni/mutmut-approvals

The solution is to allow multiple verify calls per test method, see here - https://github.com/nitsanavni/mutmut-approvals/blob/master/tests/conftests.py#L7

I think we should add this to the docs.

[nitsanavni]

Also, we should add an option to disallow this error per session.

[udelledo]

If I add the option in a global fixture, all my tests fails, not only in mutmut, but also with plain pytest tests command. Not sure if it could be caused by the fact that in some cases I use parametrized test, with a different namer?!

error

ns-teamcity-lib-venv/lib64/python3.9/site-packages/approvaltests/namer/stack_frame_namer.py:42: FrameNotFound
________________________________________________________________________________ ERROR at setup of TestTeamCityBasicAuth.test_get_project_by_name _________________________________________________________________________________

    @pytest.fixture(scope="session", autouse=True)
    def configure_approvals_fixture():
>       approvaltests.approvals.settings().allow_multiple_verify_calls_for_this_method()

tests/conftest.py:10: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ns-teamcity-lib-venv/lib64/python3.9/site-packages/approvaltests/namer/default_name.py:8: in get_default_namer
    return StackFrameNamer(extension)
ns-teamcity-lib-venv/lib64/python3.9/site-packages/approvaltests/namer/stack_frame_namer.py:20: in __init__
    self.set_for_stack(inspect.stack(1))
ns-teamcity-lib-venv/lib64/python3.9/site-packages/approvaltests/namer/stack_frame_namer.py:25: in set_for_stack
    frame = self.get_test_frame_index(caller)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

caller = [FrameInfo(frame=<frame at 0x7f4d284c2200, file '/webdev/teamcity-lib/ns-teamcity-lib-venv/lib64/python3.9/site-packag...no=898, function='call_fixture_func', code_context=['        fixture_result = fixturefunc(**kwargs)\n'], index=0), ...]

        @staticmethod
        def get_test_frame_index(caller: List[FrameInfo]) -> int:
            tmp_array = []
            for index, frame in enumerate(caller):
                if StackFrameNamer.is_test_method(frame):
                    tmp_array.append(index)
            if tmp_array:
                return tmp_array[-1]
            message = """Could not find test method/function. Possible reasons could be:
    1) approvaltests is not being used inside a test function
    2) your test framework is not supported by ApprovalTests (unittest and pytest are currently supported)."""
>           raise FrameNotFound(message)
E           approvaltests.approval_exception.FrameNotFound: Could not find test method/function. Possible reasons could be:
E           1) approvaltests is not being used inside a test function
E           2) your test framework is not supported by ApprovalTests (unittest and pytest are currently supported).

ns-teamcity-lib-venv/lib64/python3.9/site-packages/approvaltests/namer/stack_frame_namer.py:42: FrameNotFound

[nitsanavni]

Ok, we'll find a better fix