Several tests failing with `AssertionError`

[GaetanLepage]

Describe the bug
Since the 0.1.7 release, several tests are failing:

To Reproduce

pytest

Expected behavior
Tests succeed.

Actual behavior

=========================== short test summary info ============================
FAILED test/test_bools.py::test_bool_args_in_help - AssertionError: assert 'usage: __mai...ult: False)\n' == 'usage: __mai...ul...
FAILED test/test_docstrings.py::test_weird_docstring_with_field_like - AssertionError: assert AttributeDocS..._docstring='') == AttributeDocS..._d...
FAILED test/test_docstrings.py::test_docstring_builds_upon_bases - AssertionError: assert AttributeDocS..._docstring='') == AttributeDocS..._d...
FAILED test/test_docstrings.py::test_desc_from_cls_docstring - AssertionError: assert AttributeDocS...y default 32') == AttributeDocS...y ...
FAILED test/test_docstrings.py::test_help_takes_value_from_docstring - AssertionError: assert AttributeDocS...Display logs') == AttributeDocS...Di...
FAILED test/test_examples.py::test_running_example_outputs_expected_without_arg[examples/aliases/aliases_example.py] - AssertionError: (['Namespace(train=RunSettings(debug=False, verbose=False),...
FAILED test/test_examples.py::test_running_example_outputs_expected_without_arg[examples/simple/flag.py] - AssertionError: (["HParams(num_layers=4, num_units=64, optimizer='ADAM', le...
FAILED test/test_examples.py::test_running_example_outputs_expected_without_arg[examples/simple/option_strings.py] - AssertionError: (["HParams(num_layers=4, num_units=64, optimizer='ADAM', le...
FAILED test/test_issue_48.py::test_issue_48 - AssertionError: 
FAILED test/test_subgroups.py::test_help_string[AB-get_help_text_args1-should_contain1] - assert '--a_or_b {a,b}       (default: a)' in "usage: __main__.py [-h] [--a...
FAILED test/test_subgroups.py::test_help_string[MultipleSubgroupsDifferentLevel-get_help_text_args4-should_contain4] - assert '--e bool' in "usage: __main__.py [-h] [--ab_or_cd {ab,cd}] [--e_or_...
FAILED test/test_subgroups.py::test_help_string_displays_default_factory_arguments[a_factory0-b_factory0] - assert '--a float  (default: 321)' in "usage: __main__.py [-h] [--a_or_b {a...
FAILED test/test_subgroups.py::test_help_string_displays_default_factory_arguments[a_factory1-b_factory1] - assert '--a float  (default: 321)' in "usage: __main__.py [-h] [--a_or_b {a...
FAILED test/test_subgroups.py::test_help_string_displays_default_factory_arguments[a_factory2-b_factory2] - assert '--a float  (default: 111)' in "usage: __main__.py [-h] [--a_or_b {a...
FAILED test/test_subgroups.py::test_subgroup_partial_with_nested_field - assert '--a int       (default: 123)' in "usage: __main__.py [-h] [--foo {s...
= 15 failed, 1264 passed, 11 skipped, 96 xfailed, 9 xpassed, 2 warnings in 17.59s =

Full logs: https://paste.glepage.com/raw/sloth-bear-raven

Desktop (please complete the following information):

• Version: 0.1.7
• OS: NixOS
• Python version: 3.12

Additional context
Bumping SimpleParsing in nixpkgs: NixOS/nixpkgs#375473

[lebrice]

Hey there @GaetanLepage , thanks for pointing this out!

I'll double check. I suspect this has to do with the python version used for tests involving checking the "--help" string or the output, since argparse format varies slightly between Python versions.

[GaetanLepage]

Hey there @GaetanLepage , thanks for pointing this out!

I'll double check. I suspect this has to do with the python version used for tests involving checking the "--help" string or the output, since argparse format varies slightly between Python versions.

Ok thanks for the update.
Feel free to ping me if testing is needed !

[lebrice]

Hey @GaetanLepage , I switched from Poetry to UV in #340. Do you still see issues in the tests with the code from master?

[GaetanLepage]

Thanks a lot @lebrice it now works perfectly !

[GaetanLepage]

Are you planning to make a minor release to ship this fix ?

[lebrice]

Probably, yes.

[GaetanLepage]

Those tests still fail with 0.1.8.

[lebrice]

Hey @GaetanLepage , thanks for letting me know. I can't seem to replicate the test failures on Python 3.12 locally.
Can you perhaps give me a bit more insight into what is being done in NixOS/nixpkgs#482119, and where I might be able to find the most recent test failures in the CI steps?

[GaetanLepage]

Hey @GaetanLepage , thanks for letting me know. I can't seem to replicate the test failures on Python 3.12 locally. Can you perhaps give me a bit more insight into what is being done in NixOS/nixpkgs#482119, and where I might be able to find the most recent test failures in the CI steps?

Unfortunately, the CI run in question happened locally, before I reverted to skipping the tests.
FWIW, we simply run pytest without any specific options.

Here is an example of the logs showing one of these tests failing:

__ test_help_string_displays_default_factory_arguments[a_factory2-b_factory2] __

a_factory = functools.partial(<class 'test.test_subgroups.A'>, a=111)
b_factory = functools.partial(<class 'test.test_subgroups.B'>, b='foobar')

    @pytest.mark.parametrize(
        "a_factory, b_factory",
        [
            (partial(A, a=321), partial(B, b="foobar")),
            (partial(partial(A, a=111), a=321), partial(partial(B), b="foobar")),
            (partial(partial(A, a=111)), partial(partial(B, b="foobar"))),
            lambdas_arent_supported_yet(lambda: A(a=123), lambda: B(b="foooo")),
        ],
    )
    def test_help_string_displays_default_factory_arguments(
        a_factory: Callable[[], A], b_factory: Callable[[], B]
    ):
        """The help string should be basically the same when using a `partial` or a lambda that returns
        a dataclass, as using just the class itself.
    
        When using `functools.partial` or lambda expressions, we'd ideally also like the help text to
        show the field values from inside the `partial` or lambda, if possible.
        """
    
        # NOTE: Here we need to return just A() and B() with these default factories, so the defaults
        # for the fields are the same
        @dataclass
        class Foo(TestSetup):
            a_or_b: A | B = subgroups({"a": a_factory, "b": b_factory}, default="a")
    
        a_default_from_factory = a_factory().a
        b_default_from_factory = b_factory().b
        a_default_from_field = A().a
        b_default_from_field = B().b
    
        # Just to make sure that we're using a different value for the default.
        assert a_default_from_field != a_default_from_factory
        assert b_default_from_field != b_default_from_factory
    
        # Check that it doesn't use the default value from the field in the help text.
        assert f"--a float (default: {a_default_from_field})" not in Foo.get_help_text("")
        assert f"--a float (default: {a_default_from_field})" not in Foo.get_help_text("--a_or_b a")
        assert f"--b str (default: {b_default_from_field})" not in Foo.get_help_text("--a_or_b b")
    
        # Check that it uses the default value from the factory in the help text.
>       assert f"--a float  (default: {a_default_from_factory})" in Foo.get_help_text("")
E       assert '--a float  (default: 111)' in "usage: __main__.py [-h] [--a_or_b {a,b}] [-a float]\n\noptions:\n  -h, --help      show this help message and exit\n\...>)\n\n  --a_or_b {a,b}  (default: a)\n\nA ['foo.a_or_b']:\n  A(a: 'float' = 0.0)\n\n  -a, --a float   (default: 111)\n"
E        +  where "usage: __main__.py [-h] [--a_or_b {a,b}] [-a float]\n\noptions:\n  -h, --help      show this help message and exit\n\...>)\n\n  --a_or_b {a,b}  (default: a)\n\nA ['foo.a_or_b']:\n  A(a: 'float' = 0.0)\n\n  -a, --a float   (default: 111)\n" = get_help_text('')
E        +    where get_help_text = <class 'test.test_subgroups.test_help_string_displays_default_factory_arguments.<locals>.Foo'>.get_help_text

test/test_subgroups.py:494: AssertionError

[lebrice]

Can you give me some steps to try to replicate this failure? What OS are you on, and which Python version?
This works on the GitHub CI and locally, on all OS's and Python versions I've tried. I would need a bit more information about your setup to better understand why help strings are different on your end, and how to potentially xfail the tests (as well as to identify which tests specifically need to be marked) in a way that doesn't degrade the CI here.

[GaetanLepage]

I was able to reproduce it by:

git clone git@github.com:lebrice/SimpleParsing.git

cd SimpleParsing/

git checkout v0.1.8

uv run pytest