447 add image schema by scpmw · Pull Request #460 · casangi/xradio

scpmw · 2025-08-07T14:15:33Z

Very rough indeed, test cases fail, and likely incompatible with JW's
newer changes.

Notable changes in the infrastructure:

Add float arrays as a primitive type
Move dataclasses to be kw_only so that we can use subclassing
(this requires Python >=3.10, but that isn't a problem at this
point).

This should make schemas portable to other languages and environments.

This should also finally fix the test cases...

Somewhat surprisingly, this still passes the unit tests - numpy defines a "smart" equality operator for numpy.dtype that automatically considers the string representation.

Including re-import, as requested by Simon

We were actually still generating numpy dtypes (they just converted to strings automatically), and dimensions were using tuples. There's now a test that checks that schemas don't change on the JSON round-trip.

Not sure about all of these...

Very rough indeed, test cases fail, and likely incompatible with JW's newer changes. Notable changes in the infrastructure: * Add float arrays as a primitive type * Move dataclasses to be kw_only so that we can use subclassing (this requires Python >=3.10, but that isn't a problem at this point).

codecov · 2025-08-07T14:25:44Z

❌ 16 Tests Failed:

Tests completed	Failed	Passed	Skipped
380	16	364	0

View the top 3 failed test(s) by shortest run time

tests/unit/schema/test_schema.py::test_check_array_constructor_array_style

Stack Traces | 0.001s run time

def test_check_array_constructor_array_style():
        # Try by using constructor xarray.DataArray-style
        data = numpy.zeros(10, dtype=complex)
        coords = [("coord", numpy.arange(10, dtype=float))]
        attrs = {"attr1": "str", "attr2": 1234, "attr3": 345}
>       array = _TestArraySchema(data=data, coords=coords, attrs=attrs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.../unit/schema/test_schema.py:152: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11.../xradio/schema/bases.py:113: in _dataarray_new
    mapping = sig.bind_partial(data, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11/inspect.py:3202: in bind_partial
    return self._bind(args, kwargs, partial=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Signature (*, data: Annotated[Union[xradio.schema.typing.Labeled[Literal['coord']], Collection[complex], complex], <R...typing.Annotated[int, <Role.ATTR: 'attr'>] = 123, attr3: Optional[Annotated[int, <Role.ATTR: 'attr'>]] = None) -> None>
args = (array([0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j,
       0.+0.j, 0.+0.j]),)
kwargs = {}

    def _bind(self, args, kwargs, *, partial=False):
        """Private method. Don't use directly."""
    
        arguments = {}
    
        parameters = iter(self.parameters.values())
        parameters_ex = ()
        arg_vals = iter(args)
    
        while True:
            # Let's iterate through the positional arguments and corresponding
            # parameters
            try:
                arg_val = next(arg_vals)
            except StopIteration:
                # No more positional arguments
                try:
                    param = next(parameters)
                except StopIteration:
                    # No more parameters. That's it. Just need to check that
                    # we have no `kwargs` after this while loop
                    break
                else:
                    if param.kind == _VAR_POSITIONAL:
                        # That's OK, just empty *args.  Let's start parsing
                        # kwargs
                        break
                    elif param.name in kwargs:
                        if param.kind == _POSITIONAL_ONLY:
                            msg = '{arg!r} parameter is positional only, ' \
                                  'but was passed as a keyword'
                            msg = msg.format(arg=param.name)
                            raise TypeError(msg) from None
                        parameters_ex = (param,)
                        break
                    elif (param.kind == _VAR_KEYWORD or
                                                param.default is not _empty):
                        # That's fine too - we have a default value for this
                        # parameter.  So, lets start parsing `kwargs`, starting
                        # with the current parameter
                        parameters_ex = (param,)
                        break
                    else:
                        # No default, not VAR_KEYWORD, not VAR_POSITIONAL,
                        # not in `kwargs`
                        if partial:
                            parameters_ex = (param,)
                            break
                        else:
                            msg = 'missing a required argument: {arg!r}'
                            msg = msg.format(arg=param.name)
                            raise TypeError(msg) from None
            else:
                # We have a positional argument to process
                try:
                    param = next(parameters)
                except StopIteration:
                    raise TypeError('too many positional arguments') from None
                else:
                    if param.kind in (_VAR_KEYWORD, _KEYWORD_ONLY):
                        # Looks like we have no parameter for this positional
                        # argument
>                       raise TypeError(
                            'too many positional arguments') from None
E                       TypeError: too many positional arguments

.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11/inspect.py:3121: TypeError

tests/unit/schema/test_schema.py::test_check_array_constructor_from_dataarray_override

Stack Traces | 0.001s run time

def test_check_array_constructor_from_dataarray_override():
        # Create schema-conformant DataArray
        data = numpy.zeros(10, dtype=complex)
        coords = [("coord", numpy.arange(10, dtype=float))]
        attrs = {"attr1": "str", "attr2": 1234, "attr3": 345}
        array = xarray.DataArray(data, coords, attrs=attrs)
    
        # Check that we can create a copy form it using constructor,
        # but override attributes and coordinates
>       array = _TestArraySchema(
            data=array,
            coords=[("coord", 1 + numpy.arange(10, dtype=float))],
            attrs={"attr1": "strstr", "attr2": 12345},
        )

.../unit/schema/test_schema.py:216: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11.../xradio/schema/bases.py:113: in _dataarray_new
    mapping = sig.bind_partial(data, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11/inspect.py:3202: in bind_partial
    return self._bind(args, kwargs, partial=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Signature (*, data: Annotated[Union[xradio.schema.typing.Labeled[Literal['coord']], Collection[complex], complex], <R...typing.Annotated[int, <Role.ATTR: 'attr'>] = 123, attr3: Optional[Annotated[int, <Role.ATTR: 'attr'>]] = None) -> None>
args = (<xarray.DataArray (coord: 10)> Size: 160B
array([0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j,
    ...loat64 80B 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0
Attributes:
    attr1:    str
    attr2:    1234
    attr3:    345,)
kwargs = {}

    def _bind(self, args, kwargs, *, partial=False):
        """Private method. Don't use directly."""
    
        arguments = {}
    
        parameters = iter(self.parameters.values())
        parameters_ex = ()
        arg_vals = iter(args)
    
        while True:
            # Let's iterate through the positional arguments and corresponding
            # parameters
            try:
                arg_val = next(arg_vals)
            except StopIteration:
                # No more positional arguments
                try:
                    param = next(parameters)
                except StopIteration:
                    # No more parameters. That's it. Just need to check that
                    # we have no `kwargs` after this while loop
                    break
                else:
                    if param.kind == _VAR_POSITIONAL:
                        # That's OK, just empty *args.  Let's start parsing
                        # kwargs
                        break
                    elif param.name in kwargs:
                        if param.kind == _POSITIONAL_ONLY:
                            msg = '{arg!r} parameter is positional only, ' \
                                  'but was passed as a keyword'
                            msg = msg.format(arg=param.name)
                            raise TypeError(msg) from None
                        parameters_ex = (param,)
                        break
                    elif (param.kind == _VAR_KEYWORD or
                                                param.default is not _empty):
                        # That's fine too - we have a default value for this
                        # parameter.  So, lets start parsing `kwargs`, starting
                        # with the current parameter
                        parameters_ex = (param,)
                        break
                    else:
                        # No default, not VAR_KEYWORD, not VAR_POSITIONAL,
                        # not in `kwargs`
                        if partial:
                            parameters_ex = (param,)
                            break
                        else:
                            msg = 'missing a required argument: {arg!r}'
                            msg = msg.format(arg=param.name)
                            raise TypeError(msg) from None
            else:
                # We have a positional argument to process
                try:
                    param = next(parameters)
                except StopIteration:
                    raise TypeError('too many positional arguments') from None
                else:
                    if param.kind in (_VAR_KEYWORD, _KEYWORD_ONLY):
                        # Looks like we have no parameter for this positional
                        # argument
>                       raise TypeError(
                            'too many positional arguments') from None
E                       TypeError: too many positional arguments

.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11/inspect.py:3121: TypeError

tests/unit/schema/test_schema.py::test_check_array_constructor_list

Stack Traces | 0.001s run time

def test_check_array_constructor_list():
        # Check that we can use lists instead of numpy arrays, and they get
        # converted into numpy arrays of the schema type
>       array = _TestArraySchema(
            data=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
            coord=("coord", [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], {"asd": "foo"}),
            attr1="str",
            attr2=1234,
            attr3=345,
        )

.../unit/schema/test_schema.py:254: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11.../xradio/schema/bases.py:113: in _dataarray_new
    mapping = sig.bind_partial(data, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11/inspect.py:3202: in bind_partial
    return self._bind(args, kwargs, partial=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Signature (*, data: Annotated[Union[xradio.schema.typing.Labeled[Literal['coord']], Collection[complex], complex], <R...typing.Annotated[int, <Role.ATTR: 'attr'>] = 123, attr3: Optional[Annotated[int, <Role.ATTR: 'attr'>]] = None) -> None>
args = ([0, 0, 0, 0, 0, 0, ...],)
kwargs = {'attr1': 'str', 'attr2': 1234, 'attr3': 345, 'coord': ('coord', [0, 1, 2, 3, 4, 5, ...], {'asd': 'foo'})}

    def _bind(self, args, kwargs, *, partial=False):
        """Private method. Don't use directly."""
    
        arguments = {}
    
        parameters = iter(self.parameters.values())
        parameters_ex = ()
        arg_vals = iter(args)
    
        while True:
            # Let's iterate through the positional arguments and corresponding
            # parameters
            try:
                arg_val = next(arg_vals)
            except StopIteration:
                # No more positional arguments
                try:
                    param = next(parameters)
                except StopIteration:
                    # No more parameters. That's it. Just need to check that
                    # we have no `kwargs` after this while loop
                    break
                else:
                    if param.kind == _VAR_POSITIONAL:
                        # That's OK, just empty *args.  Let's start parsing
                        # kwargs
                        break
                    elif param.name in kwargs:
                        if param.kind == _POSITIONAL_ONLY:
                            msg = '{arg!r} parameter is positional only, ' \
                                  'but was passed as a keyword'
                            msg = msg.format(arg=param.name)
                            raise TypeError(msg) from None
                        parameters_ex = (param,)
                        break
                    elif (param.kind == _VAR_KEYWORD or
                                                param.default is not _empty):
                        # That's fine too - we have a default value for this
                        # parameter.  So, lets start parsing `kwargs`, starting
                        # with the current parameter
                        parameters_ex = (param,)
                        break
                    else:
                        # No default, not VAR_KEYWORD, not VAR_POSITIONAL,
                        # not in `kwargs`
                        if partial:
                            parameters_ex = (param,)
                            break
                        else:
                            msg = 'missing a required argument: {arg!r}'
                            msg = msg.format(arg=param.name)
                            raise TypeError(msg) from None
            else:
                # We have a positional argument to process
                try:
                    param = next(parameters)
                except StopIteration:
                    raise TypeError('too many positional arguments') from None
                else:
                    if param.kind in (_VAR_KEYWORD, _KEYWORD_ONLY):
                        # Looks like we have no parameter for this positional
                        # argument
>                       raise TypeError(
                            'too many positional arguments') from None
E                       TypeError: too many positional arguments

.../hostedtoolcache/Python/3.11.13........./x64/lib/python3.11/inspect.py:3121: TypeError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

scpmw added 14 commits May 1, 2025 18:55

Replace Python annotations with explicit type/literal descriptions

9af7c7f

This should make schemas portable to other languages and environments.

Introduce separate ValueSchema so we can tackle decorator checks

1401e70

This should also finally fix the test cases...

Use string representation of numpy.dtype

cc4dd2b

Somewhat surprisingly, this still passes the unit tests - numpy defines a "smart" equality operator for numpy.dtype that automatically considers the string representation.

Rename type_name to type

13fb99e

Add proof-of-concept of schema export

a36f70d

Including re-import, as requested by Simon

Add schemas as exported by exporter

a0ce42b

Add schema exporter script

f591954

Reblackify

ed45ee3

Add tests, fix some small issues

c34ad33

We were actually still generating numpy dtypes (they just converted to strings automatically), and dimensions were using tuples. There's now a test that checks that schemas don't change on the JSON round-trip.

Merge remote-tracking branch 'origin/main' into 385-add-schema-export

3390651

Make test cases pass

55be5cf

Not sure about all of these...

Add test to confirm that exported schemas are synchronised

676dab1

Merge branch '385-add-schema-export' into add-image-schema

e91fe88

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

447 add image schema#460

447 add image schema#460
scpmw wants to merge 14 commits intomainfrom
447-add-image-schema

scpmw commented Aug 7, 2025

Uh oh!

codecov bot commented Aug 7, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

scpmw commented Aug 7, 2025

Uh oh!

codecov bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 16 Tests Failed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

codecov bot commented Aug 7, 2025 •

edited

Loading