Fix/return validation errors instead of db errors

[Flix6x]

Description

• Return validation errors instead of db errors
• Move from load_default=None to required=False so that _serialize methods no longer need to pass through None values (apispec caught this)
• Streamline function signatures of _serialize and _deserialize methods
• The exact same JSON Marshmallow field was defined twice -> refactored
• Added changelog item in documentation/changelog.rst
• Added changelog item in documentation/api/changelog.rst
• Added changelog item in documentation/cli/changelog.rst

Look & Feel

Now matches expected responses detailed in #1922.

How to test

The new test_all_custom_fields_call_super_deserialize makes sure that all fields mapping to a db object calls super()._deserialize, which checks that the value is actually an integer ID.

Further Improvements

The test uses a util method (deserialize_calls_super) with a switch (and_queries_db). If we set this to True we would also cover all fields that do not themselves query the db as part of deserializing. I tuned this off for now, because we have various fields that worked around this bug in one way or another, using their own validation logic rather than that of their superclass Marshmallow field. For instance:

class PipedAssetFieldListField(fields.Str):
    """
    Field that represents a list of Strings, in serialized form joined by "|".
    Each one should represent a field in the AssetAPIQuerySchema.
    """

    def _deserialize(self, values: str, attr, data, **kwargs) -> list[str]:
        if not isinstance(values, str):
            raise validate.ValidationError(
                "Invalid input type, should be a string, separable by '|'."
            )
        ...

could instead have been:

class PipedAssetFieldListField(fields.Str):
    """
    Field that represents a list of Strings, in serialized form joined by "|".
    Each one should represent a field in the AssetAPIQuerySchema.
    """

    def _deserialize(self, value: typing.Any, attr, data, **kwargs) -> list[str]:
        values: str = super()._deserialize(value, attr, data, **kwargs)
        ...

in which case the ValidationError would become slightly less informative.

In this particular case, the replacement would probably still be acceptable, but some others have more explicit error messages right now.

Related Items

Closes #1922.

[read-the-docs-community]

Documentation build overview

📚 flexmeasures | 🛠️ Build #31089436 | 📁 Comparing 02a6223 against latest (d5bae41)

🔍 Preview build

Show files changed (20 files in total): 📝 19 modified | ➕ 0 added | ➖ 1 deleted

File	Status
changelog.html	📝 modified
genindex.html	📝 modified
py-modindex.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.assets.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.generic_assets.html	➖ deleted
_autosummary/flexmeasures.api.common.schemas.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.sensor_data.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.sensors.html	📝 modified
_autosummary/flexmeasures.api.common.schemas.users.html	📝 modified
_autosummary/flexmeasures.api.v3_0.assets.html	📝 modified
_autosummary/flexmeasures.api.v3_0.users.html	📝 modified
_autosummary/flexmeasures.data.schemas.account.html	📝 modified
_autosummary/flexmeasures.data.schemas.attributes.html	📝 modified
_autosummary/flexmeasures.data.schemas.generic_assets.html	📝 modified
_autosummary/flexmeasures.data.schemas.sensors.html	📝 modified
_autosummary/flexmeasures.data.schemas.times.html	📝 modified
_autosummary/flexmeasures.data.schemas.units.html	📝 modified
api/change_log.html	📝 modified
api/v3_0.html	📝 modified
cli/change_log.html	📝 modified

[nhoening]

Nice cleanup work, and good use of the inspect and importlib libraries to test this :)

Good for UX. And arguably this makes FlexMeasures a bit more secure as it leaks no internal information this way (well, the db structure was open source anyway).

I have just one request: Run make update-docs