Enforce Schema validation and preserve null values

[ruizmaa]

This PR addresses issue #396 from scratch, superseding the PR #470

Main changes

1. Schema enforcement

• Schema requirement: Writing operations now strictly require a Schema object in dataset.structure. validate_schema_exists have been implemented to ensure it, raising an Invalid error if there is no schema in the dataset to convert.

2. No longer null conversion to empty strings

• Removed .replace({"nan": "", "<NA>": ""}), .fillna("") or .replace("nan", "") from the writers.
• Replaced pd.read_csv(na_values=[""], ...) to pd.read_csv(na_values=[], ...) from the readers.

3. Normalize nulls when reading

• Keeped XML readers (__ss_aux_reader.py) null normalization .replace(np.nan, "")
• Updated CSV readers null replacement from .replace({"nan": "", "<NA>": ""}) to .replace({"nan": "NaN", "<NA>": "NaN"})
• Added null normalization in generic reader as in non generic: .replace(np.nan, "") in __reading_generic_groups(), __reading_generic_series() and __reading_generic_all()

4. Deleted __remove_optional_attributes_empty_data() and added transform_dataframe_for_writing()

Now transform_dataframe_for_writing() recives a pd.DataFrame and a Schema and converts nulls and empty values based on if it's required or not.

It relies on auxiliary functions:

• _get_value_to_write: decides based the value's dtype and if it's required whether if it should be written (if it has any value or it's null), skip it (if it's not required and has an empty value "") or write null (if its originally null (NaN or #N/A) or if it's required and has an empty value ""). It uses more auxiliary functions
  • _is_null_value
  • _is_nullable_integer_dtype: a helper function to correctly write numbers format (avoid 1 to be written as 1.0, 1.0 as 1`)
  • _get_null_representation: returns "NaN" for numeric and "#N/A" for the others based on the NUMERIC_TYPES set:

NUMERIC_TYPES = {
    DataType.BIG_INTEGER,
    DataType.COUNT,
    DataType.DECIMAL,
    DataType.DOUBLE,
    DataType.FLOAT,
    DataType.INTEGER,
    DataType.LONG,
    DataType.SHORT,
}

So the new behavior is:

Original file value	Read value	Written value (CSV)	Written value (XML)
Required "NaN"	"NaN"	"NaN"	"NaN"
Required "#N/A"	"#N/A"	"#N/A"	"#N/A"
Required "" (CSV: ,, / XML: value="")	""	"#N/A" (if not numeric) / "NaN" (if numeric)	"#N/A" (if not numeric) / "NaN" (if numeric)
Required missing value (XML MissingTags)	""	"#N/A" (if not numeric) / "NaN" (if numeric)	"#N/A" (if not numeric) / "NaN" (if numeric)
Optional "NaN"	"NaN"	"NaN"	"NaN"
Optional "#N/A"	"#N/A"	"#N/A"	"#N/A"
Optional "" (CSV: ,, / XML: value="")	""	""	Skipped (no tag)
Optional missing value (XML MissingTags)	""	""	Skipped (no tag)

“Missing value” in XML (absent attribute tag) is normalized by the reader to an empty string "", so it follows the same rules as explicit empty values.

Tests

Fixed to fit into the new Schema requirement

CSV test writers

• tests/io/csv/sdmx10/test_writer_v1.py: added a Schema fixture and fixed samples files (reference.csv and reference_attch_atts.csv) to match with the schema's dataflow id
• tests/io/csv/sdmx20/test_writer_v2.py: added a Schema fixtures with context datastructure and another with context dataprovision. Fixed samples files (reference.csv, reference_attch_atts.csv and reference_with_action.csv) to match with the schema's dataflow id
• tests/io/csv/sdmx21/test_writer_v21.py: added a Schema fixtures with context datastructure and another with context dataprovision. Fixed samples files (reference.csv, reference_attch_atts.csv, reference_with_action.csv and reference_with_append_action.csv) to match with the schema's dataflow id

General tests

• tests/io/test_general_writer.py: added a Schema fixture
• tests/io/test_writing_validation:
  • Updated test_data_rwr_no_structure, now it should raise a Invalid error (because it has no structre and Schema it's required)
  • Added a Schema to test_read_xml_write_csv_10 and test_read_xml_write_csv_20

XML tests

• tests/io/xml/sdmx21/test/reader::test_read_write_structure_specific_all: added a Schema to the structure.
• tests/io/xml/sdmx21/test/writer
  • test_data_writing.py::test_data_scape_quote: added a Schema
  • Fixed samples files (gen_ser.xml and test_generic_with_groups.xml)
• tests/io/xml/sdmx30/test/reader::test_data_dataflow_31: added a Schema

Fixed to fit into the new null values handling

• tests/io/test_writing_validation.py
  • test_data_rwr and test_write_sdmx_csv_read_back: updated the spected data to fit the new null and empty handle
• tests/io/xmls/sdmx21/test_writing_validation::test_data_write_nullable_nulltypes: extended test to verify it works as spected
• tests/io/xmls/sdmx30/test_writing_validation::test_data_write_nullable_nulltypes: extended test to verify it works as spected

Added to test the new null values handling

[javihern98]

@ruizmaa Need to check conflicts on this PR, please address them 😊