Getting pandas frame with meaningful dimensions

[Dan-Kechik]

Hello. The sdmxthon package is convenient to use, and getting pandas frame feature is great. But column names and dimension values are encoded: their names are according to message.content[DATASET_NAME].structure.content['dimensions'] elements. The same is with values. I would like to substitute columns names and values with such from metadata content. Is it some method for it?

[javihern98]

Hi @Dan-Kechik , need more details to understand your proposal:

• Are you referring to the use of the Dimensions from the DataStructureDefinition, if so, to replace the Dimension columns with its name or something similar?
• I cannot understand what do you refer by "substitute columns names and values with such from metadata content", the values in SDMX are normally related to a Codelist (via its CoreRepresentation of the related Concept or the LocalRepresentation), would you like to substitute the IDs of the codes with the referred value? (example: on Codelist FREQ, for Frequency, substitute A to Annual)

[Dan-Kechik]

Yes, I referred to Dimensions from the DataStructureDefinition. I'm new to SDMX, and I underlined here what I desired to replace:

It would be perfectly if one can select data in human readable manner relying on the DataStructureDefinition. This means not to select data where feature1234 == 5678, but use some method select(, ), e.g. select('Age', '0-4').
I'm looking through one more solution, and here SDMX message is converted to pandas frame with multiindex. As I understand (or as I want), depth of multiindex is equal to dimensions number, and index values are dimensions items.
https://pandasdmx.readthedocs.io/en/v1.0/api.html#pandasdmx.writer.pandas.write_dataset

[javihern98]

@Dan-Kechik Yes, this is a feature we are exploring to implement in the pysdmx library (Docs for pysdmx). We are adding there as well the latest SDMX 3.0 formats, VTL compatibility (a language to validate and transform data) and a toolkit to aid the user with these kind of functions, in a production-ready library. It has an easier model for these kind of use cases, as the names are simply strings.

In any case, let´s distinguish between two things:

• The name of the Dimension object, which gives the column name
• The name of the codelist, which gives the link between the value and the "human readable data"

For both of them, the overall idea is to set up a method that takes the following:

Dimension(id="FREQ", name="Frequency", ..., 
representation=Representation(codelist=Codelist(id="CL1", ..., items=[Code(id="A", name="Annual"), Code(id="M", name="Monthly"] ... ))

And this data:

DIM_1	FREQ	OBS_VALUE
XXXX	A	106
XXXX	M	210

And converts it into:

DIM_1	Frequency	OBS_VALUE
XXXX	Annual	106
XXXX	Monthly	210

It would be the same if the Dimension is related to a Concept by its "Concept Identity" and at the CoreRepresentation of the Concept it has the same Codelist, that is what you need to look into.

With that in mind, it should be easy to iterate over the components of the DataStructureDefinition and extract this information to generate the result you expect (and select the data using the Pandas selection methods). Another solution to simply use SQL over Pandas could be to use duckdb

Regarding the Multiindex, the combination of Dimensions values in the same row must be unique, so theoretically there should not be any problem to use it, in case you need it...

[Dan-Kechik]

Yes, it looks pretty. Thank you for your answers.