migration to Swiss Stats Explorer API

[JonasPipoz]

Hi!

BFS is currently migrating its STAT-TAB data portal to the new Swiss Stats Explorer platform. This transition also involves moving away from the legacy PX-Web technology toward a new API infrastructure, I guess.

Given that the current version of the BFS R package is designed to interface with the PX-Web structure, I would like to open a discussion on the following points:

• Support for the new Swiss Stats Explorer API

  Is there a plan to update the BFS package to support the new API?

• Backward Compatibility

  Will the package maintain support for PX-Web during a transition period?

  Will both APIs be supported in parallel, or will this require a breaking change?

This change is quite relevant for users relying on the package for reproducible statistical workflows and automated data retrieval. I would be happy to contribute or assist in adapting the package.

Looking forward to hearing your thoughts.

Best regards,
Jonas Pipoz
Service de Statistique Neuchâtel

[madelhorius]

I would also be interested in that. It seems that the first datasets we used have already been moved. Is anyone already working on that issue?

The easiest option would probably be to introduce new functions to interact with Swiss Stats Explorer. The new API works completely differently to STAT-TAB. Therefore, I would not adapt the existing functions, but rather introduce two new ones to retrieve data and metadata from Swiss Stats Explorer. What do you think about that?

Best regards
Marius
Fachstelle für Daten und Statistik, Kanton Zug

[lgnbhl]

Hello,

Thank you for opening this discussion on the migration to the Swiss Stats Explorer API. And sorry for the late answer, I was very busy (and took some holiday time).

I can confirm that I am planning to work on updating the BFS package to support the new API. I still haven't started working on this issue but I am planning to work on it relatively soon.

If the new API works completely differently to STAT-TAB, I agree that new functions could be an easier and safer way to migrate to the new API than trying to update the existing functions. But happy to discuss this further with you and have your thoughts about it.

Regarding reproducibility, we might want to create a detailed documentation page discussing possible breaking changes and differences between the R functions using STAT-TAB and the new functions interacting with the Swiss Stats Explorer API.

@JonasPipoz Thank you very much for offering to contribute and assist in adapting the package Jonas. I would be happy to discuss more how we can collaborate for the API migration.

@madelhorius Thank you very much for your interest, feedback and suggestions to introduce new functions Marius. Would you also be interested in collaborating to develop and test the new functions to interact with the Swiss Stats Explorer API?

If we agree that new functions should be developed (happy to discuss this in detail), then the next step can be to create a new GitHub branch to work together in developing the new functions. What do you think about this? Then I think we will have to create tests to check all possible differences between the STAT-TAB functions and the new functions which we will detail in a new page of the documentation.

What are your thoughts about all this? Very happy to hear any suggestions and comments.

Best,
Félix
Fachstelle für Statistik, Kanton St.Gallen

[madelhorius]

I would be happy to collaborate on implementing the new functions and I agree with the proposed approach.

[madelhorius]

Hi Félix,

Now that another dataset we use has moved to Swiss Stats Explorer, I have started developing meaningful functions to retrieve data from it. I forked your repository to work on it. I have developed two functions:

• bfs_get_sse_metadata(): Get codelists for a certain dataset based on the BFS_Number.
• bfs_get_sse_data(): Get data based on the BFS_Number.

I tried to make them as similar as possible to the existing functions, 'bfs_get_data()' and 'bfs_get_metadata()'. They do not work in a similar way in the background, but they should work in a similar way for the user.

Swiss Stats Explorer uses the SDMX format, and there is an R package that can retrieve SDMX-formatted data (https://github.com/eblondel/rsdmx). For now, I have decided not to use it, as I did not want to add another dependency. Honestly, I was not convinced by the package. In the README, they write:

In spite they are some R package initiatives relying on rsdmx that aim to provide a wrapper for a single data source (e.g. OECD, EUROSTAT), it is strongly recommended to rely directly on rsdmx. Indeed, one main objective of rsdmx is to promote and facilitate collating scattered data from a growing number of SDMX data providers, whatever the organization.

So, I decided to write the functions without relying on that package. I always worked with the XML format in the functions as the JSON retrieved from the API looked like a nightmare to me. The functions are fairly stable, but not very fast, as the metadata contains many variables that are not actually in the dataset. To get the data, I have to call the API three times.

1. Get the API URL based on the BFS number.
2. Get metadata.
3. Get the data.

Without using a query, this process can be very slow as the datasets can be very large. Therefore, using a query is highly recommended when calling the function bfs_get_sse_data().

I have not opened a PR yet as I do not want to merge directly onto your main branch. If you would like to test the functions, please take a look here: https://github.com/madelhorius/BFS/tree/stats_explorer.

I look forward to hearing from you!

Marius