SEP data workflow: Italian census data

[FranckCo]

Italian census data is currently produced manually. Explore possibilities of automation.

[vaccaricarlo]

Mail from Paolo confirms that it's not possible to automate the process for Census data

[FranckCo]

OK, but we still can do better than mail: have a documented manual procedure to produce the data and a fixed URL where they can be obtained.

[pafrance]

OK, but we still can do better than mail: have a documented manual procedure to produce the data and a fixed URL where they can be obtained.

Here's the permanent census data source
You can browse the data manually, customize your query and export data using the provided toolbar.
I also found a conctat page where it's possible to request data on demand, but it's still a manual task.

I would like if any data provider had a direct connection to datasets, but the reality is that any data provider out there has their own quirks and habits in their own data publication.
I think it's good to build a protocol with specifications on how to GET data from data providers or let them POST data in our repository.
Hera are some examples:

1. Auto-browsing: Whenever data are presented with an interface easy to browse, just like simple HTML or some other easy to browse format so that data extraction can be made automatically with a spider (just like google's)
2. Human-Interaction: Istat census is well made and all, but, alas it's made for HUMAN interaction.
   This is another case of study: Has it any meaning to devise a spider to browse such interfaces automatically? I don't even know if there actually are such devices at all.
3. Data provider initiative: We could devise a module in our system to ease data source referrals officers to post data on our site in a standardized approach.
   But we can never expect complete M2M compliance from data providers, at least for the foreseeable future.

[FranckCo]

I repeat, the procedure should (even if it is manual):

• be documented
• produce the expected file (the last one sent by Pina)
• make the file available at a predictable URL where it can be retrieved by an HTTP GET

[pafrance]

Unfortunately, the interface is not machine readable. It cannot generate an url to get the file. So no. The only solution is try to get machine readable data from another source. What about the SEP from Eurostat?

…

----- Messaggio originale ----- Da: "Franck Cotton" ***@***.***> A: "INTERSTAT/Statistics-Contextualized" ***@***.***> Cc: "Paolo Francescangeli" ***@***.***>, "Comment" ***@***.***> Inviato: Venerdì, 12 novembre 2021 13:49:35 Oggetto: Re: [INTERSTAT/Statistics-Contextualized] SEP data workflow: Italian census data (Issue #9) I repeat, the procedure should (even if it is manual): - be documented - produce the expected file (the last one sent by Pina) - make the file available at a predictable URL where it can be retrieved by an HTTP GET

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://urlsand.esvalabs.com/?u=https%3A%2F%2Fgithub.com%2FINTERSTAT%2FStatistics-Contextualized%2Fissues%2F9%23issuecomment-967095627&e=17c5563b&h=4b978cb1&f=n&p=y

[pafrance]

Italian census data retrieval: main steps and workflow
Census Data extraction
Step 1: Download from Istat source website
Step2: browse CENSUS OF POPULATION AND HOUSING from the left toolbar
Step3: Select Population/Demographic characteristics and citizenship/Age structure – municipalities
Step4: Select Customise/Table options from the top toolbar and a pop-up window appears
Step5: Select from panel Dimension Member Labels/All dimensions/Use codes and then View data
Step6: Select Customise/selection/Select time from the top toolbar and a pop-up window appears
Step7: Select date range (2018-2018) and then View data
Step8: Select Export/csv from the top toolbar and then select Download from the pop-up window
The downloaded file is not compliant with the required DSD.

Data transformation
The downloaded file has the following Data Structure:
ITTER107,"Territory","TIPO_DATO_CENS_POP","Datatype","SEXISTAT1","Gender","ETA1","Age class","TIME","Select time","Value","Flag Codes","Flags"

1. Data need to be filtered in order to obtain the requested Data Structure ,
2. NUTS3 variable has been added through a transformation from ITTER107 Variable, using data from ISTAT LAU archive
3. Provided metadata for NUTS3 transformation need to be downloaded from Istat website and merged.
   Metadata are referenced in a time series and Variable regarding year 2018 has been used in the script.
4. Sex codelist needs to be translated according to the standard.
   Data have been transformed through an R script provided as download together with the present documentation.

Data Load
The transformed file was uploaded into INTERSTAT GraphDB.
GraphDB allows direct link to the resources by a GET permalink , but the raw data needs a little reworking to be accessed directly.
It can be downloaded rewriting the POST URL using the ID in the permalink

Transformation script in R language
Pilot A - census data processing.txt

[francescadag]

The source file from Italian Census has been uploaded in the FTP area of the project.
As requested, metadata files for conversion from ISTAT territorial codes to LAU and NUTS3 has been uploaded to GitHub.
In addition to the metadata, the Italian NUT3 has been uploaded to GitHub as well.