014 - Allowing Downloads Of Filtered Data

[eveleighoj]

Status

Draf

Intro

The Planning Data platform allows either complete download of an entire dataset or using the API to file to only the relevant entities. The aPI is limited to 500 rows, is also expected to return in a timely fashion and is limited to json or geojson formats. This presents a probably when we want to provide custom download links for let's say a single organisations data as a csv.

to facilitate this we are proposing two additional changes:

• introducing a parquet file as one of the outputs for datasets - this allows analysts and us to query the parquet file as part of the download. allowing filtered versions of the dataset to be accessed
• for non analyst uses we suggest adding a lambda function to reproduce how an analyst can query the parquet file. this can be used to create download links for specific filters of. the data

Detail

This is the new suggested container diagram, it adds the download lambda which will be responsible for retrieving the filtered data and converting to the relevant format. This is expected to be a download to the user so it doesn't need to deliver the entire file instantly.

Creation of parquet files

We will need to alter the current baking process that takes place when we create 'flattened' representations of the datasets from the entity data. this code is ran via the collection task container shown below

this is reliant on the digital-land-python python package so changes will be. required across both the digital-land-python and collection-task repos.

Download Lambda Container

The new container will need to be added. The suggested repo has been created and can be found here. It's aim is to use duckdb to query files in s3.

Alerts & Monitoing

Sentry

Can sentry monitor lambda functions? this could help identify unhandled problems with the running of the lambda function.

AWS mettics

• request count on CDN
• Lambda failures?
• CDN Error status code metrics the standard set
• Lmbda. execution lengths
• sizes of downloaded data

AWS Alams

• identify AWS alarms that need to be set up. We have generic ones for the CDN to monitor requests
• Lambda average ttime

CI & CD

• need to examine how current lambda functions are stored and replicate
• should be too difticult to replicate

Security

• Duckdb is an extremely powerful tool which doesn't have an ORM so need to put extra protections on inputs to sanitise ttthemm and ensure parametrised queries
• review Duckdbs security guidance on any additional protections that can be added
• Following principlpe. of least privallage:
  • only needs read access to the collection data bucket. No access to other containers is required outside of read access
• Rate limiting should be applied via WAF/CDN

Performance & Scalabillity

• s3 and Lambda are extremely hoizontally scallable. So will easily be able meet demand forr increasing users. If anythinng this increased scalability needs rate limiting and other security considerations.
• by avoiding the db and platform app it won't affect the platform's API performance
• Vertical scaling problems might occur based on the size of data it's filtering and if filtering is not sufficient. There are two considerations hee:
  • the memory and storage limitations of a lambda function.
  • time limits of running lambda function

Testing

Duckdb can be. difficult to test when combined with s3. This is because the python API calls the local duckdb instance so python cannot mock objects easily which are accessed by duckdb.

There are two approaches:

• when testing just. replace s3 with local fillesystem calls. this tests the logic and assumes the inner. workings of duckdb can operate on an s3 bucket.
• use test containers to spin up a localstack container with the s3 bucket present. But this added complexity means custom duckdb instances may be needed.

Tests required: