Skip to content

Shift application to a data lake architecture #12

@jayhill

Description

@jayhill

We don't yet have clear use cases around which to build APIs, design our DynamoDB table and so on.

Instead, we're going to store the data as Iceberg/Parquet files in S3 and use Athena for queries. We'll drop daily files throughout the month and then run a compaction job (OPTIMIZE) to put everything into a monthly file at EoM.

Partitioning will be by facility ('buncombe', 'wake', etc.) and year+month.

Rework Terraform and Serverless Framework IaC definitions to accommodate this architecture.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

Status

In Progress

Relationships

None yet

Development

No branches or pull requests

Issue actions