[Discussion] Look into using Azure Data Lake

[olemp]

• I have searched for existing issues to avoid creating duplicates

TLDR

Azure Table Storage doesn't work too well with PowerBI and report in general. All the data needs to be refreshed every time. With Puzzlepart we're already at 10s of thousands of time entries that needs to be fetched each refresh. Look into using Azure Data Lake to improve the reporting.

Description

Azure Data Lake can give us faster reporting doing some magic that @joslange can elaborate on.

Azure Data Lake is built on Azure Blob storage, which is the Microsoft object storage solution for the cloud. The solution matifeatures low-cost, tiered storage and high-availability/disaster recovery capabilities. It integrates with other Azure services, including Azure Data Factory, which is a tool for creating and running extract, transform and load (ETL) and extract, load and transform (ELT) processes.

What is Azure Data Lake?

Status

@joslange will invite us to a meeting discussing usage of Data Lake/Data Factory in Did. 🚀

[joslange]

I only have general knowledge of Data Lake, but in my general understanding it shouldn't be too complex scheduling incremental export jobs from Table Storage to a Lake using Data Factory. Once we have data in the Lake, start building standardized datasets in Power BI for reporting instead of creating reports pulling data straight from Table storage. I'd be happy to help set up a test from dev, just need someone with access to creating Azure services.

[olemp]

Moved to https://github.com/Puzzlepart/did/discussions/867