--quack mode

[MaxHalford]

At Carbonfact we would love to be able to have the option of running views locally with DuckDB. That would provide speedier iterations. We could also imagine pushing this, by calculating stuff with DuckDB, and pushing the tables to BigQuery for serving. For instance see this.

• https://davidgasquez.com/duckdb-bq-storage/

[MohamedBsh]

Hi @MaxHalford,

Thank you for this excellent suggestion. I'm excited to take on this issue and implement the DuckDB integration for local processing with the option to push results to BigQuery. This feature will indeed provide faster iterations and more flexibility for our data processing pipeline 🎉

Here's a summary of the actions I plan to take:

1. Add a new --quack-mode option to the CLI for running views locally with DuckDB.
2. Implement a DuckDBClient class that will handle local query execution and view materialization.
3. Add a --push-to-bigquery option to allow pushing processed data to BigQuery after local computation.
4. Leverage PyArrow extensively in this implementation:
   • Use PyArrow as an efficient intermediate format between DuckDB and BigQuery.
   • Utilize PyArrow's high-performance data transfer capabilities.
   • Take advantage of PyArrow's seamless integration with pandas for easy data manipulation.
5. Implement a push_to_bigquery method in the Runner class to handle the data transfer process.
6. Update the documentation to explain how to use these new features, including any necessary setup for DuckDB and BigQuery.
7. Add comprehensive unit tests to ensure the reliability of the new functionality.

I'll keep you updated on the progress :)

[MaxHalford]

This extension looks promising

[MaxHalford]

Relevant article

[MaxHalford]

This looks cool, but doesn't show how to write data with DuckDB

[MaxHalford]

Greybeam