S3 loader

[grzegorzewald]

Heyah,

First of all, thank you very much for keeping Open Source version of best behavioural data pipeline.

At this time instant we, as early Snowplow adopters, are considering number of approaches we can take. Therefore I have a question: do you consider delivering open source version of S3 loader as well? This component is crucial for our pipelines and we would appreciate to have alternative.

The second: do you think of delivering binary artefacts of components?

[joaolcorreia]

Hi @grzegorzewald! Apologies for the delay in the response, I didn't get a notification. Yes S3 loader is a key component. The reason it wasn't forked yet is the license hasn't changed (yet).

I will say we want to simplify the pipeline, and loading is one of the key aspects we want to improve. We are already using the JSON output option to write to Kinesis and then use Firehose to write directly to S3 a flattened JSON.

Any decent data warehouses today can easily ingest JSON, so the idea is to make the OpenSnowcat less reliant on n-vendor loading components and bet on a format that anyone can ingest.

We do make the jars available for each release but we don't have docker images publicly available.

[grzegorzewald]

Hi @joaolcorreia,

Thanks for info. We are about to start updating our stack. Firehose feels very tempting, although it may complicate our Data Lake loading process. Will see :)

[joaolcorreia]

@grzegorzewald We are developing a new loader, please reach out and we can chat about it.