Goal: Facilitate Pipeline Creation

[outerlook]

There are some repetitive parts of a general pipeline which could be further improved.

Right now, we provide building blocks for different ingestion structures. But we could further improve it with the correct abstractions on the pipeline execution. Examples:

• Data reconciliation strategies
  • By last inserted (example)
  • Full compare
  • Gap filling
  • By key
  • ...
• Data Source abstract class
  • list_all
  • get_all
  • get_by_date
  • get_by_key (example: https://github.com/trufnetwork/adapters/blob/main/src/tsn_adapters/tasks/argentina/provider.py#L23 but a lot to improve)
• Transformer abstract class to prepare for TN (example)

and it could be well structured to require only the correct methods depending on the strategies. The intention is to create a well-tested generic structure that makes it easier to include new adapters