Bypassing the stream_is_selected If Operation

[alexguanga]

Description of change

• Introduced functionality to bypass the stream_is_selected by passing an argument in the config.json when running the --properties flag.

• Desired Exact code change

• Why Are We Proposing the Change?

  • At Cherre, we are working with Taps & Target for our ingestion. There's still work that needs to be done before we fully integrate with Taps & Target. One functionality we do that is slightly different than Singer's Taps and Target implementation is that we are not using the --discover nor the --properties flag. Instead, we are piping the data from the Tap to the Target with the --config flag directly.
  • In order to only using the --config, we need to execute the functionality of the --discover and --properties in one sync and not two.

• About the Desired Code Change

  • Specifying true for the skip_stream_is_selected in the config.sample.json, we are able to bypass the --properties and --discover flag.
  • Under the hood, we are still performing the functionality that is executed from --properties and --discover flag. The difference is that we performing this with only the --config flag. To be able to this, we can either:
    1. Update the JSON that is produced from --properties
    2. PROPOSED: By adding a stream_is_selected flag, it will bypass the stream_is_selected if operation.

• Workflow
  - Please note that the workflow is an addition to what's provided by singer's io tap-s3-csv, it does not affect any current workflows.

  • Set skip_stream_is_selected to true in the config.json
  • CLI Command: tap-s3-csv --config config.json

Manual QA steps

• N/A

Risks

• Don't think there's any. Please let me know if you anything is apparent to the team @cmerrick

Rollback steps

• revert this branch

[cmerrick]

Hi @alexguanga, thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes.

[cmerrick]

You did it @alexguanga!

Thank you for signing the Singer Contribution License Agreement.

[alexguanga]

@cmerrick - Following up on the PR, is there any additional specification/requirements I should follow? Thanks!

[vianel]

Hello, Any news on this?

[alexguanga]

Hello, Any news on this?

Nope, have not heard any updates.