Skip to content

feat(#190): version 2: add multi-db, batching and tags#192

Merged
witash merged 15 commits intomainfrom
v2
Mar 28, 2025
Merged

feat(#190): version 2: add multi-db, batching and tags#192
witash merged 15 commits intomainfrom
v2

Conversation

@witash
Copy link
Copy Markdown
Contributor

@witash witash commented Feb 19, 2025

This is ready for review
It wasn't really possible to separate the issues included, so I added one big PR instead of several small ones.

It contains

  • adding source to couchdb (this is a breaking change for cht-sync)
  • adding dbname and instance, relying on source to document_metadata for mulit-db and multi-
  • adds tags to base and telemetry models to allow users to separate the runs for base models and telemetry models if they want.
  • adds batch_incremental macro to allow for small batches for document_metadata

@witash witash changed the title V2 feat: V2 Mar 4, 2025
@witash witash changed the title feat: V2 feat(190): V2 Mar 4, 2025
@witash witash changed the title feat(190): V2 feat(#190): V2 Mar 4, 2025
@witash witash marked this pull request as ready for review March 4, 2025 12:32
@witash witash changed the title feat(#190): V2 feat(#190): version 2: add mulit-db, batching and tags Mar 5, 2025
@witash witash requested review from jkuester and mrjones-plip March 5, 2025 07:06
Copy link
Copy Markdown

@mrjones-plip mrjones-plip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't spent enough time with the code prior to v1 to be confident in doing an in depth code review. I'll defer to @jkuester for any deep code dives!

I'll be sure to do a fair amount of black box testing in the docs PR I'm working and suggest we wait to have both PRs approved before merging. For example, there's no upgrade path documented and I'm not sure if this will impose additional changes on this v2 PR

otherwise approving to unblock!

Copy link
Copy Markdown

@jkuester jkuester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! Just one big question about our batching logic. At the end of the day duping less than 1000 rows in a 100_000 row batch is probably not too big of a deal, but maybe we can avoid it....

Comment thread macros/batch_incremental.sql Outdated
Comment thread models/root/tests/document_metadata.yml Outdated
Comment thread macros/batch_incremental.sql Outdated
Comment thread tests/run_dbt_tests_docker.sh Outdated
@witash witash changed the title feat(#190): version 2: add mulit-db, batching and tags feat(#190): version 2: add multi-db, batching and tags Mar 7, 2025
@witash witash merged commit d2067b6 into main Mar 28, 2025
5 checks passed
@witash witash deleted the v2 branch March 28, 2025 09:46
medic-ci pushed a commit that referenced this pull request Mar 28, 2025
# [2.0.0](v1.4.0...v2.0.0) (2025-03-28)

### Features

* **#190:** version 2: add multi-db, batching and tags ([#192](#192)) ([d2067b6](d2067b6)), closes [#190](#190) [#172](#172) [#190](#190) [#156](#156)

### BREAKING CHANGES

* **#190:** batching, instance and dbname added to document_metadata
@medic-ci
Copy link
Copy Markdown

🎉 This PR is included in version 2.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants