Concurrent storage write batch support #425
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 0fb42f9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
rkistner
changed the title
This was referenced Dec 4, 2025
rkistner
force-pushed
the
concurrent-storage-batches
branch
from
6522374 to
841273e
Compare
Collaborator
stevensJourney
left a comment
stevensJourney
left a comment
There was a problem hiding this comment.
Looks really cool. I could not spot anything major which stands-out as being a potential issue.
| if (afterId) { | ||
| // Insert or update | ||
| const after_key: SourceKey = { g: this.group_id, t: sourceTable.id, k: afterId }; | ||
| const after_key: SourceKey = { g: this.group_id, t: sourceTable.id as bson.ObjectId, k: afterId }; |
Collaborator
There was a problem hiding this comment.
Shouldn't this type cast be using mongoTableId?
| $cond: [ | ||
| can_checkpoint, | ||
| { | ||
| $max: ['$last_checkpoint', { $literal: this.persisted_op }, { $toLong: '$keepalive_op' }] |
Collaborator
There was a problem hiding this comment.
I'm not sure about the specifics here, but I noticed the Postgres implementation coerces the, potentially null persisted_op to zero. Should we do that here also?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This modifies the APIs and implementation of
BucketStorageBatchto allow using multiple batches concurrently without introducing consistency issues. The first goal of this is to allow reading the replication stream while snapshotting - see #426 for a Postgres implementation.The actual writes to the database still rely on exclusive database locks to a large extent, so there isn't true concurrency yet. But it does allow running a replication stream concurrently with replication snapshots without breaking.
Specifically, this changes:
commit()andkeepalive(). The only remaining difference is that keepalive is essentiallycommit({allowEmptyCommit: true}).snapshot_doneis now explicitly set after completing a snapshot, rather than setting it automatically on commit. No commit will go through until snapshot_done is set.current_datadeletes are now soft-deletes (storing an empty document/row instead of deleting the document/row), converted into hard deletes on the next commit. We need this to consistently handle snapshots while streaming.SourceTable.idto be strongly typed, specifically checking forstringin Postgres storage andObjectIdin MongoDB storage. This unfortunately had a big knock-on effect in tests, including now having different checksums in postgres storage vs mongodb storage.Other more minor changes:
causetoReplicationAbortedError.