Add WAL for direct deployment state recovery#4106
Open
varundeepsaini wants to merge 9 commits intodatabricks:mainfrom
Open
Add WAL for direct deployment state recovery#4106varundeepsaini wants to merge 9 commits intodatabricks:mainfrom
varundeepsaini wants to merge 9 commits intodatabricks:mainfrom
Conversation
varundeepsaini
requested review from
andrewnester,
anton-107,
denik,
pietern and
shreyas-goenka
as code owners
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
2 times, most recently
from
b51a199 to
7f67cb2
Compare
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
7f67cb2 to
baf371e
Compare
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
86b90ce to
98a1893
Compare
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
98a1893 to
5cb5da4
Compare
Contributor
Author
|
@denik @andrewnester the pr is ready for review |
Contributor
Author
|
@denik @andrewnester bumping again ^^ |
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
5cb5da4 to
0e7c9fa
Compare
Contributor
Author
|
@denik fixed the build failures, could you approve the workflows |
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
1b3af30 to
34e0f37
Compare
Contributor
Author
|
@denik could you approve the workflow |
Contributor
Author
|
@andrewnester @denik bump ^^ |
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
2 times, most recently
from
b6f651d to
f509784
Compare
denik
reviewed
Feb 3, 2026
Contributor
denik
left a comment
denik
left a comment
There was a problem hiding this comment.
Thanks - looks good. Left a few comments.
Contributor
Author
|
@denik could you approve the workflows |
Collaborator
|
Commit: 6b1c292
17 interesting tests: 10 SKIP, 7 RECOVERED
Top 24 slowest tests (at least 2 minutes):
|
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
1485ae4 to
1154360
Compare
Contributor
Author
|
tests finally ran |
Contributor
Author
|
@denik bump ^^ |
1 similar comment
Contributor
Author
|
@denik bump ^^ |
denik
reviewed
Mar 19, 2026
acceptance/bundle/artifacts/artifact_upload_with_no_library_reference/test.toml
Outdated
Show resolved
Hide resolved
acceptance/bundle/artifacts/shell/default/out.deploy.direct.txt
Outdated
Show resolved
Hide resolved
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
1154360 to
552e3e5
Compare
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
4 times, most recently
from
f34406a to
63b765b
Compare
Contributor
Author
|
@denik could you please run the ci |
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
63b765b to
79e3593
Compare
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Signed-off-by: Varun Deep Saini <deepsainivarun@gmail.com>
varundeepsaini
force-pushed
the
feature/deploy-append-log
branch
from
79e3593 to
6e12ee1
Compare
Contributor
Author
|
@denik i looked at the test failures, rebasing should solve them, have done that. |
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Contributor
Author
|
@denik |
denik
reviewed
Mar 27, 2026
| if len(plan.Plan) == 0 { | ||
| // Avoid creating state file if nothing to deploy | ||
| if b.StateDB.RecoveredFromWAL() { | ||
| if err := b.StateDB.Finalize(); err != nil { |
Contributor
There was a problem hiding this comment.
Why do we need to call Finalize() here? We already called DeploymentState.Open() which already saved the state, right?
denik
reviewed
Mar 27, 2026
| ] | ||
|
|
||
| [[Repls]] | ||
| Old = 'Updating deployment state...\n' |
Contributor
There was a problem hiding this comment.
When I said we should not have this difference I meant it should not be printed in the first place, not that we should hide it :)
Contributor
|
Hi @varundeepsaini thanks a lot, this is great work. There are additional considerations, e.g. recently added state migration. I'm going to take over this PR as I'd like to have WAL functionality in. |
Contributor
Author
|
Sure @denik |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #4090
Changes
Add write-ahead log (WAL) to record state changes during direct deployment, enabling recovery of partial state if deployment is interrupted.
Added an Offset To KillCaller. Now it starts killing the process after Offset Successful requests to the endpoint
Why
Today, if deployment is interrupted before
Finalize(), no state is saved, and created resources become orphaned. The WAL writes each state change immediately to disk and replays them on restart.Tests
Tests added for WAL save/replay, delete/replay, finalize cleanup, and edge cases.