Add bundle.engine config setting for deployment engine selection (#4749)

## Summary

Add a new `engine` field at the bundle/target level that allows users to
configure the deployment engine (`"terraform"` or `"direct"`) in their
bundle config.

### Behavior
- New bundles (no existing state): uses the configured engine, or
defaults to `terraform`
- Existing bundles (state present): if the configured engine doesn't
match state, a **warning** is issued pointing to the config source, and
the existing state's engine is used. Note, this is different from
current behaviour, where databricks CLI refuses to proceed with
config/state mismatch.
- Priority: state > `DATABRICKS_BUNDLE_ENGINE` env var > `bundle.engine`
config > default

### Example
```yaml
bundle:
  name: my-bundle
  engine: direct

targets:
  production:
    bundle:
      engine: terraform
```

## Test plan
Unit tests and acceptance tests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: denik

NEXT_CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## Release v0.295.0
 
+### Notable Changes
+
+- Add `bundle.engine` config setting to select the deployment engine (`terraform` or `direct`). The `DATABRICKS_BUNDLE_ENGINE` environment variable takes precedence over this setting. When the configured engine doesn't match existing deployment state, a warning is issued and the existing engine is used ([#4749](https://github.com/databricks/cli/pull/4749)).
+
 ### CLI
 
 ### Bundles

acceptance/bundle/deploy/readplan/terraform-error/output.txt

@@ -1,4 +1,4 @@
-Error: --plan is only supported with direct engine (set DATABRICKS_BUNDLE_ENGINE=direct)
+Error: --plan is only supported with direct engine (set bundle.engine to "direct" or DATABRICKS_BUNDLE_ENGINE=direct)
 
 
 Exit code: 1

acceptance/bundle/migrate/basic/out.plan_update.json

@@ -2,7 +2,7 @@
   "plan_version": 2,
   "cli_version": "[DEV_VERSION]",
   "lineage": "[UUID]",
-  "serial": 7,
+  "serial": 8,
   "plan": {
     "resources.jobs.test_job": {
       "action": "update",

acceptance/bundle/migrate/basic/output.txt

@@ -2,7 +2,7 @@
 >>> musterr [CLI] bundle deployment migrate
 Error: This command migrates the existing Terraform state file (terraform.tfstate) to a direct deployment state file (resources.json). However, no existing local or remote state was found.
 
-To start using direct engine, deploy with DATABRICKS_BUNDLE_ENGINE=direct env var set.
+To start using direct engine, set "engine: direct" under bundle in your databricks.yml or deploy with DATABRICKS_BUNDLE_ENGINE=direct env var set.
 
 >>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
 Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/dev/files...
@@ -27,26 +27,19 @@ To undo the migration, remove [TEST_TMP_DIR]/.databricks/bundle/dev/resources.js
 >>> jq .state."resources.pipelines.test_pipeline".state.tags.volume_storage_location out.new_state.json
 "s3://deco-uc-prod-isolated-aws-us-east-1/metastore/[UUID]/volumes/[UUID]"
 
->>> DATABRICKS_BUNDLE_ENGINE=terraform musterr [CLI] bundle plan
-Error: Required engine "terraform" does not match present state files. Clear "DATABRICKS_BUNDLE_ENGINE" env var to use engine appropriate for the state.
-
-Available state files:
-- terraform.tfstate: remote terraform state serial=4 lineage="[UUID]"
-- [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=6 lineage="[UUID]"
-
-
->>> DATABRICKS_BUNDLE_ENGINE=terraform musterr [CLI] bundle deploy
-Error: Required engine "terraform" does not match present state files. Clear "DATABRICKS_BUNDLE_ENGINE" env var to use engine appropriate for the state.
-
-Available state files:
-- terraform.tfstate: remote terraform state serial=4 lineage="[UUID]"
-- [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=6 lineage="[UUID]"
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 3 unchanged
 
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/migrate-basic-test/dev/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
 
 === Should show that it's already migrated
 >>> musterr [CLI] bundle deployment migrate
 Error: already using direct engine
-Details: [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=6 lineage="[UUID]"
+Details: [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=7 lineage="[UUID]"
 
 >>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle plan
 Plan: 0 to add, 0 to change, 0 to delete, 3 unchanged
@@ -93,14 +86,14 @@ Deployment complete!
 === Should show that it's already migrated
 >>> musterr [CLI] bundle deployment migrate
 Error: already using direct engine
-Details: [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=7 lineage="[UUID]"
+Details: [TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=8 lineage="[UUID]"
 
 >>> DATABRICKS_BUNDLE_ENGINE= [CLI] bundle debug states
-[TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=7 lineage="[UUID]"
+[TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=8 lineage="[UUID]"
 
 >>> DATABRICKS_BUNDLE_ENGINE= [CLI] bundle debug states --force-pull
-resources.json: remote direct state serial=7 lineage="[UUID]"
-[TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=7 lineage="[UUID]"
+resources.json: remote direct state serial=8 lineage="[UUID]"
+[TEST_TMP_DIR]/.databricks/bundle/dev/resources.json: local direct state serial=8 lineage="[UUID]"
 
 === Extra plan: should have no drift
 >>> DATABRICKS_BUNDLE_ENGINE= [CLI] bundle plan

acceptance/bundle/migrate/basic/script

@@ -14,9 +14,9 @@ rm out.backup_state.json
 # check that this is stored as resolved value, not as reference:
 trace jq '.state."resources.pipelines.test_pipeline".state.tags.volume_storage_location' out.new_state.json | contains.py 's3://'
 
-# this is expected to fail because state does not match env var
-trace DATABRICKS_BUNDLE_ENGINE=terraform musterr $CLI bundle plan
-trace DATABRICKS_BUNDLE_ENGINE=terraform musterr $CLI bundle deploy
+# this is expected to warn because state does not match env var, but still succeed using state engine
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle plan
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
 
 title "Should show that it's already migrated"
 trace musterr $CLI bundle deployment migrate

acceptance/bundle/migrate/dashboards/out.plan_after_migrate.json

@@ -2,7 +2,7 @@
   "plan_version": 2,
   "cli_version": "[DEV_VERSION]",
   "lineage": "[UUID]",
-  "serial": 3,
+  "serial": 4,
   "plan": {
     "resources.dashboards.dashboard1": {
       "action": "skip",

acceptance/bundle/migrate/dashboards/output.txt

@@ -17,21 +17,14 @@ The state file is not synchronized to the workspace yet. To do that and finalize
 To undo the migration, remove [TEST_TMP_DIR]/.databricks/bundle/default/resources.json and rename [TEST_TMP_DIR]/.databricks/bundle/default/terraform/terraform.tfstate.backup to [TEST_TMP_DIR]/.databricks/bundle/default/terraform/terraform.tfstate
 
 
->>> DATABRICKS_BUNDLE_ENGINE=terraform musterr [CLI] bundle plan
-Error: Required engine "terraform" does not match present state files. Clear "DATABRICKS_BUNDLE_ENGINE" env var to use engine appropriate for the state.
-
-Available state files:
-- terraform.tfstate: remote terraform state serial=1 lineage="[UUID]"
-- [TEST_TMP_DIR]/.databricks/bundle/default/resources.json: local direct state serial=3 lineage="[UUID]"
-
-
->>> DATABRICKS_BUNDLE_ENGINE=terraform musterr [CLI] bundle deploy
-Error: Required engine "terraform" does not match present state files. Clear "DATABRICKS_BUNDLE_ENGINE" env var to use engine appropriate for the state.
-
-Available state files:
-- terraform.tfstate: remote terraform state serial=1 lineage="[UUID]"
-- [TEST_TMP_DIR]/.databricks/bundle/default/resources.json: local direct state serial=3 lineage="[UUID]"
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
 
+>>> DATABRICKS_BUNDLE_ENGINE=terraform [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/files...
+Deploying resources...
+Updating deployment state...
+Deployment complete!
 
 >>> DATABRICKS_BUNDLE_ENGINE=direct [CLI] bundle plan
 Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged
@@ -54,11 +47,11 @@ Deployment complete!
 === Should show that it's already migrated
 >>> musterr [CLI] bundle deployment migrate
 Error: already using direct engine
-Details: [TEST_TMP_DIR]/.databricks/bundle/default/resources.json: local direct state serial=4 lineage="[UUID]"
+Details: [TEST_TMP_DIR]/.databricks/bundle/default/resources.json: local direct state serial=5 lineage="[UUID]"
 
 >>> DATABRICKS_BUNDLE_ENGINE= [CLI] bundle debug states
-[TEST_TMP_DIR]/.databricks/bundle/default/resources.json: local direct state serial=4 lineage="[UUID]"
+[TEST_TMP_DIR]/.databricks/bundle/default/resources.json: local direct state serial=5 lineage="[UUID]"
 
 >>> DATABRICKS_BUNDLE_ENGINE= [CLI] bundle debug states --force-pull
-resources.json: remote direct state serial=4 lineage="[UUID]"
-[TEST_TMP_DIR]/.databricks/bundle/default/resources.json: local direct state serial=4 lineage="[UUID]"
+resources.json: remote direct state serial=5 lineage="[UUID]"
+[TEST_TMP_DIR]/.databricks/bundle/default/resources.json: local direct state serial=5 lineage="[UUID]"

acceptance/bundle/migrate/dashboards/script

@@ -6,9 +6,9 @@ print_state.py > out.original_state.json
 trace $CLI bundle deployment migrate 2>&1 | contains.py 'Migrated 1 resources'
 print_state.py > out.new_state.json
 
-# this is expected to fail because state does not match env var
-trace DATABRICKS_BUNDLE_ENGINE=terraform musterr $CLI bundle plan
-trace DATABRICKS_BUNDLE_ENGINE=terraform musterr $CLI bundle deploy
+# this is expected to warn because state does not match env var, but still succeed using state engine
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle plan
+trace DATABRICKS_BUNDLE_ENGINE=terraform $CLI bundle deploy
 
 rm out.requests.txt
 # Both DATABRICKS_BUNDLE_ENGINE=direct and DATABRICKS_BUNDLE_ENGINE= work

acceptance/bundle/migrate/engine-config-direct/databricks.yml

@@ -0,0 +1,3 @@
+bundle:
+  name: test-migrate-engine
+  engine: direct