Skip to content

Commit f2f1ea9

Browse files
pieternclaude
pietern
and
claude
authored
Add Lakebase Autoscaling support for Databricks Asset Bundles (#4423)
## Changes This PR adds support for Lakebase Autoscaling resources, enabling declarative configuration of projects, branches, and endpoints. For more information about the product, see https://docs.databricks.com/aws/en/oltp/projects/. These APIs use a "spec" field to capture the user intent in requests and a "status" field to contain the applied values in responses. To match DABs UX patterns, this PR embeds the "spec" type at the top level of the resource. ## Example Example configuration: ```yaml resources: postgres_projects: my_db: project_id: dabs-test-prod-app display_name: "Production Database" pg_version: 17 postgres_branches: main: parent: ${resources.postgres_projects.my_db.id} branch_id: main is_protected: false no_expiry: true postgres_endpoints: primary: parent: ${resources.postgres_branches.main.id} endpoint_id: primary endpoint_type: ENDPOINT_TYPE_READ_WRITE autoscaling_limit_min_cu: 0.5 autoscaling_limit_max_cu: 4 ``` After deploying this, you can access the endpoint using the `psql` command (see #4399): ``` % databricks psql projects/dabs-test-prod-app/branches/main/endpoints/primary -- -c 'select 1;' Project: Production Database Branch: main Endpoint: primary Connecting to read-write endpoint... ?column? ---------- 1 (1 row) ``` **Note:** the default database name assumed by the command is `databricks_postgres`. If this doesn't work, you can try `--dbname lakebase`. ## Tests * Acceptance tests on AWS. * API coverage in the testserver. * Manual testing. # Known limitations * Resource IDs are required: project_id, branch_id, and endpoint_id must be explicitly specified. They're planned to be optional and auto-generated by the server. * ID field updates in Terraform don't trigger recreation as expected. * Endpoint reconciliation conflicts prevent reliable Terraform-based recreate tests for endpoints. --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
1 parent 05f860a commit f2f1ea9

File tree

193 files changed

+7328
-6
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

193 files changed

+7328
-6
lines changed

‎NEXT_CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111

1212
### Bundles
1313

14+
* Add support for Lakebase resources (postgres_projects, postgres_branches, postgres_endpoints) ([#4423](https://github.com/databricks/cli/pull/4423))
1415
* Add missing values to SchemaGrantPrivilege enum ([#4380](https://github.com/databricks/cli/pull/4380))
1516
* Added support for UC catalogs (only in direct mode) ([#4342](https://github.com/databricks/cli/pull/4342))
1617
* engine/direct: Fix updates to fields being ignored for database_catalogs, synced_database_tables (([#4388](https://github.com/databricks/cli/pull/4388)))

‎acceptance/bin/read_id.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,8 @@ def get_id_terraform(filename, name):
2828
if r_name == name:
2929
for inst in r["instances"]:
3030
attribute_values = inst.get("attributes") or {}
31-
return attribute_values.get("id")
31+
# Try "id" first, fall back to "name" for postgres resources
32+
return attribute_values.get("id") or attribute_values.get("name")
3233

3334
print(f"Cannot find resource with {name=}. Available: {available}", file=sys.stderr)
3435

‎acceptance/bundle/invariant/configs/postgres_branch.yml.tmpl‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
resources:
5+
postgres_projects:
6+
project:
7+
project_id: test-pg-project-$UNIQUE_NAME
8+
display_name: Test Postgres Project
9+
10+
postgres_branches:
11+
foo:
12+
parent: ${resources.postgres_projects.project.name}
13+
branch_id: test-branch-$UNIQUE_NAME
14+
no_expiry: true

‎acceptance/bundle/invariant/configs/postgres_endpoint.yml.tmpl‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
resources:
5+
postgres_projects:
6+
project:
7+
project_id: test-pg-project-$UNIQUE_NAME
8+
display_name: Test Postgres Project
9+
10+
postgres_branches:
11+
branch:
12+
parent: ${resources.postgres_projects.project.name}
13+
branch_id: test-branch-$UNIQUE_NAME
14+
no_expiry: true
15+
16+
postgres_endpoints:
17+
foo:
18+
parent: ${resources.postgres_branches.branch.name}
19+
endpoint_id: test-endpoint-$UNIQUE_NAME
20+
endpoint_type: ENDPOINT_TYPE_READ_WRITE

‎acceptance/bundle/invariant/configs/postgres_project.yml.tmpl‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
resources:
5+
postgres_projects:
6+
foo:
7+
project_id: test-pg-project-$UNIQUE_NAME
8+
display_name: Test Postgres Project

‎acceptance/bundle/invariant/migrate/out.test.toml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/no_drift/out.test.toml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/test.toml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,9 @@ EnvMatrix.INPUT_CONFIG = [
3030
"model.yml.tmpl",
3131
"model_serving_endpoint.yml.tmpl",
3232
"pipeline.yml.tmpl",
33+
"postgres_branch.yml.tmpl",
34+
"postgres_endpoint.yml.tmpl",
35+
"postgres_project.yml.tmpl",
3336
"registered_model.yml.tmpl",
3437
"schema.yml.tmpl",
3538
"secret_scope.yml.tmpl",
@@ -40,6 +43,11 @@ EnvMatrix.INPUT_CONFIG = [
4043
[EnvMatrixExclude]
4144
no_alert_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=alert.yml.tmpl"]
4245

46+
# Postgres resources only work on AWS
47+
no_postgres_project_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_project.yml.tmpl"]
48+
no_postgres_branch_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_branch.yml.tmpl"]
49+
no_postgres_endpoint_on_cloud = ["CONFIG_Cloud=true", "INPUT_CONFIG=postgres_endpoint.yml.tmpl"]
50+
4351
# Fake SQL endpoint for local tests
4452
[[Server]]
4553
Pattern = "POST /api/2.0/sql/statements/"

‎acceptance/bundle/refschema/out.fields.txt‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3593,6 +3593,131 @@ resources.pipelines.*.permissions.permissions[*].group_name string ALL
35933593
resources.pipelines.*.permissions.permissions[*].permission_level iam.PermissionLevel ALL
35943594
resources.pipelines.*.permissions.permissions[*].service_principal_name string ALL
35953595
resources.pipelines.*.permissions.permissions[*].user_name string ALL
3596+
resources.postgres_branches.*.branch_id string INPUT STATE
3597+
resources.postgres_branches.*.create_time *time.Time REMOTE
3598+
resources.postgres_branches.*.expire_time *time.Time INPUT STATE
3599+
resources.postgres_branches.*.id string INPUT
3600+
resources.postgres_branches.*.is_protected bool INPUT STATE
3601+
resources.postgres_branches.*.lifecycle resources.Lifecycle INPUT
3602+
resources.postgres_branches.*.lifecycle.prevent_destroy bool INPUT
3603+
resources.postgres_branches.*.modified_status string INPUT
3604+
resources.postgres_branches.*.name string REMOTE
3605+
resources.postgres_branches.*.no_expiry bool INPUT STATE
3606+
resources.postgres_branches.*.parent string ALL
3607+
resources.postgres_branches.*.source_branch string INPUT STATE
3608+
resources.postgres_branches.*.source_branch_lsn string INPUT STATE
3609+
resources.postgres_branches.*.source_branch_time *time.Time INPUT STATE
3610+
resources.postgres_branches.*.spec *postgres.BranchSpec REMOTE
3611+
resources.postgres_branches.*.spec.expire_time *time.Time REMOTE
3612+
resources.postgres_branches.*.spec.is_protected bool REMOTE
3613+
resources.postgres_branches.*.spec.no_expiry bool REMOTE
3614+
resources.postgres_branches.*.spec.source_branch string REMOTE
3615+
resources.postgres_branches.*.spec.source_branch_lsn string REMOTE
3616+
resources.postgres_branches.*.spec.source_branch_time *time.Time REMOTE
3617+
resources.postgres_branches.*.spec.ttl *duration.Duration REMOTE
3618+
resources.postgres_branches.*.status *postgres.BranchStatus REMOTE
3619+
resources.postgres_branches.*.status.current_state postgres.BranchStatusState REMOTE
3620+
resources.postgres_branches.*.status.default bool REMOTE
3621+
resources.postgres_branches.*.status.expire_time *time.Time REMOTE
3622+
resources.postgres_branches.*.status.is_protected bool REMOTE
3623+
resources.postgres_branches.*.status.logical_size_bytes int64 REMOTE
3624+
resources.postgres_branches.*.status.pending_state postgres.BranchStatusState REMOTE
3625+
resources.postgres_branches.*.status.source_branch string REMOTE
3626+
resources.postgres_branches.*.status.source_branch_lsn string REMOTE
3627+
resources.postgres_branches.*.status.source_branch_time *time.Time REMOTE
3628+
resources.postgres_branches.*.status.state_change_time *time.Time REMOTE
3629+
resources.postgres_branches.*.ttl *duration.Duration INPUT STATE
3630+
resources.postgres_branches.*.uid string REMOTE
3631+
resources.postgres_branches.*.update_time *time.Time REMOTE
3632+
resources.postgres_branches.*.url string INPUT
3633+
resources.postgres_endpoints.*.autoscaling_limit_max_cu float64 INPUT STATE
3634+
resources.postgres_endpoints.*.autoscaling_limit_min_cu float64 INPUT STATE
3635+
resources.postgres_endpoints.*.create_time *time.Time REMOTE
3636+
resources.postgres_endpoints.*.disabled bool INPUT STATE
3637+
resources.postgres_endpoints.*.endpoint_id string INPUT STATE
3638+
resources.postgres_endpoints.*.endpoint_type postgres.EndpointType INPUT STATE
3639+
resources.postgres_endpoints.*.id string INPUT
3640+
resources.postgres_endpoints.*.lifecycle resources.Lifecycle INPUT
3641+
resources.postgres_endpoints.*.lifecycle.prevent_destroy bool INPUT
3642+
resources.postgres_endpoints.*.modified_status string INPUT
3643+
resources.postgres_endpoints.*.name string REMOTE
3644+
resources.postgres_endpoints.*.no_suspension bool INPUT STATE
3645+
resources.postgres_endpoints.*.parent string ALL
3646+
resources.postgres_endpoints.*.settings *postgres.EndpointSettings INPUT STATE
3647+
resources.postgres_endpoints.*.settings.pg_settings map[string]string INPUT STATE
3648+
resources.postgres_endpoints.*.settings.pg_settings.* string INPUT STATE
3649+
resources.postgres_endpoints.*.spec *postgres.EndpointSpec REMOTE
3650+
resources.postgres_endpoints.*.spec.autoscaling_limit_max_cu float64 REMOTE
3651+
resources.postgres_endpoints.*.spec.autoscaling_limit_min_cu float64 REMOTE
3652+
resources.postgres_endpoints.*.spec.disabled bool REMOTE
3653+
resources.postgres_endpoints.*.spec.endpoint_type postgres.EndpointType REMOTE
3654+
resources.postgres_endpoints.*.spec.no_suspension bool REMOTE
3655+
resources.postgres_endpoints.*.spec.settings *postgres.EndpointSettings REMOTE
3656+
resources.postgres_endpoints.*.spec.settings.pg_settings map[string]string REMOTE
3657+
resources.postgres_endpoints.*.spec.settings.pg_settings.* string REMOTE
3658+
resources.postgres_endpoints.*.spec.suspend_timeout_duration *duration.Duration REMOTE
3659+
resources.postgres_endpoints.*.status *postgres.EndpointStatus REMOTE
3660+
resources.postgres_endpoints.*.status.autoscaling_limit_max_cu float64 REMOTE
3661+
resources.postgres_endpoints.*.status.autoscaling_limit_min_cu float64 REMOTE
3662+
resources.postgres_endpoints.*.status.current_state postgres.EndpointStatusState REMOTE
3663+
resources.postgres_endpoints.*.status.disabled bool REMOTE
3664+
resources.postgres_endpoints.*.status.endpoint_type postgres.EndpointType REMOTE
3665+
resources.postgres_endpoints.*.status.hosts *postgres.EndpointHosts REMOTE
3666+
resources.postgres_endpoints.*.status.hosts.host string REMOTE
3667+
resources.postgres_endpoints.*.status.pending_state postgres.EndpointStatusState REMOTE
3668+
resources.postgres_endpoints.*.status.settings *postgres.EndpointSettings REMOTE
3669+
resources.postgres_endpoints.*.status.settings.pg_settings map[string]string REMOTE
3670+
resources.postgres_endpoints.*.status.settings.pg_settings.* string REMOTE
3671+
resources.postgres_endpoints.*.status.suspend_timeout_duration *duration.Duration REMOTE
3672+
resources.postgres_endpoints.*.suspend_timeout_duration *duration.Duration INPUT STATE
3673+
resources.postgres_endpoints.*.uid string REMOTE
3674+
resources.postgres_endpoints.*.update_time *time.Time REMOTE
3675+
resources.postgres_endpoints.*.url string INPUT
3676+
resources.postgres_projects.*.create_time *time.Time REMOTE
3677+
resources.postgres_projects.*.default_endpoint_settings *postgres.ProjectDefaultEndpointSettings INPUT STATE
3678+
resources.postgres_projects.*.default_endpoint_settings.autoscaling_limit_max_cu float64 INPUT STATE
3679+
resources.postgres_projects.*.default_endpoint_settings.autoscaling_limit_min_cu float64 INPUT STATE
3680+
resources.postgres_projects.*.default_endpoint_settings.no_suspension bool INPUT STATE
3681+
resources.postgres_projects.*.default_endpoint_settings.pg_settings map[string]string INPUT STATE
3682+
resources.postgres_projects.*.default_endpoint_settings.pg_settings.* string INPUT STATE
3683+
resources.postgres_projects.*.default_endpoint_settings.suspend_timeout_duration *duration.Duration INPUT STATE
3684+
resources.postgres_projects.*.display_name string INPUT STATE
3685+
resources.postgres_projects.*.history_retention_duration *duration.Duration INPUT STATE
3686+
resources.postgres_projects.*.id string INPUT
3687+
resources.postgres_projects.*.lifecycle resources.Lifecycle INPUT
3688+
resources.postgres_projects.*.lifecycle.prevent_destroy bool INPUT
3689+
resources.postgres_projects.*.modified_status string INPUT
3690+
resources.postgres_projects.*.name string REMOTE
3691+
resources.postgres_projects.*.pg_version int INPUT STATE
3692+
resources.postgres_projects.*.project_id string INPUT STATE
3693+
resources.postgres_projects.*.spec *postgres.ProjectSpec REMOTE
3694+
resources.postgres_projects.*.spec.default_endpoint_settings *postgres.ProjectDefaultEndpointSettings REMOTE
3695+
resources.postgres_projects.*.spec.default_endpoint_settings.autoscaling_limit_max_cu float64 REMOTE
3696+
resources.postgres_projects.*.spec.default_endpoint_settings.autoscaling_limit_min_cu float64 REMOTE
3697+
resources.postgres_projects.*.spec.default_endpoint_settings.no_suspension bool REMOTE
3698+
resources.postgres_projects.*.spec.default_endpoint_settings.pg_settings map[string]string REMOTE
3699+
resources.postgres_projects.*.spec.default_endpoint_settings.pg_settings.* string REMOTE
3700+
resources.postgres_projects.*.spec.default_endpoint_settings.suspend_timeout_duration *duration.Duration REMOTE
3701+
resources.postgres_projects.*.spec.display_name string REMOTE
3702+
resources.postgres_projects.*.spec.history_retention_duration *duration.Duration REMOTE
3703+
resources.postgres_projects.*.spec.pg_version int REMOTE
3704+
resources.postgres_projects.*.status *postgres.ProjectStatus REMOTE
3705+
resources.postgres_projects.*.status.branch_logical_size_limit_bytes int64 REMOTE
3706+
resources.postgres_projects.*.status.default_endpoint_settings *postgres.ProjectDefaultEndpointSettings REMOTE
3707+
resources.postgres_projects.*.status.default_endpoint_settings.autoscaling_limit_max_cu float64 REMOTE
3708+
resources.postgres_projects.*.status.default_endpoint_settings.autoscaling_limit_min_cu float64 REMOTE
3709+
resources.postgres_projects.*.status.default_endpoint_settings.no_suspension bool REMOTE
3710+
resources.postgres_projects.*.status.default_endpoint_settings.pg_settings map[string]string REMOTE
3711+
resources.postgres_projects.*.status.default_endpoint_settings.pg_settings.* string REMOTE
3712+
resources.postgres_projects.*.status.default_endpoint_settings.suspend_timeout_duration *duration.Duration REMOTE
3713+
resources.postgres_projects.*.status.display_name string REMOTE
3714+
resources.postgres_projects.*.status.history_retention_duration *duration.Duration REMOTE
3715+
resources.postgres_projects.*.status.owner string REMOTE
3716+
resources.postgres_projects.*.status.pg_version int REMOTE
3717+
resources.postgres_projects.*.status.synthetic_storage_size_bytes int64 REMOTE
3718+
resources.postgres_projects.*.uid string REMOTE
3719+
resources.postgres_projects.*.update_time *time.Time REMOTE
3720+
resources.postgres_projects.*.url string INPUT
35963721
resources.quality_monitors.*.assets_dir string ALL
35973722
resources.quality_monitors.*.baseline_table_name string ALL
35983723
resources.quality_monitors.*.custom_metrics []catalog.MonitorMetric ALL

‎acceptance/bundle/resources/postgres_branches/basic/databricks.yml.tmpl‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
bundle:
2+
name: deploy-postgres-branch-$UNIQUE_NAME
3+
4+
sync:
5+
paths: []
6+
7+
resources:
8+
postgres_projects:
9+
my_project:
10+
project_id: test-pg-proj-$UNIQUE_NAME
11+
display_name: "Test Project for Branch"
12+
pg_version: 16
13+
history_retention_duration: "604800s"
14+
default_endpoint_settings:
15+
autoscaling_limit_min_cu: 0.5
16+
autoscaling_limit_max_cu: 4
17+
suspend_timeout_duration: "300s"
18+
19+
postgres_branches:
20+
main:
21+
parent: ${resources.postgres_projects.my_project.id}
22+
branch_id: main
23+
no_expiry: true

0 commit comments

Comments
 (0)