Skip to content

Commit 994c9d7

Browse files
andrewnesterandrewnester
authored
Added support for UC catalogs (only in direct mode) (#4342)
## Changes Added support for UC catalogs (only in direct mode) ## Why Supporting UC catalogs is a natural conrinuation of DABs support for UC schemas. ## Tests Added acceptance test <!-- If your PR needs to be included in the release notes for next release, add a separate entry in NEXT_CHANGELOG.md as part of your PR. -->
1 parent 892754a commit 994c9d7

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

63 files changed

+1785
-4
lines changed

‎NEXT_CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
### Bundles
1212

1313
* Add missing values to SchemaGrantPrivilege enum ([#4380](https://github.com/databricks/cli/pull/4380))
14+
* Added support for UC catalogs (only in direct mode) ([#4342](https://github.com/databricks/cli/pull/4342))
1415
* engine/direct: Fix updates to fields being ignored for database_catalogs, synced_database_tables (([#4388](https://github.com/databricks/cli/pull/4388)))
1516

1617
### Dependency updates

‎acceptance/bundle/invariant/configs/catalog.yml.tmpl‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
bundle:
2+
name: test-bundle-$UNIQUE_NAME
3+
4+
resources:
5+
catalogs:
6+
foo:
7+
name: test-catalog-$UNIQUE_NAME
8+
comment: This is a test catalog

‎acceptance/bundle/invariant/no_drift/out.test.toml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/invariant/test.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ EnvMatrix.DATABRICKS_BUNDLE_ENGINE = [
2020
EnvMatrix.INPUT_CONFIG = [
2121
"alert.yml.tmpl",
2222
"app.yml.tmpl",
23+
"catalog.yml.tmpl",
2324
"cluster.yml.tmpl",
2425
"dashboard.yml.tmpl",
2526
"database_catalog.yml.tmpl",

‎acceptance/bundle/refschema/out.fields.txt‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -208,6 +208,52 @@ resources.apps.*.permissions.permissions[*].group_name string ALL
208208
resources.apps.*.permissions.permissions[*].permission_level iam.PermissionLevel ALL
209209
resources.apps.*.permissions.permissions[*].service_principal_name string ALL
210210
resources.apps.*.permissions.permissions[*].user_name string ALL
211+
resources.catalogs.*.browse_only bool REMOTE
212+
resources.catalogs.*.catalog_type catalog.CatalogType REMOTE
213+
resources.catalogs.*.comment string ALL
214+
resources.catalogs.*.connection_name string ALL
215+
resources.catalogs.*.created_at int64 REMOTE
216+
resources.catalogs.*.created_by string REMOTE
217+
resources.catalogs.*.effective_predictive_optimization_flag *catalog.EffectivePredictiveOptimizationFlag REMOTE
218+
resources.catalogs.*.effective_predictive_optimization_flag.inherited_from_name string REMOTE
219+
resources.catalogs.*.effective_predictive_optimization_flag.inherited_from_type catalog.EffectivePredictiveOptimizationFlagInheritedFromType REMOTE
220+
resources.catalogs.*.effective_predictive_optimization_flag.value catalog.EnablePredictiveOptimization REMOTE
221+
resources.catalogs.*.enable_predictive_optimization catalog.EnablePredictiveOptimization REMOTE
222+
resources.catalogs.*.full_name string REMOTE
223+
resources.catalogs.*.grants []resources.CatalogGrant INPUT
224+
resources.catalogs.*.grants[*] resources.CatalogGrant INPUT
225+
resources.catalogs.*.grants[*].principal string INPUT
226+
resources.catalogs.*.grants[*].privileges []resources.CatalogGrantPrivilege INPUT
227+
resources.catalogs.*.grants[*].privileges[*] resources.CatalogGrantPrivilege INPUT
228+
resources.catalogs.*.id string INPUT
229+
resources.catalogs.*.isolation_mode catalog.CatalogIsolationMode REMOTE
230+
resources.catalogs.*.lifecycle resources.Lifecycle INPUT
231+
resources.catalogs.*.lifecycle.prevent_destroy bool INPUT
232+
resources.catalogs.*.metastore_id string REMOTE
233+
resources.catalogs.*.modified_status string INPUT
234+
resources.catalogs.*.name string ALL
235+
resources.catalogs.*.options map[string]string ALL
236+
resources.catalogs.*.options.* string ALL
237+
resources.catalogs.*.owner string REMOTE
238+
resources.catalogs.*.properties map[string]string ALL
239+
resources.catalogs.*.properties.* string ALL
240+
resources.catalogs.*.provider_name string ALL
241+
resources.catalogs.*.provisioning_info *catalog.ProvisioningInfo REMOTE
242+
resources.catalogs.*.provisioning_info.state catalog.ProvisioningInfoState REMOTE
243+
resources.catalogs.*.securable_type catalog.SecurableType REMOTE
244+
resources.catalogs.*.share_name string ALL
245+
resources.catalogs.*.storage_location string REMOTE
246+
resources.catalogs.*.storage_root string ALL
247+
resources.catalogs.*.updated_at int64 REMOTE
248+
resources.catalogs.*.updated_by string REMOTE
249+
resources.catalogs.*.url string INPUT
250+
resources.catalogs.*.grants.full_name string ALL
251+
resources.catalogs.*.grants.grants []dresources.GrantAssignment ALL
252+
resources.catalogs.*.grants.grants[*] dresources.GrantAssignment ALL
253+
resources.catalogs.*.grants.grants[*].principal string ALL
254+
resources.catalogs.*.grants.grants[*].privileges []catalog.Privilege ALL
255+
resources.catalogs.*.grants.grants[*].privileges[*] catalog.Privilege ALL
256+
resources.catalogs.*.grants.securable_type string ALL
211257
resources.clusters.*.apply_policy_default_values bool INPUT STATE
212258
resources.clusters.*.autoscale *compute.AutoScale ALL
213259
resources.clusters.*.autoscale.max_workers int ALL

‎acceptance/bundle/resources/catalogs/auto-approve/catalog.yml.tmpl‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
resources:
2+
catalogs:
3+
bar:
4+
name: test-catalog-$UNIQUE_NAME
5+
comment: This catalog was created from DABs

‎acceptance/bundle/resources/catalogs/auto-approve/databricks.yml.tmpl‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
bundle:
2+
name: uc-catalog-$UNIQUE_NAME
3+
4+
workspace:
5+
root_path: ~/.bundle/$UNIQUE_NAME
6+
7+
resources:
8+
schemas:
9+
foo:
10+
name: test-schema-$UNIQUE_NAME
11+
catalog_name: ${resources.catalogs.bar.name}
12+
comment: This schema is in the test catalog
13+
14+
include:
15+
- "*.yml"
16+
17+
targets:
18+
development:
19+
default: true

‎acceptance/bundle/resources/catalogs/auto-approve/out.test.toml‎

Lines changed: 6 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎acceptance/bundle/resources/catalogs/auto-approve/output.txt‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
2+
>>> [CLI] bundle deploy
3+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
4+
Deploying resources...
5+
Updating deployment state...
6+
Deployment complete!
7+
8+
=== Assert the catalog is created
9+
>>> [CLI] catalogs get test-catalog-[UNIQUE_NAME]
10+
{
11+
"full_name": "test-catalog-[UNIQUE_NAME]",
12+
"comment": "This catalog was created from DABs"
13+
}
14+
15+
=== Assert the schema is created and uses the catalog
16+
>>> [CLI] schemas get test-catalog-[UNIQUE_NAME].test-schema-[UNIQUE_NAME]
17+
{
18+
"full_name": "test-catalog-[UNIQUE_NAME].test-schema-[UNIQUE_NAME]",
19+
"catalog_name": "test-catalog-[UNIQUE_NAME]",
20+
"comment": "This schema is in the test catalog"
21+
}
22+
23+
=== Create a volume in the schema, and add a file to it. This ensures that the
24+
catalog has some data in it and deletion will fail unless force deletion
25+
is enabled.
26+
>>> [CLI] volumes create test-catalog-[UNIQUE_NAME] test-schema-[UNIQUE_NAME] test-volume-[UNIQUE_NAME] MANAGED
27+
{
28+
"full_name": "test-catalog-[UNIQUE_NAME].test-schema-[UNIQUE_NAME].test-volume-[UNIQUE_NAME]"
29+
}
30+
31+
>>> [CLI] fs cp test-file-[UNIQUE_NAME].txt dbfs:/Volumes/test-catalog-[UNIQUE_NAME]/test-schema-[UNIQUE_NAME]/test-volume-[UNIQUE_NAME]
32+
test-file-[UNIQUE_NAME].txt -> dbfs:/Volumes/test-catalog-[UNIQUE_NAME]/test-schema-[UNIQUE_NAME]/test-volume-[UNIQUE_NAME]/test-file-[UNIQUE_NAME].txt
33+
34+
=== Remove the UC catalog from the resource configuration.
35+
>>> rm catalog.yml
36+
37+
=== Try to redeploy the bundle - should fail without --auto-approve
38+
>>> [CLI] bundle deploy
39+
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]/files...
40+
Error: cannot plan resources.schemas.foo: cannot resolve "${resources.catalogs.bar.name}": internal error: resources.catalogs.bar: action is "delete" missing new_state
41+
42+
Error: planning failed
43+
44+
45+
=== Test cleanup
46+
>>> [CLI] bundle destroy --auto-approve
47+
The following resources will be deleted:
48+
delete resources.catalogs.bar
49+
delete resources.schemas.foo
50+
51+
This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
52+
delete resources.schemas.foo
53+
54+
All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/[UNIQUE_NAME]
55+
56+
Deleting files...
57+
Destroy complete!
58+
59+
=== Assert the catalog is deleted
60+
Exit code: 1

‎acceptance/bundle/resources/catalogs/auto-approve/script‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
envsubst < databricks.yml.tmpl > databricks.yml
2+
envsubst < catalog.yml.tmpl > catalog.yml
3+
4+
CATALOG_NAME="test-catalog-${UNIQUE_NAME}"
5+
SCHEMA_NAME="test-schema-${UNIQUE_NAME}"
6+
VOLUME_NAME="test-volume-${UNIQUE_NAME}"
7+
8+
cleanup() {
9+
title "Test cleanup"
10+
trace $CLI bundle destroy --auto-approve
11+
12+
title "Assert the catalog is deleted"
13+
trace errcode $CLI catalogs get "${CATALOG_NAME}" 2>/dev/null
14+
}
15+
trap cleanup EXIT
16+
17+
trace $CLI bundle deploy
18+
19+
title "Assert the catalog is created"
20+
trace $CLI catalogs get "${CATALOG_NAME}" | jq "{full_name, comment}"
21+
22+
title "Assert the schema is created and uses the catalog"
23+
trace $CLI schemas get "${CATALOG_NAME}.${SCHEMA_NAME}" | jq "{full_name, catalog_name, comment}"
24+
25+
title "Create a volume in the schema, and add a file to it. This ensures that the
26+
catalog has some data in it and deletion will fail unless force deletion
27+
is enabled."
28+
trace $CLI volumes create "${CATALOG_NAME}" "${SCHEMA_NAME}" "${VOLUME_NAME}" MANAGED | jq "{full_name}"
29+
30+
FILE_NAME="test-file-${UNIQUE_NAME}.txt"
31+
echo "Hello, world!" > $FILE_NAME
32+
trace $CLI fs cp "${FILE_NAME}" "dbfs:/Volumes/${CATALOG_NAME}/${SCHEMA_NAME}/${VOLUME_NAME}"
33+
34+
title "Remove the UC catalog from the resource configuration."
35+
trace rm catalog.yml
36+
37+
title "Try to redeploy the bundle - should fail without --auto-approve"
38+
trace $CLI bundle deploy

0 commit comments

Comments
 (0)