Skip to content

Commit b7c67d4

Browse files
shreyas-goenkashreyas-goenka
authored
Add warning for when required bundle fields are not set (#3044)
## Changes This PR adds code to emit warnings when required values are not set in the bundle configuration tree. Note: We do not show a warning for zero values like "" or `0`, only when the value is actually omitted from YAML. Acceptance test at `bundle/validate/required` has coverage for these cases. ## Why Better user experience for the customers. ## Tests A new acceptance test at `bundle/validate/required`. The warnings also show up in existing acceptance tests.
1 parent dec5727 commit b7c67d4

File tree

23 files changed

+575
-5
lines changed

23 files changed

+575
-5
lines changed

‎acceptance/bundle/artifacts/issue_3109/output.txt‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
Warning: required field "entry_point" is not set
2+
at resources.jobs.job1.tasks[0].python_wheel_task
3+
in src/job1/databricks_job/resources/job1.yaml:16:17
4+
15
Building wheels...
26
Uploading dist/my_default_python-0.0.1-py3-none-any.whl...
37
Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle/dev/files...

‎acceptance/bundle/debug/direct/out.stderr.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@
5050
10:07:59 Debug: Apply pid=12345 mutator=ProcessStaticResources mutator=TranslatePathsDashboards
5151
10:07:59 Debug: Apply pid=12345 mutator=PythonMutator(load_resources)
5252
10:07:59 Debug: Apply pid=12345 mutator=PythonMutator(apply_mutators)
53+
10:07:59 Debug: Apply pid=12345 mutator=validate:required
5354
10:07:59 Debug: Apply pid=12345 mutator=CheckPermissions
5455
10:07:59 Debug: Apply pid=12345 mutator=TranslatePaths
5556
10:07:59 Debug: Apply pid=12345 mutator=PythonWrapperWarning

‎acceptance/bundle/debug/direct/output.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ Validation OK!
1414
+>>> [CLI] bundle validate --debug
1515
10:07:59 Info: start pid=12345 version=[DEV_VERSION] args="[CLI], bundle, validate, --debug"
1616
10:07:59 Debug: Found bundle root at [TEST_TMP_DIR] (file [TEST_TMP_DIR]/databricks.yml) pid=12345
17-
@@ -60,8 +62,4 @@
17+
@@ -61,8 +63,4 @@
1818
10:07:59 Debug: Apply pid=12345 mutator=metadata.AnnotateJobs
1919
10:07:59 Debug: Apply pid=12345 mutator=metadata.AnnotatePipelines
2020
-10:07:59 Debug: Apply pid=12345 mutator=terraform.Initialize

‎acceptance/bundle/debug/tf/out.stderr.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@
4848
10:07:59 Debug: Apply pid=12345 mutator=ProcessStaticResources mutator=TranslatePathsDashboards
4949
10:07:59 Debug: Apply pid=12345 mutator=PythonMutator(load_resources)
5050
10:07:59 Debug: Apply pid=12345 mutator=PythonMutator(apply_mutators)
51+
10:07:59 Debug: Apply pid=12345 mutator=validate:required
5152
10:07:59 Debug: Apply pid=12345 mutator=CheckPermissions
5253
10:07:59 Debug: Apply pid=12345 mutator=TranslatePaths
5354
10:07:59 Debug: Apply pid=12345 mutator=PythonWrapperWarning

‎acceptance/bundle/paths/fallback/output.txt‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11

22
>>> [CLI] bundle validate -t development -o json
3+
Warning: required field "entry_point" is not set
4+
at resources.jobs.my_job.tasks[2].python_wheel_task
5+
in resources/my_job.yml:33:13
6+
override_job.yml:29:17
7+
38
Warn: path ../src/notebook.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:9:32). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
49
Warn: path ../src/file.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:13:30). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
510
Warn: path ../src/dbt_project is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_job.yml:17:36). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
@@ -10,6 +15,10 @@ Warn: path ../src/notebook2.py is defined relative to the [TEST_TMP_DIR]/resourc
1015
Warn: path ../src/file2.py is defined relative to the [TEST_TMP_DIR]/resources directory ([TEST_TMP_DIR]/override_pipeline.yml:10:23). Please update the path to be relative to the file where it is defined. The current value will no longer be valid in the next release.
1116

1217
>>> [CLI] bundle validate -t error
18+
Warning: required field "entry_point" is not set
19+
at resources.jobs.my_job.tasks[2].python_wheel_task
20+
in resources/my_job.yml:33:13
21+
1322
Error: notebook "resources/this value is overridden" not found. Local notebook references are expected
1423
to contain one of the following file extensions: [.py, .r, .scala, .sql, .ipynb]
1524

@@ -19,6 +28,6 @@ Workspace:
1928
User: [USERNAME]
2029
Path: /Workspace/Users/[USERNAME]/.bundle/fallback/error
2130

22-
Found 1 error
31+
Found 1 error and 1 warning
2332

2433
Exit code: 1

‎acceptance/bundle/paths/nominal/output.txt‎

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,52 @@
11

22
>>> [CLI] bundle validate -t development -o json
3+
Warning: required field "entry_point" is not set
4+
at resources.jobs.my_job.tasks[4].python_wheel_task
5+
in resources/my_job.yml:33:13
6+
override_job.yml:29:17
7+
8+
Warning: required field "inputs" is not set
9+
at resources.jobs.my_job.tasks[1].for_each_task
10+
in resources/my_job.yml:47:13
11+
override_job.yml:45:17
12+
13+
Warning: required field "inputs" is not set
14+
at resources.jobs.my_job.tasks[2].for_each_task
15+
in resources/my_job.yml:53:13
16+
override_job.yml:51:17
17+
18+
Warning: required field "task_key" is not set
19+
at resources.jobs.my_job.tasks[1].for_each_task.task
20+
in resources/my_job.yml:48:15
21+
override_job.yml:46:19
22+
23+
Warning: required field "task_key" is not set
24+
at resources.jobs.my_job.tasks[2].for_each_task.task
25+
in resources/my_job.yml:54:15
26+
override_job.yml:52:19
27+
328

429
>>> [CLI] bundle validate -t error
30+
Warning: required field "entry_point" is not set
31+
at resources.jobs.my_job.tasks[4].python_wheel_task
32+
in resources/my_job.yml:33:13
33+
34+
Warning: required field "inputs" is not set
35+
at resources.jobs.my_job.tasks[1].for_each_task
36+
in resources/my_job.yml:47:13
37+
38+
Warning: required field "inputs" is not set
39+
at resources.jobs.my_job.tasks[2].for_each_task
40+
in resources/my_job.yml:53:13
41+
42+
Warning: required field "task_key" is not set
43+
at resources.jobs.my_job.tasks[1].for_each_task.task
44+
in resources/my_job.yml:48:15
45+
46+
Warning: required field "task_key" is not set
47+
at resources.jobs.my_job.tasks[2].for_each_task.task
48+
in resources/my_job.yml:54:15
49+
550
Error: notebook "resources/this value is overridden" not found. Local notebook references are expected
651
to contain one of the following file extensions: [.py, .r, .scala, .sql, .ipynb]
752

@@ -11,6 +56,6 @@ Workspace:
1156
User: [USERNAME]
1257
Path: /Workspace/Users/[USERNAME]/.bundle/nominal/error
1358

14-
Found 1 error
59+
Found 1 error and 5 warnings
1560

1661
Exit code: 1

‎acceptance/bundle/quality_monitor/output.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11

22
>>> [CLI] bundle validate -o json -t development
3+
Warning: required field "quartz_cron_expression" is not set
4+
at resources.quality_monitors.my_monitor.schedule
5+
in databricks.yml:17:9
6+
7+
Warning: required field "timezone_id" is not set
8+
at resources.quality_monitors.my_monitor.schedule
9+
in databricks.yml:17:9
10+
311
{
412
"mode": "development",
513
"quality_monitors": {

‎acceptance/bundle/validate/empty_resources/empty_dict/output.txt‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@
3535
}
3636

3737
=== resources.models.rname ===
38+
Warning: required field "name" is not set
39+
at resources.models.rname
40+
in databricks.yml:6:12
41+
3842
{
3943
"models": {
4044
"rname": {
@@ -54,27 +58,71 @@
5458
}
5559

5660
=== resources.registered_models.rname ===
61+
Warning: required field "catalog_name" is not set
62+
at resources.registered_models.rname
63+
in databricks.yml:6:12
64+
65+
Warning: required field "name" is not set
66+
at resources.registered_models.rname
67+
in databricks.yml:6:12
68+
69+
Warning: required field "schema_name" is not set
70+
at resources.registered_models.rname
71+
in databricks.yml:6:12
72+
5773
{
5874
"registered_models": {
5975
"rname": {}
6076
}
6177
}
6278

6379
=== resources.quality_monitors.rname ===
80+
Warning: required field "assets_dir" is not set
81+
at resources.quality_monitors.rname
82+
in databricks.yml:6:12
83+
84+
Warning: required field "output_schema_name" is not set
85+
at resources.quality_monitors.rname
86+
in databricks.yml:6:12
87+
88+
Warning: required field "table_name" is not set
89+
at resources.quality_monitors.rname
90+
in databricks.yml:6:12
91+
6492
{
6593
"quality_monitors": {
6694
"rname": {}
6795
}
6896
}
6997

7098
=== resources.schemas.rname ===
99+
Warning: required field "catalog_name" is not set
100+
at resources.schemas.rname
101+
in databricks.yml:6:12
102+
103+
Warning: required field "name" is not set
104+
at resources.schemas.rname
105+
in databricks.yml:6:12
106+
71107
{
72108
"schemas": {
73109
"rname": {}
74110
}
75111
}
76112

77113
=== resources.volumes.rname ===
114+
Warning: required field "catalog_name" is not set
115+
at resources.volumes.rname
116+
in databricks.yml:6:12
117+
118+
Warning: required field "name" is not set
119+
at resources.volumes.rname
120+
in databricks.yml:6:12
121+
122+
Warning: required field "schema_name" is not set
123+
at resources.volumes.rname
124+
in databricks.yml:6:12
125+
78126
{
79127
"volumes": {
80128
"rname": {
@@ -102,6 +150,14 @@
102150
}
103151

104152
=== resources.apps.rname ===
153+
Warning: required field "name" is not set
154+
at resources.apps.rname
155+
in databricks.yml:6:12
156+
157+
Warning: required field "source_code_path" is not set
158+
at resources.apps.rname
159+
in databricks.yml:6:12
160+
105161
Error: Missing app source code path
106162
in databricks.yml:6:12
107163

@@ -129,6 +185,10 @@ app resource 'rname' is missing required source_code_path field
129185
}
130186

131187
=== resources.secret_scopes.rname ===
188+
Warning: required field "name" is not set
189+
at resources.secret_scopes.rname
190+
in databricks.yml:6:12
191+
132192
{
133193
"secret_scopes": {
134194
"rname": {

‎acceptance/bundle/validate/empty_resources/with_grants/output.txt‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,10 @@ Warning: unknown field: grants
4747
at resources.models.rname
4848
in databricks.yml:7:7
4949

50+
Warning: required field "name" is not set
51+
at resources.models.rname
52+
in databricks.yml:7:7
53+
5054
{
5155
"models": {
5256
"rname": {
@@ -70,6 +74,18 @@ Warning: unknown field: grants
7074
}
7175

7276
=== resources.registered_models.rname ===
77+
Warning: required field "catalog_name" is not set
78+
at resources.registered_models.rname
79+
in databricks.yml:7:7
80+
81+
Warning: required field "name" is not set
82+
at resources.registered_models.rname
83+
in databricks.yml:7:7
84+
85+
Warning: required field "schema_name" is not set
86+
at resources.registered_models.rname
87+
in databricks.yml:7:7
88+
7389
{
7490
"registered_models": {
7591
"rname": {
@@ -83,13 +99,33 @@ Warning: unknown field: grants
8399
at resources.quality_monitors.rname
84100
in databricks.yml:7:7
85101

102+
Warning: required field "assets_dir" is not set
103+
at resources.quality_monitors.rname
104+
in databricks.yml:7:7
105+
106+
Warning: required field "output_schema_name" is not set
107+
at resources.quality_monitors.rname
108+
in databricks.yml:7:7
109+
110+
Warning: required field "table_name" is not set
111+
at resources.quality_monitors.rname
112+
in databricks.yml:7:7
113+
86114
{
87115
"quality_monitors": {
88116
"rname": {}
89117
}
90118
}
91119

92120
=== resources.schemas.rname ===
121+
Warning: required field "catalog_name" is not set
122+
at resources.schemas.rname
123+
in databricks.yml:7:7
124+
125+
Warning: required field "name" is not set
126+
at resources.schemas.rname
127+
in databricks.yml:7:7
128+
93129
{
94130
"schemas": {
95131
"rname": {
@@ -99,6 +135,18 @@ Warning: unknown field: grants
99135
}
100136

101137
=== resources.volumes.rname ===
138+
Warning: required field "catalog_name" is not set
139+
at resources.volumes.rname
140+
in databricks.yml:7:7
141+
142+
Warning: required field "name" is not set
143+
at resources.volumes.rname
144+
in databricks.yml:7:7
145+
146+
Warning: required field "schema_name" is not set
147+
at resources.volumes.rname
148+
in databricks.yml:7:7
149+
102150
{
103151
"volumes": {
104152
"rname": {
@@ -139,6 +187,14 @@ Warning: unknown field: grants
139187
at resources.apps.rname
140188
in databricks.yml:7:7
141189

190+
Warning: required field "name" is not set
191+
at resources.apps.rname
192+
in databricks.yml:7:7
193+
194+
Warning: required field "source_code_path" is not set
195+
at resources.apps.rname
196+
in databricks.yml:7:7
197+
142198
Error: Missing app source code path
143199
in databricks.yml:7:7
144200

@@ -174,6 +230,10 @@ Warning: unknown field: grants
174230
at resources.secret_scopes.rname
175231
in databricks.yml:7:7
176232

233+
Warning: required field "name" is not set
234+
at resources.secret_scopes.rname
235+
in databricks.yml:7:7
236+
177237
{
178238
"secret_scopes": {
179239
"rname": {

0 commit comments

Comments
 (0)