Add scripts to DABs (#2813)

## Changes
Scripts can be defined in a bundle YAML file as:
```
scripts:
  my_script:
    content: |
      echo "one"
      echo "two"
```

They can then be run by calling `databricks bundle run my_script`

## Why
There are four primary benefits of having the scripts section in DABs:
1. Customers can scripts (like integration testing scripts) live and be
versioned with their bundle. Since integration tests based on spark
connect require authentication, scripts run via this mechanism will
automatically have the appropriate authentication configured.
2. Simplifies setup for users who might not have their own build tools
configured.
3. Allows us to stardarize documentation to always refer to the scripts
section.
4. Make DABs in the workspace integration possible. Since DABs in the
workspace cannot read arbitrary build file formats like Makefile.

### Why do we have an additional level of indentation with the `content`
field? Why not inline it?
The two main reasons are:
1. This mirrors what the artifacts block looks like. Users in the
artifact block are able to specify which shell they want to run their
build command in. We should offer the same functionality in the scripts
section.
5. Leaves the room open for us to provide users with additional
modifiers they might like to apply to their scripts.

## Tests
Acceptance tests.

---------

Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>

Author: shreyas-goenka

NEXT_CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Release v0.259.0
 
 ### Notable Changes
+* Add support for arbitrary scripts in DABs. Users can now define scripts in their bundle configuration. These scripts automatically inherit the bundle's credentials for authentication. They can be invoked with the `bundle run` command. ([#2813](https://github.com/databricks/cli/pull/2813))
 * Error when the absolute path to `databricks.yml` contains a glob character. These are: `*`, `?`, `[`, `]` and `^`. If the path to the `databricks.yml` file on your local filesystem contains one of these characters, that could lead to incorrect computation of glob patterns for the `includes` block and might cause resources to be deleted. After this patch users will not be at risk for unexpected deletions due to this issue. ([#3096](https://github.com/databricks/cli/pull/3096))
 * Diagnostics messages are no longer buffered to be printed at the end of command, flushed after every mutator ([#3175](https://github.com/databricks/cli/pull/3175))
 * Diagnostics are now always rendered with forward slashes in file paths, even on Windows ([#3175](https://github.com/databricks/cli/pull/3175))

acceptance/bundle/debug/direct/out.stderr.txt

@@ -19,6 +19,7 @@
 10:07:59 Info: Phase: initialize pid=12345
 10:07:59 Debug: Apply pid=12345 mutator=validate:AllResourcesHaveValues
 10:07:59 Debug: Apply pid=12345 mutator=validate:interpolation_in_auth_config
+10:07:59 Debug: Apply pid=12345 mutator=validate:scripts
 10:07:59 Debug: Apply pid=12345 mutator=RewriteSyncPaths
 10:07:59 Debug: Apply pid=12345 mutator=SyncDefaultPath
 10:07:59 Debug: Apply pid=12345 mutator=SyncInferRoot

acceptance/bundle/debug/direct/output.txt

@@ -14,7 +14,7 @@ Validation OK!
 +>>> [CLI] bundle validate --debug
  10:07:59 Info: start pid=12345 version=[DEV_VERSION] args="[CLI], bundle, validate, --debug"
  10:07:59 Debug: Found bundle root at [TEST_TMP_DIR] (file [TEST_TMP_DIR]/databricks.yml) pid=12345
-@@ -59,8 +61,4 @@
+@@ -60,8 +62,4 @@
  10:07:59 Debug: Apply pid=12345 mutator=metadata.AnnotateJobs
  10:07:59 Debug: Apply pid=12345 mutator=metadata.AnnotatePipelines
 -10:07:59 Debug: Apply pid=12345 mutator=terraform.Initialize

acceptance/bundle/debug/tf/out.stderr.txt

@@ -17,6 +17,7 @@
 10:07:59 Info: Phase: initialize pid=12345
 10:07:59 Debug: Apply pid=12345 mutator=validate:AllResourcesHaveValues
 10:07:59 Debug: Apply pid=12345 mutator=validate:interpolation_in_auth_config
+10:07:59 Debug: Apply pid=12345 mutator=validate:scripts
 10:07:59 Debug: Apply pid=12345 mutator=RewriteSyncPaths
 10:07:59 Debug: Apply pid=12345 mutator=SyncDefaultPath
 10:07:59 Debug: Apply pid=12345 mutator=SyncInferRoot

acceptance/bundle/run/inline-script/cwd/script

@@ -2,5 +2,5 @@ trace cd a/b/c
 
 trace python3 -c 'import os; print(os.getcwd())'
 
-# Scripts that bundle run executes should run from the bundle root.
+# Scripts that bundle run executes inline should run from the current working directory.
 trace $CLI bundle run -- python3 -c 'import os; print(os.getcwd())'

acceptance/bundle/run/scripts/basic/databricks.yml

@@ -0,0 +1,7 @@
+scripts:
+  foo:
+    content: |
+      echo 'hello'
+
+  me:
+    content: $CLI current-user me

acceptance/bundle/run/scripts/basic/out.test.toml

@@ -0,0 +1,5 @@
+Local = true
+Cloud = false
+
+[EnvMatrix]
+  DATABRICKS_CLI_DEPLOYMENT = ["terraform", "direct-exp"]

acceptance/bundle/run/scripts/basic/output.txt

@@ -0,0 +1,14 @@
+
+>>> [CLI] bundle run foo
+hello
+
+>>> [CLI] bundle run me
+{
+  "id":"[USERID]",
+  "userName":"[USERNAME]"
+}
+
+>>> [CLI] bundle run foo arg1 arg2
+Error: additional arguments are not supported for scripts. Got: [arg1 arg2]. We recommend using environment variables to pass runtime arguments to a script. For example: FOO=bar databricks bundle run my_script.
+
+Exit code: 1

acceptance/bundle/run/scripts/basic/script

@@ -0,0 +1,5 @@
+trace $CLI bundle run foo
+
+trace $CLI bundle run me
+
+errcode trace $CLI bundle run foo arg1 arg2