Add invariant tests (#4326)

## Changes

Add a new kind of test - invariant test for DABs (a subtype of
acceptance test).

Invariant tests are acceptance tests that can be run against many
configs to check for certain properties.
Unlike regular acceptance tests full output is not recorded, unless the
condition is not met.

In this PR there is one invariant test added - no_drift, that checks
that there are no actions planned after successful deploy. If that's not
the case, the test will dump full JSON plan to the output.

## Why
Simplify resource implementation testing - implementors only need to
provide a config to get all invariants check.

With no_drift in particular, we've had several cases where this
condition is not met and it's still the case for some configurations of
permissions and grants. It's also going to become more important as we
replace "server_side_default" with something more precise.

The other use of invariant tests is fuzzing - since they work on any
config, they can be used on randomly generated configs to find bugs.

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: denik

acceptance/bin/verify_no_drift.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""
+Check that all actions in plan are "skip".
+"""
+
+import sys
+import json
+import pprint
+
+
+def check_plan(path):
+    with open(path) as fobj:
+        raw = fobj.read()
+
+    changes_detected = 0
+
+    try:
+        data = json.loads(raw)
+        for key, value in data["plan"].items():
+            action = value.get("action")
+            if action != "skip":
+                print(f"Unexpected {action=} for {key}")
+                changes_detected += 1
+    except Exception:
+        print(raw, flush=True)
+        raise
+
+    if changes_detected:
+        print(raw, flush=True)
+        sys.exit(10)
+
+
+def main():
+    for path in sys.argv[1:]:
+        check_plan(path)
+
+
+if __name__ == "__main__":
+    main()

acceptance/bundle/invariant/README.md

@@ -0,0 +1,6 @@
+Invariant tests are acceptance tests that can be run against many configs to check for certain properties.
+Unlike regular acceptance tests full output is not recorded, unless the condition is not met. For example,
+no_drift test checks that there are no actions planned after successful deploy. If that's not the case, the
+test will dump full JSON plan to the output.
+
+In order to add a new test, add a config to configs/ and include it in test.toml.

acceptance/bundle/invariant/configs/alert.yml.tmpl

@@ -0,0 +1,9 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  alerts:
+    foo:
+      warehouse_id: $TEST_DEFAULT_WAREHOUSE_ID
+      display_name: test-alert-$UNIQUE_NAME
+      file_path: ./alert.dbalert.json

acceptance/bundle/invariant/configs/app.yml.tmpl

@@ -0,0 +1,8 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  apps:
+    foo:
+      name: app-$UNIQUE_NAME
+      source_code_path: ./app

acceptance/bundle/invariant/configs/cluster.yml.tmpl

@@ -0,0 +1,10 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  clusters:
+    foo:
+      cluster_name: test-cluster-$UNIQUE_NAME
+      spark_version: 13.3.x-scala2.12
+      node_type_id: i3.xlarge
+      num_workers: 1

acceptance/bundle/invariant/configs/dashboard.yml.tmpl

@@ -0,0 +1,9 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  dashboards:
+    foo:
+      warehouse_id: $TEST_DEFAULT_WAREHOUSE_ID
+      display_name: test-dashboard-$UNIQUE_NAME
+      file_path: ./dashboard.lvdash.json

acceptance/bundle/invariant/configs/database_catalog.yml.tmpl

@@ -0,0 +1,15 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  database_instances:
+    instance1:
+      name: test-db-instance-$UNIQUE_NAME
+      capacity: CU_1
+
+  database_catalogs:
+    foo:
+      database_instance_name: ${resources.database_instances.instance1.name}
+      database_name: test_db
+      name: test-catalog-$UNIQUE_NAME
+      create_database_if_not_exists: true

acceptance/bundle/invariant/configs/database_instance.yml.tmpl

@@ -0,0 +1,8 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  database_instances:
+    foo:
+      name: test-db-instance-$UNIQUE_NAME
+      capacity: CU_1

acceptance/bundle/invariant/configs/experiment.yml.tmpl

@@ -0,0 +1,7 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  experiments:
+    foo:
+      name: /Users/$CURRENT_USER_NAME/test-experiment-$UNIQUE_NAME

acceptance/bundle/invariant/configs/job.yml.tmpl

@@ -0,0 +1,7 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  jobs:
+    foo:
+      name: test-job-$UNIQUE_NAME