Used KeyedSllice for job fields in bundle plan

[ilyakuz-db]

Changes

Use key-slice for job.parameters, job.job_clusters and job.environments to make comparison order-independent and more predictable

Why

For non-keyed arrays when the new item is added we show the full array as replaced instead of only one last element

Example:

// array recreation when new item is added

// Example diff from in config-remote-sync:

        parameters:
 -        - name: catalog
 -          default: ${var.catalog}
 -        - name: schema
 -          default: ${var.schema}
 +        - default: main
 +          name: catalog
 +        - default: ilya_kuznetsov
 +          name: schema
 +        - default: bar
 +          name: foo

Tests

Updated existing tests and added separate test suite for variables (I will be add more fields there in future PRs)

Also I tested that key-value pairs cannot be duplicated as this is the only thing that can affect the behavior, see this comment #4921 (comment)

[ilyakuz-db]

Tested the behavior for duplicated items with the same key, using this config:

resources:
    jobs:
      my_job:
        name: my_job

        parameters:
          - name: catalog
            default: first
          - name: schema
            default: first
          - name: catalog
            default: second
          - name: schema
            default: second

        job_clusters:
          - job_cluster_key: my_cluster
            new_cluster:
              spark_version: "15.4.x-scala2.12"
              num_workers: 1
              node_type_id: "i3.xlarge"
          - job_cluster_key: my_cluster
            new_cluster:
              spark_version: "14.3.x-scala2.12"
              num_workers: 2
              node_type_id: "i3.2xlarge"

        environments:
          - environment_key: my_env
            spec: { client: "1" }
          - environment_key: my_env
            spec: { client: "2" }

        tasks:
          - task_key: my_task
            notebook_task:
              notebook_path: ./notebook.py

 Field: parameters
  validate: No warning
  Terraform deploy: Succeeds
  Direct deploy: Succeeds
  API stores: 2 entries (deduped, last wins)
  ────────────────────────────────────────
  Field: job_clusters
  validate: No warning
  Terraform deploy: Succeeds
  Direct deploy: Succeeds
  API stores: 1 entry (deduped, field-level merge)
  ────────────────────────────────────────
  Field: environments
  validate: No warning
  Terraform deploy: API error
  Direct deploy: API error
  API stores: N/A — rejected

[github-actions]

Reviewers by ownership area

This PR crosses multiple ownership areas. Each area needs at least one
approval from its owners (or any maintainer can approve everything).

/acceptance/bundle/

Files: acceptance/bundle/artifacts/whl_dynamic/out.plan_update.direct.json, acceptance/bundle/config-remote-sync/job_fields/output.txt, acceptance/bundle/config-remote-sync/job_params_variables/databricks.yml.tmpl, acceptance/bundle/config-remote-sync/job_params_variables/out.test.toml, acceptance/bundle/config-remote-sync/job_params_variables/output.txt, acceptance/bundle/config-remote-sync/job_params_variables/script, acceptance/bundle/config-remote-sync/job_params_variables/test.toml, acceptance/bundle/resource_deps/jobs_update_remote/out.plan_update.direct.json
Suggested: @denik
Also eligible: @shreyas-goenka, @andrewnester, @anton-107, @pietern, @lennartkats-db

/bundle/

Files: bundle/direct/dresources/job.go
Suggested: @denik
Also eligible: @shreyas-goenka, @andrewnester, @anton-107, @pietern, @lennartkats-db

_{Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum) can approve all areas.
See OWNERS for ownership rules.}