Add discovery login via login.databricks.com (#4702)

## Why

When users run `databricks auth login` without specifying a host, the
CLI had no fallback. Users had to know their workspace URL upfront. This
adds a discovery flow via login.databricks.com where users authenticate
in the browser, select a workspace, and the CLI automatically discovers
the workspace URL.

This PR adds discovery login as the default option. A follow-up PR once
this is merged, will merge the profile / new profile selector from `auth
token` with `auth login` to ensure a uniform experience, and ensuring
that users who already have profiles setup will be given the option to
re-login as the primary option.

## Changes

Before: `databricks auth login` without `--host` prompted for a host URL
or failed.
Now: `databricks auth login` without `--host` opens login.databricks.com
in the browser. After the user authenticates and selects a workspace,
the CLI saves the profile with the discovered host, account ID, and
workspace ID.

Implementation details:
- `shouldUseDiscovery()` detects when no host is available from flags,
args, or existing profile
- `discoveryLogin()` orchestrates the browser-based flow using the SDK's
discovery OAuth support
- `IntrospectToken()` (in `libs/auth`) calls the workspace introspection
endpoint to extract account/workspace IDs (best-effort, non-fatal on
failure)
- `splitScopes()` deduplicates scope parsing across login paths
- Error messages include actionable tips (e.g., "you can specify a
workspace directly with: databricks auth login --host <url>")
- Discovery dependencies (`PersistentAuth`, `OAuthArgument`,
`IntrospectToken`) are injectable via package-level vars for testability

### Post-review fixes

- `IntrospectToken` accepts an `*http.Client` parameter instead of using
`http.DefaultClient` (enterprise proxy/TLS compatibility)
- Introspection HTTP client clones `http.DefaultTransport` to inherit
system CA certs, proxy settings, and timeouts
- Discovery relogin now reuses existing profile scopes when `--scopes`
is not explicitly set (consistent with normal login path)
- Explicitly clear stale routing fields (`account_id`, `workspace_id`,
`experimental_is_unified_host`, `cluster_id`, `serverless_compute_id`)
before saving discovery profile
- `workspace_id` only set when introspection returns a fresh value
- Consolidate error message construction with `discoveryErr()` helper

## Test plan

- [x] Unit tests for `shouldUseDiscovery` (table-driven, 6 cases)
- [x] Unit tests for `splitScopes` (empty, single, whitespace, empty
entries)
- [x] Unit tests for `IntrospectToken` (success, zero workspace ID, HTTP
errors, malformed JSON, auth header, endpoint path, custom HTTP client)
- [x] Integration test for `discoveryLogin` with introspection failure
(verifies profile is still saved)
- [x] Integration test for `discoveryLogin` with introspection success
(verifies metadata extraction)
- [x] Regression test for relogin preserving existing profile scopes
- [x] Regression test for clearing stale routing fields from unified
profile
- [x] `go test ./cmd/auth/... ./libs/auth/...` passes

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: simonfaltum

acceptance/bin/discovery_browser.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""
+Simulates the login.databricks.com discovery flow for acceptance tests.
+
+When the CLI opens this "browser" with the login.databricks.com URL,
+the script extracts the OAuth parameters from the destination_url,
+constructs a callback to localhost with an iss parameter pointing
+at the testserver, and fetches it.
+
+Usage: discovery_browser.py <url>
+"""
+
+import os
+import sys
+import urllib.parse
+import urllib.request
+
+if len(sys.argv) < 2:
+    sys.stderr.write("Usage: discovery_browser.py <url>\n")
+    sys.exit(1)
+
+url = sys.argv[1]
+parsed = urllib.parse.urlparse(url)
+top_params = urllib.parse.parse_qs(parsed.query)
+
+destination_url = top_params.get("destination_url", [None])[0]
+if not destination_url:
+    sys.stderr.write(f"No destination_url found in: {url}\n")
+    sys.exit(1)
+
+dest_parsed = urllib.parse.urlparse(destination_url)
+dest_params = urllib.parse.parse_qs(dest_parsed.query)
+
+redirect_uri = dest_params.get("redirect_uri", [None])[0]
+state = dest_params.get("state", [None])[0]
+
+if not redirect_uri or not state:
+    sys.stderr.write(f"Missing redirect_uri or state in destination_url: {destination_url}\n")
+    sys.exit(1)
+
+# The testserver's host acts as the workspace issuer.
+testserver_host = os.environ.get("DATABRICKS_HOST", "")
+if not testserver_host:
+    sys.stderr.write("DATABRICKS_HOST not set\n")
+    sys.exit(1)
+
+issuer = testserver_host.rstrip("/") + "/oidc"
+
+# Build the callback URL with code, state, and iss (the workspace issuer).
+callback_params = urllib.parse.urlencode(
+    {
+        "code": "oauth-code",
+        "state": state,
+        "iss": issuer,
+    }
+)
+callback_url = f"{redirect_uri}?{callback_params}"
+
+try:
+    response = urllib.request.urlopen(callback_url)
+    if response.status != 200:
+        sys.stderr.write(f"Callback failed: {callback_url} (status {response.status})\n")
+        sys.exit(1)
+except Exception as e:
+    sys.stderr.write(f"Callback failed: {callback_url} ({e})\n")
+    sys.exit(1)
+
+sys.exit(0)

acceptance/cmd/auth/login/discovery/out.databrickscfg

@@ -0,0 +1,10 @@
+; The profile defined in the DEFAULT section is to be used as a fallback when no profile is explicitly specified.
+[DEFAULT]
+
+[discovery-test]
+host         = [DATABRICKS_URL]
+workspace_id = 12345
+auth_type    = databricks-cli
+
+[__settings__]
+default_profile = discovery-test

acceptance/cmd/auth/login/discovery/out.test.toml

@@ -0,0 +1,14 @@
+
+>>> [CLI] auth login --profile discovery-test
+Opening login.databricks.com in your browser...
+Profile discovery-test was successfully saved
+
+>>> [CLI] auth profiles
+Name                      Host                    Valid
+discovery-test (Default)  [DATABRICKS_URL]  YES
+
+>>> print_requests.py --get //tokens/introspect
+{
+  "method": "GET",
+  "path": "/api/2.0/tokens/introspect"
+}

acceptance/cmd/auth/login/discovery/output.txt

@@ -0,0 +1,14 @@
+sethome "./home"
+
+# Use the discovery browser script that simulates login.databricks.com
+export BROWSER="discovery_browser.py"
+
+trace $CLI auth login --profile discovery-test
+
+trace $CLI auth profiles
+
+# Verify the introspection endpoint was called (workspace_id in profile confirms this too).
+trace print_requests.py --get //tokens/introspect
+
+# Track the .databrickscfg file that was created to surface changes.
+mv "./home/.databrickscfg" "./out.databrickscfg"

acceptance/cmd/auth/login/discovery/script

@@ -0,0 +1,18 @@
+Ignore = [
+    "home"
+]
+RecordRequests = true
+
+# Override the introspection endpoint so we can verify it gets called.
+[[Server]]
+Pattern = "GET /api/2.0/tokens/introspect"
+Response.Body = '''
+{
+    "principal_context": {
+        "authentication_scope": {
+            "account_id": "test-account-123",
+            "workspace_id": 12345
+        }
+    }
+}
+'''