Validate Databricks CLI token scopes against SDK configuration (#689)

## Summary

Detects when a cached Databricks CLI token was issued with different
OAuth scopes than what the SDK configuration requires, and surfaces an
actionable error telling the user how to re-authenticate instead of
silently making requests with the wrong scopes.

Mirrors Python SDK PR:
databricks/databricks-sdk-py#1286

## Why

The `databricks auth token` CLI command does not accept scopes — it
returns whatever token was cached from the last `databricks auth login`.
If a user configures specific scopes in the SDK (e.g. `scopes=["sql"]`,
either directly in code or loaded from a CLI profile), but their cached
token was issued with different scopes (e.g. `all-apis`), every API
request silently uses the wrong scopes. This is confusing to debug
because authentication succeeds — it just grants the wrong permissions.

This is especially likely now that the CLI writes scopes into profiles,
meaning users who switch between different scope configurations will hit
this without realizing it.

## What changed

### Behavioral changes

- **Scope mismatch error** — When scopes are explicitly set in the SDK
config and the cached CLI token's JWT `scope` claim doesn't match, a
`DatabricksException` is raised with a message like:

```
Token issued by Databricks CLI has scopes [all-apis] which do not match the configured scopes [sql]. Please re-authenticate with the desired scopes by running `databricks auth login` with the --scopes flag. Scopes default to all-apis.
```

- **Credential chain fallthrough** — When `authType` is not explicitly
set to `"databricks-cli"` (i.e. we're being tried as part of the default
credential chain), a scope mismatch logs a warning and returns `null` so
other providers get a chance. When `authType="databricks-cli"` is
explicitly set, the error propagates immediately.

- **`offline_access` is ignored during comparison** — This scope
controls refresh token issuance, not API permissions. Its presence or
absence on either side does not trigger a mismatch.

- **Validation is skipped when scopes are not configured** —
`getScopes()` defaults to `["all-apis"]` when nothing is set, which
would cause false-positive mismatches. Validation only runs when scopes
are explicitly set.

### Internal changes

- **`DatabricksCliCredentialsProvider.validateTokenScopes()`** — Decodes
the JWT `scope` claim (databricks uses space-delimited strings according
to RFC 9068), filters out `offline_access`, and compares against
configured scopes using strict set equality.

- **`DatabricksCliCredentialsProvider.getJwtClaims()`** — Decodes JWT
payload using `Base64.getUrlDecoder()` (URL-safe base64 per RFC 7519).

- **`DatabricksConfig.isScopesExplicitlySet()`** — Package private
method on Config that returns true when the raw `scopes` field is
non-null and non-empty.

## How is this tested?

- **Parametrized tests** covering: exact match, mismatch (error),
`offline_access` filtering (both directions), non-JWT tokens (skip), no
scope claim (skip), and error message format with re-auth command.
- **Unit tests** for `isScopesExplicitlySet()` flag behavior.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: tejaskochar-db

NEXT_CHANGELOG.md

@@ -5,6 +5,7 @@
 ### New Features and Improvements
 
 ### Bug Fixes
+* Fixed Databricks CLI authentication to detect when the cached token's scopes don't match the SDK's configured scopes. Previously, a scope mismatch was silently ignored, causing requests to use wrong permissions. The SDK now raises an error with instructions to re-authenticate.
 
 ### Security Vulnerabilities
 

databricks-sdk-java/src/main/java/com/databricks/sdk/core/DatabricksCliCredentialsProvider.java

@@ -2,8 +2,13 @@
 
 import com.databricks.sdk.core.oauth.CachedTokenSource;
 import com.databricks.sdk.core.oauth.OAuthHeaderFactory;
+import com.databricks.sdk.core.oauth.Token;
+import com.databricks.sdk.core.oauth.TokenSource;
 import com.databricks.sdk.core.utils.OSUtils;
 import com.databricks.sdk.support.InternalApi;
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.nio.charset.StandardCharsets;
 import java.util.*;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -15,6 +20,22 @@ public class DatabricksCliCredentialsProvider implements CredentialsProvider {
 
   public static final String DATABRICKS_CLI = "databricks-cli";
 
+  private static final ObjectMapper MAPPER = new ObjectMapper();
+
+  /** Thrown when the cached CLI token's scopes don't match the SDK's configured scopes. */
+  static class ScopeMismatchException extends DatabricksException {
+    ScopeMismatchException(String message) {
+      super(message);
+    }
+  }
+
+  /**
+   * offline_access controls whether the IdP issues a refresh token. It does not grant any API
+   * permissions, so its presence or absence should not cause a scope mismatch error.
+   */
+  private static final Set<String> SCOPES_IGNORED_FOR_COMPARISON =
+      Collections.singleton("offline_access");
+
   @Override
   public String authType() {
     return DATABRICKS_CLI;
@@ -92,13 +113,41 @@ public OAuthHeaderFactory configure(DatabricksConfig config) {
         return null;
       }
 
+      // Wrap the token source with scope validation so that every token — both the
+      // initial fetch and subsequent refreshes — is checked against the configured scopes.
+      TokenSource effectiveSource;
+      if (config.isScopesExplicitlySet()) {
+        List<String> scopes = config.getScopes();
+        effectiveSource =
+            new TokenSource() {
+              @Override
+              public Token getToken() {
+                Token t = tokenSource.getToken();
+                validateTokenScopes(t, scopes);
+                return t;
+              }
+            };
+      } else {
+        effectiveSource = tokenSource;
+      }
+
       CachedTokenSource cachedTokenSource =
-          new CachedTokenSource.Builder(tokenSource)
+          new CachedTokenSource.Builder(effectiveSource)
               .setAsyncDisabled(config.getDisableAsyncTokenRefresh())
               .build();
       cachedTokenSource.getToken(); // We need this for checking if databricks CLI is installed.
 
       return OAuthHeaderFactory.fromTokenSource(cachedTokenSource);
+    } catch (ScopeMismatchException e) {
+      // Scope validation failed. When the user explicitly selected databricks-cli auth,
+      // surface the mismatch immediately so they get an actionable error. When we're being
+      // tried as part of the default credential chain, step aside so other providers get
+      // a chance.
+      if (DATABRICKS_CLI.equals(config.getAuthType())) {
+        throw e;
+      }
+      LOG.warn("Databricks CLI token scope mismatch, skipping: {}", e.getMessage());
+      return null;
     } catch (DatabricksException e) {
       String stderr = e.getMessage();
       if (stderr.contains("not found")) {
@@ -112,4 +161,86 @@ public OAuthHeaderFactory configure(DatabricksConfig config) {
       throw e;
     }
   }
+
+  /**
+   * Validate that the token's scopes match the requested scopes from the config.
+   *
+   * <p>The {@code databricks auth token} command does not accept scopes yet. It returns whatever
+   * token was cached from the last {@code databricks auth login}. If a user configures specific
+   * scopes in the SDK config but their cached CLI token was issued with different scopes, requests
+   * will silently use the wrong scopes. This check surfaces that mismatch early with an actionable
+   * error telling the user how to re-authenticate with the correct scopes.
+   */
+  static void validateTokenScopes(Token token, List<String> requestedScopes) {
+    Map<String, Object> claims = getJwtClaims(token.getAccessToken());
+    if (claims == null) {
+      LOG.debug("Could not decode token as JWT to validate scopes");
+      return;
+    }
+
+    Object tokenScopesRaw = claims.get("scope");
+    if (tokenScopesRaw == null) {
+      LOG.debug("Token does not contain 'scope' claim, skipping scope validation");
+      return;
+    }
+
+    Set<String> tokenScopes = parseScopeClaim(tokenScopesRaw);
+    if (tokenScopes == null) {
+      LOG.debug("Unexpected 'scope' claim type: {}", tokenScopesRaw.getClass());
+      return;
+    }
+
+    tokenScopes.removeAll(SCOPES_IGNORED_FOR_COMPARISON);
+    Set<String> requested = new HashSet<>(requestedScopes);
+    requested.removeAll(SCOPES_IGNORED_FOR_COMPARISON);
+
+    if (!tokenScopes.equals(requested)) {
+      List<String> sortedTokenScopes = new ArrayList<>(tokenScopes);
+      Collections.sort(sortedTokenScopes);
+      List<String> sortedRequested = new ArrayList<>(requested);
+      Collections.sort(sortedRequested);
+
+      throw new ScopeMismatchException(
+          String.format(
+              "Token issued by Databricks CLI has scopes %s which do not match "
+                  + "the configured scopes %s. Please re-authenticate "
+                  + "with the desired scopes by running `databricks auth login` with the --scopes flag. "
+                  + "Scopes default to all-apis.",
+              sortedTokenScopes, sortedRequested));
+    }
+  }
+
+  /**
+   * Decode a JWT access token and return its payload claims. Returns null if the token is not a
+   * valid JWT. No signature verification is performed — the token was already authenticated by the
+   * CLI, and we only need to read the scope claim for comparison.
+   */
+  private static Map<String, Object> getJwtClaims(String accessToken) {
+    String[] parts = accessToken.split("\\.");
+    if (parts.length != 3) {
+      LOG.debug("Tried to decode access token as JWT, but failed: {} components", parts.length);
+      return null;
+    }
+    try {
+      byte[] payloadBytes = Base64.getUrlDecoder().decode(parts[1]);
+      String payloadJson = new String(payloadBytes, StandardCharsets.UTF_8);
+      @SuppressWarnings("unchecked")
+      Map<String, Object> claims = MAPPER.readValue(payloadJson, Map.class);
+      return claims;
+    } catch (IllegalArgumentException | JsonProcessingException e) {
+      LOG.debug("Failed to decode JWT claims: {}", e.getMessage());
+      return null;
+    }
+  }
+
+  /**
+   * Parse the JWT "scope" claim. Per RFC 9068, this is a space-delimited string. Returns null if
+   * the type is unexpected.
+   */
+  private static Set<String> parseScopeClaim(Object scopeClaim) {
+    if (scopeClaim instanceof String) {
+      return new HashSet<>(Arrays.asList(((String) scopeClaim).split("\\s+")));
+    }
+    return null;
+  }
 }

databricks-sdk-java/src/main/java/com/databricks/sdk/core/DatabricksConfig.java

@@ -459,6 +459,15 @@ public DatabricksConfig setScopes(List<String> scopes) {
     return this;
   }
 
+  /**
+   * Returns true if scopes were explicitly configured (either directly in code or loaded from a
+   * config file). When scopes are not set, getScopes() defaults to ["all-apis"], which would cause
+   * false-positive mismatches during scope validation.
+   */
+  boolean isScopesExplicitlySet() {
+    return scopes != null && !scopes.isEmpty();
+  }
+
   public String getProfile() {
     return profile;
   }

databricks-sdk-java/src/test/java/com/databricks/sdk/core/DatabricksCliScopeValidationTest.java

@@ -0,0 +1,143 @@
+package com.databricks.sdk.core;
+
+import static org.junit.jupiter.api.Assertions.*;
+
+import com.databricks.sdk.core.oauth.Token;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.nio.charset.StandardCharsets;
+import java.time.Instant;
+import java.util.*;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+
+class DatabricksCliScopeValidationTest {
+
+  private static final ObjectMapper MAPPER = new ObjectMapper();
+
+  /** Builds a fake JWT (header.payload.signature) with the given claims. */
+  private static String makeJwt(Map<String, Object> claims) {
+    try {
+      String header =
+          Base64.getUrlEncoder()
+              .withoutPadding()
+              .encodeToString("{\"alg\":\"none\"}".getBytes(StandardCharsets.UTF_8));
+      String payload =
+          Base64.getUrlEncoder().withoutPadding().encodeToString(MAPPER.writeValueAsBytes(claims));
+      return header + "." + payload + ".sig";
+    } catch (Exception e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  private static Token makeToken(Map<String, Object> claims) {
+    return new Token(makeJwt(claims), "Bearer", Instant.now().plusSeconds(3600));
+  }
+
+  static List<Arguments> scopeValidationCases() {
+    return Arrays.asList(
+        // Exact match (offline_access filtered out).
+        Arguments.of(
+            Collections.singletonMap("scope", "sql offline_access"),
+            Collections.singletonList("sql"),
+            false,
+            "match"),
+        // Mismatch throws.
+        Arguments.of(
+            Collections.singletonMap("scope", "all-apis offline_access"),
+            Collections.singletonList("sql"),
+            true,
+            "mismatch"),
+        // offline_access on token only — still equivalent.
+        Arguments.of(
+            Collections.singletonMap("scope", "all-apis offline_access"),
+            Collections.singletonList("all-apis"),
+            false,
+            "offline_access_on_token_only"),
+        // offline_access in config only — still equivalent.
+        Arguments.of(
+            Collections.singletonMap("scope", "all-apis"),
+            Arrays.asList("all-apis", "offline_access"),
+            false,
+            "offline_access_in_config_only"),
+        // Order should not matter.
+        Arguments.of(
+            Collections.singletonMap("scope", "clusters sql"),
+            Arrays.asList("sql", "clusters"),
+            false,
+            "multiple_scopes_order_independent"),
+        // Partial overlap is still a mismatch.
+        Arguments.of(
+            Collections.singletonMap("scope", "sql clusters"),
+            Arrays.asList("sql", "compute"),
+            true,
+            "multiple_scopes_partial_overlap_mismatch"),
+        // No scope claim in token — validation is skipped.
+        Arguments.of(
+            Collections.singletonMap("sub", "user@example.com"),
+            Collections.singletonList("sql"),
+            false,
+            "no_scope_claim_skips_validation"));
+  }
+
+  @ParameterizedTest(name = "{3}")
+  @MethodSource("scopeValidationCases")
+  void testScopeValidation(
+      Map<String, Object> tokenClaims,
+      List<String> configuredScopes,
+      boolean expectError,
+      String testName) {
+    Token token = makeToken(tokenClaims);
+
+    if (expectError) {
+      assertThrows(
+          DatabricksCliCredentialsProvider.ScopeMismatchException.class,
+          () -> DatabricksCliCredentialsProvider.validateTokenScopes(token, configuredScopes));
+    } else {
+      assertDoesNotThrow(
+          () -> DatabricksCliCredentialsProvider.validateTokenScopes(token, configuredScopes));
+    }
+  }
+
+  @Test
+  void testNonJwtTokenSkipsValidation() {
+    Token token = new Token("opaque-token-string", "Bearer", Instant.now().plusSeconds(3600));
+    assertDoesNotThrow(
+        () ->
+            DatabricksCliCredentialsProvider.validateTokenScopes(
+                token, Collections.singletonList("sql")));
+  }
+
+  @Test
+  void testErrorMessageContainsReauthCommand() {
+    Token token = makeToken(Collections.singletonMap("scope", "all-apis"));
+    DatabricksCliCredentialsProvider.ScopeMismatchException e =
+        assertThrows(
+            DatabricksCliCredentialsProvider.ScopeMismatchException.class,
+            () ->
+                DatabricksCliCredentialsProvider.validateTokenScopes(
+                    token, Arrays.asList("sql", "offline_access")));
+    assertTrue(
+        e.getMessage().contains("databricks auth login"),
+        "Expected re-auth command in error message, got: " + e.getMessage());
+    assertTrue(
+        e.getMessage().contains("do not match the configured scopes"),
+        "Expected scope mismatch details in error message, got: " + e.getMessage());
+  }
+
+  @Test
+  void testScopesExplicitlySetFlag() {
+    DatabricksConfig config = new DatabricksConfig();
+    assertFalse(config.isScopesExplicitlySet());
+
+    config.setScopes(Arrays.asList("sql", "clusters"));
+    assertTrue(config.isScopesExplicitlySet());
+
+    config.setScopes(Collections.emptyList());
+    assertFalse(config.isScopesExplicitlySet(), "Empty list should not count as explicitly set");
+
+    config.setScopes(null);
+    assertFalse(config.isScopesExplicitlySet(), "null should not count as explicitly set");
+  }
+}