Add ServerSidePlannedTable DSv2 implementation and tests #4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
murali-db
added a commit
that referenced
this pull request
Nov 2, 2025
This PR adds comprehensive integration tests that validate the entire server-side planning stack from DeltaCatalog through to data reading. Test Coverage: - Full stack integration: DeltaCatalog → ServerSidePlannedTable → Client → Data - SELECT query execution through server-side planning path - Aggregation queries (SUM, COUNT, GROUP BY) - Verification that normal path is unaffected when feature disabled Test Strategy: 1. Enable DeltaCatalog as Spark catalog 2. Create Parquet tables with test data 3. Enable forceServerSidePlanning flag 4. Configure ServerSidePlanningTestClientFactory 5. Execute queries and verify results 6. Verify scan plan discovered files Test Cases: - E2E full stack integration with SELECT query - E2E aggregation query (SUM, COUNT, GROUP BY) - Normal path verification (feature disabled) Assertions: - Query results are correct - Files are discovered via server-side planning - Aggregations produce correct values - Normal table loading works when feature disabled This completes the test pyramid: - PR #1: Test infrastructure (REST server) - PR #2: Client unit tests - PR #3: DSv2 Table unit and integration tests - PR #4: DeltaCatalog integration (no new tests - minimal change) - PR #5: Full stack E2E integration tests (this PR) All functionality is now fully tested from unit to integration level.
5 tasks
murali-db
force-pushed
the
server-side-planning-2-client-interface
branch
from
c49fd19 to
f9a93e1
Compare
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
from
5b5e717 to
fd30578
Compare
murali-db
added a commit
that referenced
this pull request
Nov 3, 2025
This PR adds comprehensive integration tests that validate the entire server-side planning stack from DeltaCatalog through to data reading. Test Coverage: - Full stack integration: DeltaCatalog → ServerSidePlannedTable → Client → Data - SELECT query execution through server-side planning path - Aggregation queries (SUM, COUNT, GROUP BY) - Verification that normal path is unaffected when feature disabled Test Strategy: 1. Enable DeltaCatalog as Spark catalog 2. Create Parquet tables with test data 3. Enable forceServerSidePlanning flag 4. Configure ServerSidePlanningTestClientFactory 5. Execute queries and verify results 6. Verify scan plan discovered files Test Cases: - E2E full stack integration with SELECT query - E2E aggregation query (SUM, COUNT, GROUP BY) - Normal path verification (feature disabled) Assertions: - Query results are correct - Files are discovered via server-side planning - Aggregations produce correct values - Normal table loading works when feature disabled This completes the test pyramid: - PR #1: Test infrastructure (REST server) - PR #2: Client unit tests - PR #3: DSv2 Table unit and integration tests - PR #4: DeltaCatalog integration (no new tests - minimal change) - PR #5: Full stack E2E integration tests (this PR) All functionality is now fully tested from unit to integration level.
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
from
fd30578 to
3abb6ec
Compare
murali-db
added a commit
that referenced
this pull request
Nov 3, 2025
This PR adds comprehensive integration tests that validate the entire server-side planning stack from DeltaCatalog through to data reading. Test Coverage: - Full stack integration: DeltaCatalog → ServerSidePlannedTable → Client → Data - SELECT query execution through server-side planning path - Aggregation queries (SUM, COUNT, GROUP BY) - Verification that normal path is unaffected when feature disabled Test Strategy: 1. Enable DeltaCatalog as Spark catalog 2. Create Parquet tables with test data 3. Enable forceServerSidePlanning flag 4. Configure ServerSidePlanningTestClientFactory 5. Execute queries and verify results 6. Verify scan plan discovered files Test Cases: - E2E full stack integration with SELECT query - E2E aggregation query (SUM, COUNT, GROUP BY) - Normal path verification (feature disabled) Assertions: - Query results are correct - Files are discovered via server-side planning - Aggregations produce correct values - Normal table loading works when feature disabled This completes the test pyramid: - PR #1: Test infrastructure (REST server) - PR #2: Client unit tests - PR #3: DSv2 Table unit and integration tests - PR #4: DeltaCatalog integration (no new tests - minimal change) - PR #5: Full stack E2E integration tests (this PR) All functionality is now fully tested from unit to integration level.
murali-db
force-pushed
the
server-side-planning-2-client-interface
branch
from
2dcd2e8 to
43a3d0c
Compare
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
from
3abb6ec to
d9a5b07
Compare
murali-db
added a commit
that referenced
this pull request
Nov 3, 2025
This PR adds comprehensive integration tests that validate the entire server-side planning stack from DeltaCatalog through to data reading. Test Coverage: - Full stack integration: DeltaCatalog → ServerSidePlannedTable → Client → Data - SELECT query execution through server-side planning path - Aggregation queries (SUM, COUNT, GROUP BY) - Verification that normal path is unaffected when feature disabled Test Strategy: 1. Enable DeltaCatalog as Spark catalog 2. Create Parquet tables with test data 3. Enable forceServerSidePlanning flag 4. Configure ServerSidePlanningTestClientFactory 5. Execute queries and verify results 6. Verify scan plan discovered files Test Cases: - E2E full stack integration with SELECT query - E2E aggregation query (SUM, COUNT, GROUP BY) - Normal path verification (feature disabled) Assertions: - Query results are correct - Files are discovered via server-side planning - Aggregations produce correct values - Normal table loading works when feature disabled This completes the test pyramid: - PR #1: Test infrastructure (REST server) - PR #2: Client unit tests - PR #3: DSv2 Table unit and integration tests - PR #4: DeltaCatalog integration (no new tests - minimal change) - PR #5: Full stack E2E integration tests (this PR) All functionality is now fully tested from unit to integration level.
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
from
d9a5b07 to
755e14b
Compare
murali-db
added a commit
that referenced
this pull request
Nov 3, 2025
This PR adds comprehensive integration tests that validate the entire server-side planning stack from DeltaCatalog through to data reading. Test Coverage: - Full stack integration: DeltaCatalog → ServerSidePlannedTable → Client → Data - SELECT query execution through server-side planning path - Aggregation queries (SUM, COUNT, GROUP BY) - Verification that normal path is unaffected when feature disabled Test Strategy: 1. Enable DeltaCatalog as Spark catalog 2. Create Parquet tables with test data 3. Enable forceServerSidePlanning flag 4. Configure ServerSidePlanningTestClientFactory 5. Execute queries and verify results 6. Verify scan plan discovered files Test Cases: - E2E full stack integration with SELECT query - E2E aggregation query (SUM, COUNT, GROUP BY) - Normal path verification (feature disabled) Assertions: - Query results are correct - Files are discovered via server-side planning - Aggregations produce correct values - Normal table loading works when feature disabled This completes the test pyramid: - PR #1: Test infrastructure (REST server) - PR #2: Client unit tests - PR #3: DSv2 Table unit and integration tests - PR #4: DeltaCatalog integration (no new tests - minimal change) - PR #5: Full stack E2E integration tests (this PR) All functionality is now fully tested from unit to integration level.
murali-db
force-pushed
the
server-side-planning-2-client-interface
branch
from
57e1430 to
37c120c
Compare
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
from
755e14b to
673a081
Compare
murali-db
added a commit
that referenced
this pull request
Nov 3, 2025
This PR adds comprehensive integration tests that validate the entire server-side planning stack from DeltaCatalog through to data reading. Test Coverage: - Full stack integration: DeltaCatalog → ServerSidePlannedTable → Client → Data - SELECT query execution through server-side planning path - Aggregation queries (SUM, COUNT, GROUP BY) - Verification that normal path is unaffected when feature disabled Test Strategy: 1. Enable DeltaCatalog as Spark catalog 2. Create Parquet tables with test data 3. Enable forceServerSidePlanning flag 4. Configure ServerSidePlanningTestClientFactory 5. Execute queries and verify results 6. Verify scan plan discovered files Test Cases: - E2E full stack integration with SELECT query - E2E aggregation query (SUM, COUNT, GROUP BY) - Normal path verification (feature disabled) Assertions: - Query results are correct - Files are discovered via server-side planning - Aggregations produce correct values - Normal table loading works when feature disabled This completes the test pyramid: - PR #1: Test infrastructure (REST server) - PR #2: Client unit tests - PR #3: DSv2 Table unit and integration tests - PR #4: DeltaCatalog integration (no new tests - minimal change) - PR #5: Full stack E2E integration tests (this PR) All functionality is now fully tested from unit to integration level.
murali-db
force-pushed
the
server-side-planning-2-client-interface
branch
from
37c120c to
dfc533e
Compare
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
from
673a081 to
91c4414
Compare
murali-db
force-pushed
the
server-side-planning-2-client-interface
branch
from
dfc533e to
cf88294
Compare
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
from
91c4414 to
9d514e9
Compare
murali-db
force-pushed
the
server-side-planning-2-client-interface
branch
from
cf88294 to
1f9a958
Compare
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
2 times, most recently
from
6a25ba6 to
819df83
Compare
Implements DSv2 Table/Scan/Batch interfaces for server-side planning: - ServerSidePlannedTable: DSv2 table backed by server-side planning client - ServerSidePlannedScan/Batch: Scan and batch implementations - ServerSidePlannedFilePartition: Custom InputPartition for file-based reads - ServerSidePlanningTestClient: Test client using Spark for file discovery This allows Delta to read tables using file lists from remote planning services (e.g., Iceberg REST catalog) without local file discovery. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Rename 'namespace' parameter to 'database' throughout ServerSidePlannedTable and its related classes to match the naming in ServerSidePlanningClient.planScan(). Also update tests to use the new parameter names and fix references to removed schema field in ScanPlan and renamed factory class. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
murali-db
force-pushed
the
server-side-planning-3-dsv2-table-impl
branch
from
819df83 to
47e6fa9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the Spark DataSource V2 Table that uses server-side scan planning to read data files. This is the core read path implementation that will be used by DeltaCatalog when credentials are unavailable.
Stack:
Changes
DSv2 Table Implementation (
spark/.../catalog/ServerSidePlannedTable.scala)Class Hierarchy:
ServerSidePlannedTable (Table, SupportsRead)
└─> ServerSidePlannedScanBuilder (ScanBuilder)
└─> ServerSidePlannedScan (Scan, Batch)
├─> ServerSidePlannedFileInputPartition (InputPartition)
└─> ServerSidePlannedFilePartitionReaderFactory (PartitionReaderFactory)
└─> ServerSidePlannedFilePartitionReader (PartitionReader[InternalRow])
Key Classes:
BATCH_READcapabilityclient.planScan()to get file list from serverTests (
spark/src/test/.../ServerSidePlannedTableSuite.scala)Unit Tests (with MockServerSidePlanningClient):
Integration Tests (with ServerSidePlanningTestClient):
input_file_name()Which Delta project/connector is this regarding?
Description
How was this patch tested?
Does this PR introduce any user-facing changes?