[Server-Side Planning] Wire filter projection pushdown end-to-end (PR23/D9)

[murali-db]

Wire projection pushdown through the entire stack, from Spark's SupportsPushDownRequiredColumns interface to Iceberg REST API.

Changes

Spark Module (4 files)

• ServerSidePlannedScanBuilder: Implement SupportsPushDownRequiredColumns
  • Add pruneColumns() to capture required schema from Spark optimizer
  • Pass both tableSchema and requiredSchema to scan
• ServerSidePlannedScan: Thread projection through to planning client
  • Only pass projection if different from full table schema
  • Allows server to optimize for column pruning
• ServerSidePlannedFilePartitionReaderFactory: Support projection pushdown
  • Accept both dataSchema (full) and requiredSchema (pruned)
  • ParquetFileFormat uses requiredSchema to read only needed columns
• Add ProjectionCapturingTestClient for test verification
• Add 3 E2E integration tests (implementation complete, test setup needs adjustment)

Iceberg Module (1 file)

• IcebergRESTCatalogPlanningClient: Convert and send projection
  • Use SparkToIcebergSchemaConverter to convert StructType to Iceberg Schema
  • Call withProjectedSchema() on PlanTableScanRequest builder
  • Enables Iceberg REST API to receive projection information

Test Status

• All 11 existing tests pass (spark module)
• All 49 existing tests pass (iceberg module: 23 expr + 19 schema + 7 REST)
• 3 new projection tests added (implementation complete, test setup needs work)

Design notes

• Follows same pattern as filter pushdown
• Uses Spark StructType as catalog-agnostic representation
• Each catalog converts to native format (Iceberg Schema)
• Zero behavior change when no projection (full table scan)

🤖 Generated with Claude Code

Co-Authored-By: Claude noreply@anthropic.com