Skip to content

Core: TableMetadata Projection #14502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
singhpk234 wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Core: TableMetadata Projection #14502

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
be138e7
Core: Add snapshotTransformer API for TableMetadata
sfc-gh-prsingh Nov 4, 2025
0ccb33d
Add projection for TableMetadata
sfc-gh-prsingh Nov 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions core/src/main/java/org/apache/iceberg/TableMetadata.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,11 +343,15 @@ public String toString() {
// changes are carried through until metadata is read from a file
this.changes = changes;

this.snapshotsById = indexAndValidateSnapshots(snapshots, lastSequenceNumber);
this.snapshotsById =
this.snapshots != null
? indexAndValidateSnapshots(this.snapshots, lastSequenceNumber)
: ImmutableMap.of();
Copy link
Contributor

@rdblue rdblue Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this need to change TableMetadata? I think it can be dangerous if now there are cases where snapshots is null and that doesn't represent the actual table state. If I understand the intent of this PR, it is to have an alternative to TableMetadata that suppresses some information.

Copy link
Contributor

@rdblue rdblue Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see that this is because TableMetadataProjection inherits from TableMetadata. I think that pattern would cause problems as I mentioned above because TableMetadataProjection is a perfectly valid TableMetadata.

Is this really needed? If the idea is to transform snapshots when they are returned by accessors, you may not need to change this class at all.

Copy link
Contributor Author

@singhpk234 singhpk234 Jan 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, this is not needed for this pr at all this is mostly for case when snapshot is null and we call indexAndValidateSnapshots which fails with an NPE here

  private static Map<Long, Snapshot> indexAndValidateSnapshots(
      List<Snapshot> snapshots, long lastSequenceNumber) {
    for (Snapshot snap : snapshots) {
      ....
    }
    return builder.build();
  }

Happy to remove it or create a seperate pr for fixing this, caught this while testing projection

Copy link
Contributor

@rdblue rdblue Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that there should never be a case where snapshots is null. I'd probably keep that assumption in the code and, if anything, just add a check in the constructor that the list is valid.

this.schemasById = indexSchemas();
this.specsById = PartitionUtil.indexSpecs(specs);
this.sortOrdersById = indexSortOrders(sortOrders);
this.refs = validateRefs(currentSnapshotId, refs, snapshotsById);
this.refs =
this.snapshots != null ? validateRefs(currentSnapshotId, refs, snapshotsById) : refs;
this.statisticsFiles = ImmutableList.copyOf(statisticsFiles);
this.partitionStatisticsFiles = ImmutableList.copyOf(partitionStatisticsFiles);

Expand Down Expand Up @@ -395,7 +399,9 @@ public String toString() {
previous.timestampMillis);
}

validateCurrentSnapshot();
if (this.snapshots != null) {
validateCurrentSnapshot();
}
}

public int formatVersion() {
Expand Down Expand Up @@ -522,6 +528,10 @@ public Snapshot currentSnapshot() {
return snapshotsById.get(currentSnapshotId);
}

public long currentSnapshotId() {
return currentSnapshotId;
}

public List<Snapshot> snapshots() {
ensureSnapshotsLoaded();

Expand Down
204 changes: 204 additions & 0 deletions core/src/main/java/org/apache/iceberg/TableMetadataProjection.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.apache.iceberg;

import java.util.List;
import java.util.Map;
import java.util.function.Function;
import org.apache.iceberg.relocated.com.google.common.base.Preconditions;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableList;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;

/**
* A projection wrapper for {@link TableMetadata} that transforms snapshots on-demand.
*
* <p>This projection follows the same pattern as StructProjection, applying transformations lazily
* when snapshot-related methods are accessed. Unlike embedding transformation logic in
* TableMetadata itself, this keeps the transformation concerns separate.
*
* <p>The projection is lightweight - it extends TableMetadata and overrides only snapshot-related
* methods, allowing all other metadata (schemas, specs, properties, etc.) to be inherited
* naturally.
*
* <p>Example use cases:
*
* <ul>
* <li>Filtering sensitive data from snapshot summaries before exposing metadata
* <li>Enriching snapshots with additional computed properties
* <li>Testing scenarios requiring modified snapshot metadata without persisting changes
* </ul>
*
* <p>Example usage:
*
* <pre>
* // Remove sensitive data from snapshot summaries
* TableMetadata cleaned = TableMetadataProjection.create(
* originalMetadata,
* snapshot -&gt; {
* if (snapshot instanceof BaseSnapshot) {
* BaseSnapshot base = (BaseSnapshot) snapshot;
* Map&lt;String, String&gt; filteredSummary = filterSensitiveKeys(base.summary());
* return new BaseSnapshot(
* base.sequenceNumber(), base.snapshotId(), base.parentId(),
* base.timestampMillis(), base.operation(), filteredSummary,
* base.schemaId(), base.manifestListLocation(),
* base.firstRowId(), base.addedRows(), base.keyId());
* }
* return snapshot;
* });
* </pre>
*/
public class TableMetadataProjection extends TableMetadata {

/**
* Creates a projecting wrapper for {@link TableMetadata} that transforms snapshots.
*
* @param metadata the base table metadata to wrap
* @param snapshotTransformer function to transform each snapshot
* @return a projection that applies the transformer to all snapshots
*/
public static TableMetadata create(
TableMetadata metadata, Function<Snapshot, Snapshot> snapshotTransformer) {
Preconditions.checkArgument(metadata != null, "Metadata cannot be null");
Preconditions.checkArgument(snapshotTransformer != null, "Snapshot transformer cannot be null");
return new TableMetadataProjection(metadata, snapshotTransformer);
}

private final Function<Snapshot, Snapshot> snapshotTransformer;
private volatile List<Snapshot> transformedSnapshots;
private volatile Map<Long, Snapshot> transformedSnapshotsById;
private volatile Map<String, SnapshotRef> transformedRefs;

private TableMetadataProjection(
TableMetadata base, Function<Snapshot, Snapshot> snapshotTransformer) {
// Call super constructor with all fields from base metadata
// Pass null for snapshots since we'll provide them via override
super(
base.metadataFileLocation(),
base.formatVersion(),
base.uuid(),
base.location(),
base.lastSequenceNumber(),
base.lastUpdatedMillis(),
base.lastColumnId(),
base.currentSchemaId(),
base.schemas(),
base.defaultSpecId(),
base.specs(),
base.lastAssignedPartitionId(),
base.defaultSortOrderId(),
base.sortOrders(),
base.properties(),
base.currentSnapshotId(), // Get raw field value without triggering loading
null, // snapshots will be provided via override
base::snapshots, // lazy supplier to trigger transformation
base.snapshotLog(),
base.previousFiles(),
base.refs(),
base.statisticsFiles(),
base.partitionStatisticsFiles(),
base.nextRowId(),
base.encryptionKeys(),
base.changes());

this.snapshotTransformer = snapshotTransformer;
}

@Override
public List<Snapshot> snapshots() {
if (transformedSnapshots == null) {
synchronized (this) {
if (transformedSnapshots == null) {
// Trigger lazy loading via super, which will call our supplier
List<Snapshot> baseSnapshots = super.snapshots();
transformedSnapshots =
baseSnapshots.stream()
.map(snapshotTransformer)
.collect(ImmutableList.toImmutableList());
}
}
}
return transformedSnapshots;
}

@Override
public Snapshot snapshot(long snapshotId) {
// Build index lazily and look up transformed snapshot
if (transformedSnapshotsById == null) {
synchronized (this) {
if (transformedSnapshotsById == null) {
transformedSnapshotsById =
snapshots().stream()
.collect(ImmutableMap.toImmutableMap(Snapshot::snapshotId, s -> s));
}
}
}
return transformedSnapshotsById.get(snapshotId);
}

@Override
public Snapshot currentSnapshot() {
long currentId = currentSnapshotId();
if (currentId == -1) {
return null;
}

// Trigger loading if needed and return transformed snapshot
return snapshot(currentId);
}

@Override
public Map<String, SnapshotRef> refs() {
if (transformedRefs == null) {
synchronized (this) {
if (transformedRefs == null) {
// Rebuild refs to point to transformed snapshots
// Filter out any refs that point to snapshots that no longer exist after transformation
Map<Long, Snapshot> transformedById = transformedSnapshotsById();
transformedRefs =
super.refs().entrySet().stream()
.filter(entry -> transformedById.containsKey(entry.getValue().snapshotId()))
.collect(ImmutableMap.toImmutableMap(Map.Entry::getKey, Map.Entry::getValue));
}
}
}
return transformedRefs;
}

@Override
public SnapshotRef ref(String name) {
return refs().get(name);
}

private Map<Long, Snapshot> transformedSnapshotsById() {
if (transformedSnapshotsById == null) {
synchronized (this) {
if (transformedSnapshotsById == null) {
transformedSnapshotsById =
snapshots().stream()
.collect(ImmutableMap.toImmutableMap(Snapshot::snapshotId, s -> s));
}
}
}
return transformedSnapshotsById;
}

// All other methods (schemas, specs, properties, statistics, etc.) are
// automatically inherited from TableMetadata without any delegation needed
}
Loading