Incorrect Proofs for Unchanged Shards Due to Missing Upper Node Syncing

[NikhilSharmaWe]

Description

When using direct unsafe tree access to retrieve proofs other than only handling proof requests in the Flusher::flush method, unchanged shards in a block do not have their upper nodes synced. This results in stale proof data being used for these shards. The proof fails the check_proof check for the entries that are in the unchanged shards in the last block. The proof is fetched directly from the Tree in the FlusherShard after the last block is flushed.

Expected Behavior

During flush, all shards (changed or unchanged) should be synced via sync_upper_nodes so that proofs for unchanged shards are correct.

Actual Behavior:

If a shard is unchanged in a block:
The active bits are updated.
The upper nodes are not synced.
Proofs for entries in unchanged shards use stale, precomputed values.

While debugging a check_proof failure for unchanged shards, I observed:

For shard 9, SN 4096 in block 3 (shard changed), the right path is:

right_of_twig: [
  ProofNode { self_hash: [3, 0, 0, ...], peer_hash: [0, 0, 0, ...], peer_at_left: false },
  ProofNode { self_hash: [22, 73, 217, 61, ...], peer_hash: [48, 11, 66, 146, ...], peer_at_left: false },
  ...
]

For the same entry in block 4 (shard unchanged), the right path is:

right_of_twig: [
  ProofNode { self_hash: [3, 0, 0, ...], peer_hash: [0, 0, 0, ...], peer_at_left: false },
  ProofNode { self_hash: [48, 11, 66, 146, ...], peer_hash: [48, 11, 66, 146, ...], peer_at_left: false },
  ...
]

The stale proof nodes in unchanged shards seem to occur because sync_mt_for_active_bits_phase1 does not sync them when new_list is empty.

Error in proof::check_proof:Direct proof structure invalid: Mismatch at right path, level: 0

I added a method get_all_active_positions to collect all active bits in unchanged shards and populate new_list when it’s empty:

fn get_all_active_positions(&self) -> Vec {
let mut positions = Vec::new();

for (shard_idx, active_bits_shard) in self.active_bit_shards.iter().enumerate() {
    for &local_key in active_bits_shard.keys() {
        let twig_id = local_key * TWIG_SHARD_COUNT as u64 + shard_idx as u64;
        if twig_id <= self.youngest_twig_id {
            for pos_offset in 0..4 {
                positions.push(twig_id * 4 + pos_offset);
            }
        }
    }
}

positions.sort();
positions.dedup();
positions

}

Using this method in sync_mt_for_active_bits_phase1 to populate new_list ensures unchanged shards are synced and produces matching proofs for block 3 and block 4.

Am I understanding the cause correctly here? Or it can be a bug in how unchanged shards are being handled.

[NikhilSharmaWe]

Also, while accessing the tree directly from FlusherShard.
In a case, I get an entry which has next_key_hash lexicographically smaller than a key_hash, like this:

let indexer = ads.get_indexer();
indexer.for_each_adjacent_value(height, target_key_hash, |_k80, file_pos| -> bool {
            match self.read_entry_from_file_position(shard_id, file_pos) {
                Ok(Some(entry)) => {
                    if entry.key_hash < *target_key_hash && target_key_hash <= &entry.next_key_hash {
                        found_entry = Some(entry);
                        return true;
                    }
                    false
                },
                _ => false
            }
        });

The entry's shard and serial numbers are 7 and 302.

But when I try to get the proof from the tree, an error occurs: get_proof failed twig_file is empty
What could be a possible reason? Do the EntryFile and TwigFile not sync properly, or are old entries maybe truncated for some reason?

Inform if familiar.