RocksDB size continues to grow after node has finished syncing

[dancamarg0]

Description:

After syncing ~5 epochs on an Alpamayo instance (completed 5 days ago), RocksDB storage usage has continued to grow indefinitely, despite no new syncing activity.

Expected storage footprint for 5 epochs:

Blocks: ~4.70 TB

RocksDB: ~1.8–2 TB

Actual behavior:

RocksDB usage has steadily increased, with total disk usage rising from 66% → 79% in just 5 days.

It seems Alpamayo’s RocksDB may not be compacting or pruning files correctly, causing unbounded growth even though storage.blocks.max has stayed the same 2160000 for 5 epochs.

Comments from Kirill in a separate conversation:

• there should be function in rocksdb for manual compaction
  https://docs.rs/rocksdb/latest/rocksdb/type.DBWithThreadMode.html#method.compact_range_cf

• probably I can add http endpoint on metrics port to trigger compaction, wdyt?

• I'm not sure how to trigger it now, usually compaction happened automatically after reaching some threshold (by default 10% iirc)

Impact:

RocksDB growth without bounds risks filling disks, even after syncing is complete.

This could make running Alpamayo unsustainable long-term without manual intervention.

Expected Behavior:
RocksDB should compact/prune automatically after syncing completes, maintaining a stable footprint close to the expected 1.8–2 TB.

[dancamarg0]

update on this: 4d after I opened this issue and I see rocksdb pruned a bit of space on Sep 16th ~2h15 UTC and again Sep 16th ~16h55

The times don't make much sense. I expect rocksdb would be compressing / pruning old files periodically in fixed intervals but something seems broken

Today Sep 17th rocksdb is consuming 2.7T for 5 epochs

[dancamarg0]

Sep 30th