Skip to content

Modify history table exports to use channels to batch and upload #399

@chowbao

Description

@chowbao

Currently, the history export commands (export_transactions, export_operations, export_effects, export_trades, export_assets, export_contract_events, export_token_transfers, export_ledgers, export_ledger_transaction) load the entire ledger range into memory at once, write a single output file, then upload. This doesn't scale well for large ranges.

export_ledger_entry_changes already uses the right pattern: it streams data in batches via a channel, writes one file per batch, and uploads each batch immediately.

Goal: Bring all history export commands in line with export_ledger_entry_changes.

Changes:

  • Add a generic StreamLedgerBatches helper to internal/input/ that sends ledger ranges in batch-size increments via a channel
  • Each history command uses this to process, write, and upload one batch at a time
  • Replace the --output (single file) + --limit flags with --output (folder) + --batch-size, matching the export_ledger_entry_changes interface
  • Output files follow the existing {start}-{end}-{type}.txt naming convention already used by export_ledger_entry_changes

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions