Currently, the history export commands (export_transactions, export_operations, export_effects, export_trades, export_assets, export_contract_events, export_token_transfers, export_ledgers, export_ledger_transaction) load the entire ledger range into memory at once, write a single output file, then upload. This doesn't scale well for large ranges.
export_ledger_entry_changes already uses the right pattern: it streams data in batches via a channel, writes one file per batch, and uploads each batch immediately.
Goal: Bring all history export commands in line with export_ledger_entry_changes.
Changes:
- Add a generic
StreamLedgerBatches helper to internal/input/ that sends ledger ranges in batch-size increments via a channel
- Each history command uses this to process, write, and upload one batch at a time
- Replace the
--output (single file) + --limit flags with --output (folder) + --batch-size, matching the export_ledger_entry_changes interface
- Output files follow the existing
{start}-{end}-{type}.txt naming convention already used by export_ledger_entry_changes
Currently, the history export commands (
export_transactions,export_operations,export_effects,export_trades,export_assets,export_contract_events,export_token_transfers,export_ledgers,export_ledger_transaction) load the entire ledger range into memory at once, write a single output file, then upload. This doesn't scale well for large ranges.export_ledger_entry_changesalready uses the right pattern: it streams data in batches via a channel, writes one file per batch, and uploads each batch immediately.Goal: Bring all history export commands in line with
export_ledger_entry_changes.Changes:
StreamLedgerBatcheshelper tointernal/input/that sends ledger ranges inbatch-sizeincrements via a channel--output(single file) +--limitflags with--output(folder) +--batch-size, matching theexport_ledger_entry_changesinterface{start}-{end}-{type}.txtnaming convention already used byexport_ledger_entry_changes