Skip to content

feat: single transaction backtesting#41

Merged
odyslam merged 18 commits intomasterfrom
feat/single-tx-backtest
Jan 22, 2026
Merged

feat: single transaction backtesting#41
odyslam merged 18 commits intomasterfrom
feat/single-tx-backtest

Conversation

@odyslam
Copy link
Contributor

@odyslam odyslam commented Jan 21, 2026

Summary

This PR adds single transaction backtesting and automatic internal call detection to the backtesting framework.

New Features

  • Single Transaction Backtesting: Test a specific transaction by passing only its hash via executeBacktestForTransaction()
  • Automatic Internal Call Detection: The system automatically detects transactions that call the target contract internally (not just direct calls), with smart fallback:
    1. trace_filter (fastest, requires Erigon/archive node)
    2. debug_traceBlockByNumber (slower but widely supported)
    3. debug_traceTransaction (slowest, per-transaction)
    4. Direct calls only (if no trace APIs available)
  • Auto-detect Script Path: No need to manually set CREDIBLE_STD_PATH - the script path is automatically discovered
  • Improved Logging: Clear feedback about which detection method is being used and when fallbacks occur

API Changes

Removed: useTraceFilter from BacktestingConfig - trace detection is now automatic with fallback.

// Single transaction backtesting (new)
BacktestingTypes.BacktestingResults memory results = executeBacktestForTransaction(
    0xe5ebeb502ae9ac441fc2912513a7deb9e82bc4d89da91ca41b5fdd51bb96a288,
    targetContract,
    type(MyAssertion).creationCode,
    MyAssertion.check.selector,
    rpcUrl
);

// Block range backtesting (useTraceFilter removed - now automatic)
BacktestingTypes.BacktestingResults memory results = executeBacktest(
    BacktestingTypes.BacktestingConfig({
        targetContract: 0x...,
        endBlock: 1000000,
        blockRange: 100,
        assertionCreationCode: type(MyAssertion).creationCode,
        assertionSelector: MyAssertion.check.selector,
        rpcUrl: "https://...",
        detailedBlocks: false,
        forkByTxHash: true
    })
);

Example Output

=== TRANSACTION DISCOVERY ===
Target: 0xd07822ee341c11a193869034d7e5f583c4a94872
Blocks: 28144849 to 28144849

[INFO] Detecting both direct calls AND internal/nested calls to target
[INFO] Trying trace APIs with automatic fallback...

[TRACE] Using trace_filter API (fastest method for internal call detection)
[TRACE] trace_filter not supported by this RPC endpoint
[TRACE] Falling back to debug_traceBlockByNumber (slower but widely supported)

=== DISCOVERY COMPLETE ===
[INFO] Detection method: debug_traceBlockByNumber
[INFO] Internal calls: ENABLED

Total transactions found: 1

Bug Fixes

  • Fix panic error decoding to show human-readable messages
  • Fix parsing when zero transactions are found
  • Fix whitespace trimming in data extraction
  • Fix set -e blocking trace method fallback

Test Plan

  • Single transaction backtesting works for direct calls
  • Single transaction backtesting works for internal calls
  • Block range backtesting finds internal calls via trace APIs
  • Automatic fallback chain works when trace methods are unsupported
  • Script path auto-detection works

🤖 Generated with Claude Code

odyslam and others added 7 commits January 21, 2026 14:47
Add the ability to run backtests for a specific transaction by passing
its hash. This is useful for debugging specific transactions or testing
assertions against known exploit transactions.

Changes:
- Add transactionHash field to BacktestingConfig struct
- Add executeBacktestForTransaction() convenience function
- Add internal helpers to fetch and parse transaction by hash via RPC
- Update existing tests to include new field
- Add example tests demonstrating the new feature

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When an assertion fails during backtesting, the transaction is now
re-executed without try/catch so Foundry prints the full execution
trace. This helps users debug why their assertions failed.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove assertion setup from trace replay - just fork at the tx hash
(state before tx) and make the raw call. This lets Foundry show the
actual execution trace without any wrapper interference.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove verbose separators and banners
- Simplify single line status messages
- Add clear ">>> TRANSACTION TRACE BELOW <<<" marker
- Remove redundant _printSingleTransactionResults function
- Keep trace replay minimal - just the call

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The decodeRevertReason function now handles:
- Panic(uint256) errors with human-readable panic code descriptions
- Error(string) standard revert messages
- Custom errors (shows selector hex)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Return empty array instead of reverting when no transactions found.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Prevents StringsInvalidChar error when parsing transaction count.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@odyslam odyslam force-pushed the feat/single-tx-backtest branch from d8abf4a to 3b08642 Compare January 21, 2026 19:47
odyslam and others added 4 commits January 21, 2026 14:50
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove transactionHash from BacktestingConfig struct to maintain backward
compatibility with existing projects. Single transaction backtesting is now
only available via the executeBacktestForTransaction() function.

API:
- executeBacktest(config) - for block range backtesting (unchanged)
- executeBacktestForTransaction(txHash, ...) - for single transaction

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add comprehensive NatSpec documentation to all source files
- Add @title, @author, @notice, @dev, @param, @return tags
- Add code examples in contract-level documentation
- Configure forge doc with proper title and repository URL
- Add GitHub Pages workflow for automatic documentation deployment
- Documentation will be available at the repo's GitHub Pages URL

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Reorganize documentation structure for clarity
- Add documentation link to GitHub Pages
- Update Quick Start with simpler examples
- Add PhEvm cheatcodes reference table
- Add trigger types documentation
- Update backtesting section with new APIs:
  - executeBacktestForTransaction() for single tx mode
  - Updated BacktestingConfig struct (no transactionHash field)
  - Trace replay feature documentation
  - Result categorization explanation
- Add StateChanges helper documentation
- Update resource links

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
czepluch
czepluch previously approved these changes Jan 22, 2026
Copy link
Contributor

@czepluch czepluch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, did you have time to try it out yet? I can give it a shot a bit later or tomorrow if not

Add automatic path detection for transaction_fetcher.sh using find
command as fallback when standard paths fail. This removes the need
to manually set CREDIBLE_STD_PATH in most cases.

- Add more standard search paths (Soldeer, git submodules, monorepo)
- Add _autoDetectScriptPath() that uses find to locate the script
- Cache results to avoid repeated filesystem lookups

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
odyslam and others added 6 commits January 22, 2026 13:53
Remove useTraceFilter option - trace detection is now automatic.
The system tries trace_filter first, then falls back to
debug_traceBlockByNumber, debug_traceTransaction, and finally
direct-calls-only if no trace APIs are supported.

- Remove useTraceFilter from BacktestingConfig struct
- Always enable trace detection in _fetchTransactions()
- Improve logging to show which detection method is used
- Add warnings when falling back to direct-calls-only mode

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use "|| status=$?" pattern to capture exit codes without triggering
set -e exit, allowing the fallback chain to work properly.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Track version history following Keep a Changelog format.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add 18 unit tests for BacktestingUtils parsing functions
- Fix broken integration tests (remove useTraceFilter field)
- Unit tests run in CI without RPC access
- Integration tests skip gracefully when RPC not available

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add tests for all precompile addresses to prevent accidental changes
- Add tests for mapping slot calculation (used in StateChanges)
- 7 new unit tests that run without external dependencies

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@odyslam odyslam merged commit 612c418 into master Jan 22, 2026
2 checks passed
@odyslam odyslam deleted the feat/single-tx-backtest branch January 22, 2026 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants