diff --git a/credible/backtesting-reference.mdx b/credible/backtesting-reference.mdx index f2cb18a..798a1b4 100644 --- a/credible/backtesting-reference.mdx +++ b/credible/backtesting-reference.mdx @@ -13,7 +13,7 @@ New to backtesting? Start with the [Backtesting Guide](/credible/backtesting) to ### `targetContract` -The address of the contract ([Assertion Adopter](/credible/glossary#assertion-adopter)) to test assertions against. The backtesting tool will find all transactions that interact with this contract (either directly or internally, depending on `useTraceFilter`). +The address of the contract ([Assertion Adopter](/credible/glossary#assertion-adopter)) to test assertions against. The backtesting tool will find all transactions that interact with this contract, including both direct calls and internal calls (e.g., through routers or aggregators). ```solidity targetContract: 0x5fd84259d66Cd46123540766Be93DFE6D43130D7 @@ -65,51 +65,12 @@ rpcUrl: "https://sepolia.optimism.io" rpcUrl: vm.envString("MAINNET_RPC_URL") ``` -### `useTraceFilter` - -Controls how transactions are detected in the block range. - -**`true` (recommended):** Uses the `trace_filter` RPC method -- Fetches all transactions in a single RPC call per batch -- Detects both direct calls and internal calls to the target contract -- **RPC calls for 100 blocks:** ~1-2 calls - -**`false`:** Scans each block individually -- One `eth_getBlockByNumber` call per block -- Only detects direct calls where `tx.to == targetContract` -- Does not detect internal calls (e.g., when another contract calls your target contract through a router or aggregator) -- **RPC calls for 100 blocks:** ~100 calls - -Use `trace_filter` for large block ranges or when you need to detect internal calls. Use block scanning when your RPC provider doesn't support `trace_filter`. - - -Not all RPC providers support `trace_filter`, and some providers only support it on certain networks. Check your provider's documentation to confirm `trace_filter` availability for your target chain. - - -```solidity -useTraceFilter: true // Recommended for most cases -useTraceFilter: false // For RPC providers and endpoints without trace_filter support -``` - ### `forkByTxHash` -Controls how the EVM state is forked for each transaction replay. - -**`false` (recommended):** Forks at the start of the block -- Uses `vm.createSelectFork(rpcUrl, blockNumber)` -- Faster and more RPC-efficient -- Suitable for most backtesting scenarios - -**`true`:** Forks at the exact transaction -- Uses `vm.createSelectFork(rpcUrl, txHash)` -- Replays all prior transactions in the block to get exact state -- Slower but necessary for transactions where transactions earlier in the block change the state of the target contract - -Use `false` for normal backtesting. Use `true` only when debugging specific failures where exact transaction state is needed. +Controls how the EVM state is forked for each transaction replay. This parameter is kept for interface compatibility but is effectively always `true` internally to ensure correct pre-transaction state. ```solidity -forkByTxHash: false // Recommended default -forkByTxHash: true // Only for debugging specific failures +forkByTxHash: true // Default and recommended ``` ### `detailedBlocks` @@ -120,38 +81,6 @@ Controls logging verbosity. Currently reserved for future functionality. detailedBlocks: false // Standard output ``` -## RPC Call Estimates - -Understanding RPC usage helps you plan for rate limits and choose the right RPC provider. - -### With trace_filter (Recommended) - -**Configuration:** -```solidity -useTraceFilter: true -``` - -**100 blocks, 50 matching transactions:** -- **Fetching:** 1-2 calls -- **Validation:** 50 calls (one per transaction) -- **Total:** ~52 RPC calls - -### Without trace_filter - -**Configuration:** -```solidity -useTraceFilter: false -``` - -**100 blocks, 50 matching transactions:** -- **Fetching:** 100 calls (one per block) -- **Validation:** 50 calls (one per transaction) -- **Total:** ~150 RPC calls - - -For large block ranges (>1,000 blocks), use paid RPC providers to avoid rate limiting. The `useTraceFilter` option significantly reduces RPC calls. - - ## API Reference ### executeBacktest @@ -181,12 +110,54 @@ BacktestingTypes.BacktestingResults memory results = executeBacktest( assertionSelector: MyAssertion.assertionInvariant.selector, rpcUrl: "https://sepolia.optimism.io", detailedBlocks: false, - useTraceFilter: true, - forkByTxHash: false + forkByTxHash: true }) ); ``` +### executeBacktestForTransaction + +Executes a backtest for a single transaction specified by its hash. This is useful for: +- Testing assertions against specific known transactions (such as historical exploits) +- Debugging potential false positives reported during staging (see [Testing Strategy](/credible/testing-strategy) for more on the staging workflow) +- Quickly validating assertion behavior on a problematic transaction without specifying a block range + +```solidity +function executeBacktestForTransaction( + bytes32 txHash, + address targetContract, + bytes memory assertionCreationCode, + bytes4 assertionSelector, + string memory rpcUrl +) public returns (BacktestingTypes.BacktestingResults memory results) +``` + +**Parameters:** +- `txHash`: The transaction hash to backtest +- `targetContract`: The target contract address (assertion adopter) +- `assertionCreationCode`: The assertion contract creation code +- `assertionSelector`: The assertion function selector +- `rpcUrl`: The RPC URL to use + +**Returns:** +- `results`: Results struct containing test execution statistics + +**Example:** +```solidity +bytes32 exploitTxHash = 0x1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef; + +BacktestingTypes.BacktestingResults memory results = executeBacktestForTransaction( + exploitTxHash, + TARGET_CONTRACT, + type(MyAssertion).creationCode, + MyAssertion.assertionInvariant.selector, + vm.envString("MAINNET_RPC_URL") +); + +// Expect the assertion to catch the exploit +assertEq(results.assertionFailures, 1, "Assertion should catch the exploit"); +``` + ### BacktestingConfig Configuration struct for backtesting parameters. @@ -200,7 +171,6 @@ struct BacktestingConfig { bytes4 assertionSelector; // Assertion function selector string rpcUrl; // RPC endpoint bool detailedBlocks; // Detailed logging (reserved) - bool useTraceFilter; // Use trace_filter API bool forkByTxHash; // Fork at exact transaction } ``` @@ -216,8 +186,7 @@ struct BacktestingConfig { | `assertionSelector` | `bytes4` | Function selector of the assertion to execute | | `rpcUrl` | `string` | RPC endpoint URL for blockchain data | | `detailedBlocks` | `bool` | Enable detailed logging (reserved for future use) | -| `useTraceFilter` | `bool` | Use trace_filter API for transaction detection | -| `forkByTxHash` | `bool` | Fork at exact transaction vs. block start | +| `forkByTxHash` | `bool` | Fork at exact transaction (kept for interface compatibility) | ### BacktestingResults @@ -228,8 +197,9 @@ struct BacktestingResults { uint256 totalTransactions; // Total transactions found uint256 processedTransactions; // Transactions validated uint256 successfulValidations; // Passing validations - uint256 failedValidations; // Failing validations + uint256 skippedTransactions; // Transactions where assertion wasn't triggered uint256 assertionFailures; // Protocol violations detected + uint256 replayFailures; // Transactions that failed to replay uint256 unknownErrors; // Unexpected errors } ``` @@ -241,8 +211,9 @@ struct BacktestingResults { | `totalTransactions` | `uint256` | Total number of transactions found in the block range | | `processedTransactions` | `uint256` | Number of transactions that were validated | | `successfulValidations` | `uint256` | Transactions that passed validation | -| `failedValidations` | `uint256` | Transactions that failed validation | -| `assertionFailures` | `uint256` | Number of protocol violations detected | +| `skippedTransactions` | `uint256` | Transactions where the assertion wasn't triggered (function selector didn't match) | +| `assertionFailures` | `uint256` | Number of protocol violations detected (assertion reverted) | +| `replayFailures` | `uint256` | Transactions that failed to replay before assertion could execute | | `unknownErrors` | `uint256` | Unexpected errors during execution | **Interpreting Results:** @@ -259,7 +230,8 @@ if (results.assertionFailures > 0) { // Log detailed results console.log("Total transactions:", results.totalTransactions); console.log("Successful validations:", results.successfulValidations); -console.log("Failed validations:", results.failedValidations); +console.log("Skipped transactions:", results.skippedTransactions); +console.log("Assertion failures:", results.assertionFailures); ``` **Common Causes of Assertion Failures:** @@ -285,8 +257,7 @@ BacktestingTypes.BacktestingConfig({ assertionSelector: MyAssertion.assertionInvariant.selector, rpcUrl: vm.envString("MAINNET_RPC_URL"), detailedBlocks: false, - useTraceFilter: true, // Efficient fetching - forkByTxHash: false // Fast validation + forkByTxHash: true }) ``` @@ -295,6 +266,9 @@ BacktestingTypes.BacktestingConfig({ ## Learn More + + Autogenerated API documentation for backtesting + Learn how to use backtesting to validate assertions diff --git a/credible/backtesting.mdx b/credible/backtesting.mdx index e70cd63..c1f3309 100644 --- a/credible/backtesting.mdx +++ b/credible/backtesting.mdx @@ -13,14 +13,12 @@ If you're looking for the reference documentation on all configuration parameter Backtesting operates in two phases: -1. **Transaction Fetching**: Identifies all transactions to the target contract in the block range +1. **Transaction Fetching**: Identifies all transactions to the target contract in the block range using trace APIs (with automatic fallback) 2. **Transaction Validation**: Replays each transaction with your assertion enabled -The tool fetches transactions using one of two methods: -- **trace_filter API** (recommended): Single RPC call for the entire block range, detects both direct and internal calls (e.g., when Contract A calls your target Contract B through a router or aggregator) -- **Block scanning**: One RPC call per block, detects only direct calls where `tx.to` equals your target contract (does not detect internal calls) +The tool automatically detects both direct calls and internal calls (e.g., when Contract A calls your target Contract B through a router or aggregator) using trace APIs. -For validation, the tool forks the EVM at each transaction to the target contract and replays it against the specified assertion. +For validation, the tool forks the EVM at the exact transaction state and replays it against the specified assertion. ## Example @@ -28,8 +26,8 @@ For validation, the tool forks the EVM at each transaction to the target contrac // SPDX-License-Identifier: MIT pragma solidity ^0.8.20; -import {CredibleTestWithBacktesting} from "../src/backtesting/CredibleTestWithBacktesting.sol"; -import {BacktestingTypes} from "../src/backtesting/BacktestingTypes.sol"; +import {CredibleTestWithBacktesting} from "credible-std/CredibleTestWithBacktesting.sol"; +import {BacktestingTypes} from "credible-std/utils/BacktestingTypes.sol"; import {MyAssertion} from "../assertions/src/MyAssertion.a.sol"; contract MyBacktestingTest is CredibleTestWithBacktesting { @@ -43,13 +41,12 @@ contract MyBacktestingTest is CredibleTestWithBacktesting { assertionSelector: MyAssertion.assertionInvariant.selector, rpcUrl: "https://sepolia.optimism.io", detailedBlocks: false, - useTraceFilter: true, - forkByTxHash: false + forkByTxHash: true }) ); // Check results - assert(results.assertionFailures == 0, "Found protocol violations!"); + assertEq(results.assertionFailures, 0, "Found protocol violations!"); } } ``` @@ -62,7 +59,7 @@ Consider excluding backtesting from your CI/CD pipeline as it can take time to r ## Common Configurations -### Large Block Ranges (100+ blocks) +### Standard Configuration ```solidity BacktestingTypes.BacktestingConfig({ @@ -73,51 +70,11 @@ BacktestingTypes.BacktestingConfig({ assertionSelector: MyAssertion.assertionInvariant.selector, rpcUrl: vm.envString("MAINNET_RPC_URL"), detailedBlocks: false, - useTraceFilter: true, // Fast fetching - forkByTxHash: false // Fast validation -}) -``` - -Expected runtime: ~20 seconds for 100 blocks with 28 transactions - -### Debugging Failures - -```solidity -BacktestingTypes.BacktestingConfig({ - targetContract: TARGET_CONTRACT, - endBlock: FAILING_BLOCK, - blockRange: 5, // Small range around failure - assertionCreationCode: type(MyAssertion).creationCode, - assertionSelector: MyAssertion.assertionInvariant.selector, - rpcUrl: vm.envString("MAINNET_RPC_URL"), - detailedBlocks: false, - useTraceFilter: true, - forkByTxHash: true // Exact state for debugging -}) -``` - -Use this when investigating why a specific transaction failed. The `forkByTxHash: true` setting replays all prior transactions to get the exact state just before execution. -This is disabled by default, since it uses significantly more RPC calls and hence is slower. It is recommended to use this setting when debugging failures for concentrated block intervals. - -### Quick Tests (< 20 blocks) - -```solidity -BacktestingTypes.BacktestingConfig({ - targetContract: TARGET_CONTRACT, - endBlock: RECENT_BLOCK, - blockRange: 10, - assertionCreationCode: type(MyAssertion).creationCode, - assertionSelector: MyAssertion.assertionInvariant.selector, - rpcUrl: vm.envString("MAINNET_RPC_URL"), - detailedBlocks: false, - useTraceFilter: false, // Simple method sufficient - forkByTxHash: false + forkByTxHash: true }) ``` -For small ranges, the simpler block scanning method is adequate. - -### Testing Against a Specific Transaction +### Testing Against a Specific Block When you want to test your assertion against a specific known transaction (such as a historical exploit), set `blockRange: 1` with the block number containing that transaction: @@ -130,7 +87,6 @@ BacktestingTypes.BacktestingConfig({ assertionSelector: BatchSwapDeltaAssertion.assertionBatchSwapRateManipulation.selector, rpcUrl: vm.envString("MAINNET_RPC_URL"), detailedBlocks: false, - useTraceFilter: true, forkByTxHash: true }) ``` @@ -144,6 +100,27 @@ This is particularly useful for: See the [Balancer V2 Rate Manipulation Exploit](/assertions-book/previous-hacks/balancer-v2-stable-rate-exploit) for a real-world example of using backtesting to verify an assertion catches a historical exploit transaction. +### Testing a Single Transaction by Hash + +For debugging false positives or validating against a specific transaction, use `executeBacktestForTransaction`: + +```solidity +bytes32 txHash = 0x1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef; + +BacktestingTypes.BacktestingResults memory results = executeBacktestForTransaction( + txHash, + TARGET_CONTRACT, + type(MyAssertion).creationCode, + MyAssertion.assertionInvariant.selector, + vm.envString("MAINNET_RPC_URL") +); +``` + +This is particularly useful for: +- Debugging potential false positives reported during staging (see [Testing Strategy](/credible/testing-strategy)) +- Quick validation of assertion behavior on a specific transaction +- Investigating incidents without needing to know the block number + ## Understanding Results When backtesting completes, transactions are categorized into different result types to help you quickly identify issues: @@ -154,18 +131,22 @@ When backtesting completes, transactions are categorized into different result t - Transaction replayed and assertion validated without errors - **Recommended Action:** None - everything is working as expected -**NEEDS_REVIEW** - Transaction requires review -- Transaction either called a function not monitored by your assertion, or failed to replay in the test environment +**SKIP** - Assertion not triggered +- Transaction called a function not monitored by your assertion (function selector didn't match) +- This is normal behavior - your assertion only triggers on specific function calls +- **Recommended Action:** None required, unless you expected this transaction to trigger your assertion +- **Note:** These are NOT assertion violations - they indicate transactions that aren't relevant to your assertion + +**REPLAY_FAIL** - Transaction replay failed +- Transaction failed to replay before the assertion could execute - Common causes: - - **Function mismatch:** Assertion monitors `batch()` but transaction called `call()` - - **State dependencies:** Transaction depends on earlier transactions in the same block - - **Context requirements:** Transaction requires specific block state that isn't present when forking at block level -- **Recommended Action:** - - If many transactions show this status, enable `forkByTxHash: true` to fork at exact transaction state - - Review to ensure your assertion is triggering on the correct function -- **Note:** These are NOT assertion violations - they're informational and indicate transactions that either aren't relevant to your assertion or fail before the assertion is executed due to wrong configuration of the backtest + - **State dependencies:** Transaction depends on specific state that isn't present + - **Context requirements:** Transaction requires specific block context + - **Insufficient funds:** Sender balance changed between original execution and replay +- **Recommended Action:** Check the error message for details; this usually indicates an environmental issue rather than an assertion problem +- **Note:** These are NOT assertion violations -**ASSERTION_FAIL** - Assertion reverted +**FAIL** - Assertion reverted - Your assertion reverted when validating this transaction - **Most Common Causes:** - **False positive:** Your assertion logic incorrectly flags legitimate protocol behavior (most common) @@ -178,7 +159,7 @@ When backtesting completes, transactions are categorized into different result t - Check if the assertion is running out of gas and needs optimization - Verify the transaction on a block explorer to understand what it does -**UNKNOWN_ERROR** - Unexpected failure +**ERROR** - Unexpected failure - An error occurred that doesn't fit other categories - May indicate RPC issues, assertion bugs, or unexpected contract behavior - **Recommended Action:** Check the error message and retry; if persistent, [file a bug report](https://github.com/phylaxsystems/credible-std/issues). @@ -195,13 +176,14 @@ Block Range: 23697580 - 23697590 Total Transactions: 15 Processed Transactions: 15 Successful Validations: 10 -Failed Validations: 5 +Skipped Transactions: 3 +Failed Validations: 2 === ERROR BREAKDOWN === Protocol Violations (Assertion Failures): 1 -Needs Review (Selector Mismatch or Prestate Issues): 4 +Replay Failures (Tx reverted before assertion): 1 -Success Rate: 66% +Success Rate: 83% ================================ ``` @@ -242,12 +224,11 @@ See [CI/CD Integration](/credible/ci-cd-integration#foundry-profile-configuratio ## Best Practices 1. **Start small** - Test with 10-20 blocks first to verify your setup -2. **Use trace_filter** - Set `useTraceFilter: true` if your RPC provider supports it -3. **Keep forkByTxHash false** - Enable if assertions are reverting and might be affected by txs earlier in a block -4. **Use paid RPC providers** - For block ranges over 1,000 -5. **Check assertion failures** - Ensure `results.assertionFailures == 0` -6. **Run before deployment** - Catch false positives on real transactions -7. **Run manually** - Exclude from CI/CD to avoid long test runs +2. **Use paid RPC providers** - For block ranges over 1,000 +3. **Check assertion failures** - Ensure `results.assertionFailures == 0` +4. **Run before deployment** - Catch false positives on real transactions +5. **Run manually** - Exclude from CI/CD to avoid long test runs +6. **Debug with single transactions** - Use `executeBacktestForTransaction` to investigate specific failures **Learn More:** - [Backtesting Reference](/credible/backtesting-reference) - Complete configuration details diff --git a/credible/cheatcodes-reference.mdx b/credible/cheatcodes-reference.mdx index b2bcabb..0c21871 100644 --- a/credible/cheatcodes-reference.mdx +++ b/credible/cheatcodes-reference.mdx @@ -407,6 +407,9 @@ function assertionExample() external { ## Next Steps + + Autogenerated API documentation for credible-std + Learn how to write assertions using cheatcodes diff --git a/credible/cli-reference.mdx b/credible/cli-reference.mdx index abb8855..ddba9a3 100644 --- a/credible/cli-reference.mdx +++ b/credible/cli-reference.mdx @@ -36,7 +36,7 @@ Commands: help Print this message or the help of the given subcommand(s) Options: - -u, --auth-url Base URL for authentication service [env: PCL_AUTH_URL=] [default: https://dapp.internal.phylax.systems] + -u, --auth-url Base URL for authentication service [env: PCL_AUTH_URL=] [default: https://app.phylax.systems] -h, --help Print help ``` @@ -110,7 +110,7 @@ Arguments: Options: -u, --da-url URL of the assertion-DA server [env: PCL_DA_URL=] - [default: https://da.internal.phylax.systems] + [default: https://da.phylax.systems] --root Root directory of the project -h, --help Print help (see more with '--help') ``` @@ -146,7 +146,7 @@ Arguments: [CONSTRUCTOR_ARGS]... Constructor arguments for the assertion Options: - -u, --api-url Base URL for the Credible Layer dApp API [env: PCL_API_URL=] [default: https://dapp.internal.phylax.systems/api/v1] + -u, --api-url Base URL for the Credible Layer dApp API [env: PCL_API_URL=] [default: https://app.phylax.systems/api/v1] -p, --project-name Optional project name to skip interactive selection -a, --assertion Assertion in format 'Name(arg1,arg2)'. Use multiple -a flags for multiple assertions. -h, --help Print help (see more with '--help') @@ -205,7 +205,7 @@ Delete the current configuration file. ### `PCL_AUTH_URL` -Base URL for authentication service. Default: `https://dapp.internal.phylax.systems` +Base URL for authentication service. Default: `https://app.phylax.systems` ```bash export PCL_AUTH_URL=https://custom-auth-url.com @@ -214,7 +214,7 @@ pcl auth login ### `PCL_DA_URL` -URL of the [Assertion DA](/credible/glossary#assertion-da-data-availability) server. Default: `https://da.internal.phylax.systems` +URL of the [Assertion DA](/credible/glossary#assertion-da-data-availability) server. Default: `https://da.phylax.systems` ```bash export PCL_DA_URL=https://custom-da-url.com @@ -223,7 +223,7 @@ pcl store MyAssertion ### `PCL_API_URL` -Base URL for the [Credible Layer dApp](/credible/glossary#credible-dapp) API. Default: `https://dapp.internal.phylax.systems/api/v1` +Base URL for the [Credible Layer dApp](/credible/glossary#credible-dapp) API. Default: `https://app.phylax.systems/api/v1` ```bash export PCL_API_URL=https://custom-api-url.com/api/v1 diff --git a/credible/credible-std-overview.mdx b/credible/credible-std-overview.mdx index 4c75271..90ef42f 100644 --- a/credible/credible-std-overview.mdx +++ b/credible/credible-std-overview.mdx @@ -5,7 +5,7 @@ description: "Overview of the Phylax credible-std Library" `credible-std` is the standard library for implementing assertions in the Credible Layer. It provides the core contracts and interfaces needed to create and manage assertions for smart contract security monitoring. -You can find the repository on [GitHub](https://github.com/phylaxsystems/credible-std). +You can find the repository on [GitHub](https://github.com/phylaxsystems/credible-std) and the [full API reference](https://phylaxsystems.github.io/credible-std/index.html). ## Installation