feat: add priority queue for protocol messages to prevent signature loss under high load#19
feat: add priority queue for protocol messages to prevent signature loss under high load#19
Conversation
Transaction costsSizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using
Script summary
|
| Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|
| 1 | 5840 | 10.57 | 3.36 | 0.52 |
| 2 | 6038 | 12.99 | 4.13 | 0.55 |
| 3 | 6238 | 14.71 | 4.65 | 0.58 |
| 5 | 6640 | 18.43 | 5.81 | 0.63 |
| 10 | 7647 | 29.31 | 9.25 | 0.79 |
| 43 | 14282 | 98.94 | 30.92 | 1.80 |
Commit transaction costs
This uses ada-only outputs for better comparability.
| UTxO | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|
| 1 | 561 | 2.44 | 1.16 | 0.20 |
| 2 | 739 | 3.38 | 1.73 | 0.22 |
| 3 | 924 | 4.36 | 2.33 | 0.24 |
| 5 | 1282 | 6.41 | 3.60 | 0.28 |
| 10 | 2173 | 12.13 | 7.25 | 0.40 |
| 54 | 10048 | 98.61 | 68.52 | 1.88 |
CollectCom transaction costs
| Parties | UTxO (bytes) | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|---|
| 1 | 57 | 525 | 25.24 | 7.32 | 0.43 |
| 2 | 114 | 636 | 34.23 | 9.85 | 0.53 |
| 3 | 170 | 747 | 42.25 | 12.15 | 0.61 |
| 4 | 226 | 858 | 49.83 | 14.41 | 0.69 |
| 5 | 280 | 969 | 61.05 | 17.49 | 0.81 |
| 6 | 340 | 1081 | 71.10 | 20.29 | 0.92 |
| 7 | 393 | 1192 | 85.98 | 24.25 | 1.07 |
| 8 | 451 | 1303 | 86.83 | 24.81 | 1.09 |
| 9 | 504 | 1414 | 94.26 | 27.21 | 1.17 |
Cost of Increment Transaction
| Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|
| 1 | 1807 | 24.37 | 7.71 | 0.48 |
| 2 | 1922 | 25.88 | 8.79 | 0.51 |
| 3 | 2132 | 28.47 | 10.18 | 0.55 |
| 5 | 2327 | 29.88 | 11.93 | 0.58 |
| 10 | 3230 | 43.14 | 18.95 | 0.78 |
| 39 | 7622 | 97.62 | 53.47 | 1.66 |
Cost of Decrement Transaction
| Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|
| 1 | 638 | 22.54 | 7.30 | 0.41 |
| 2 | 700 | 22.55 | 7.95 | 0.42 |
| 3 | 884 | 25.47 | 9.47 | 0.46 |
| 5 | 1193 | 29.97 | 12.05 | 0.53 |
| 10 | 2049 | 39.63 | 18.05 | 0.69 |
| 43 | 6783 | 98.65 | 56.48 | 1.65 |
Close transaction costs
| Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|
| 1 | 678 | 29.13 | 8.90 | 0.48 |
| 2 | 824 | 31.54 | 10.25 | 0.52 |
| 3 | 952 | 33.47 | 11.46 | 0.55 |
| 5 | 1201 | 36.35 | 13.57 | 0.59 |
| 10 | 2228 | 47.15 | 20.05 | 0.77 |
| 37 | 6000 | 97.81 | 52.18 | 1.58 |
Contest transaction costs
| Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|
| 1 | 700 | 33.87 | 10.16 | 0.53 |
| 2 | 804 | 35.88 | 11.39 | 0.56 |
| 3 | 896 | 37.20 | 12.40 | 0.58 |
| 5 | 1223 | 41.90 | 15.05 | 0.65 |
| 10 | 2136 | 55.43 | 22.23 | 0.85 |
| 29 | 4779 | 97.69 | 46.61 | 1.49 |
Abort transaction costs
There is some variation due to the random mixture of initial and already committed outputs.
| Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|
| 1 | 5789 | 27.16 | 9.11 | 0.69 |
| 2 | 5966 | 35.89 | 12.06 | 0.79 |
| 3 | 6084 | 44.77 | 15.03 | 0.89 |
| 4 | 6213 | 51.68 | 17.35 | 0.97 |
| 5 | 6303 | 58.48 | 19.61 | 1.04 |
| 6 | 6606 | 74.66 | 25.13 | 1.23 |
| 7 | 6837 | 84.86 | 28.60 | 1.34 |
| 8 | 6838 | 88.46 | 29.77 | 1.38 |
FanOut transaction costs
Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.
| Parties | UTxO | UTxO (bytes) | Tx size | % max Mem | % max CPU | Min fee ₳ |
|---|---|---|---|---|---|---|
| 10 | 0 | 0 | 5835 | 19.63 | 6.56 | 0.61 |
| 10 | 5 | 285 | 6004 | 30.67 | 10.88 | 0.74 |
| 10 | 10 | 570 | 6175 | 39.69 | 14.52 | 0.85 |
| 10 | 20 | 1139 | 6513 | 59.98 | 22.53 | 1.08 |
| 10 | 30 | 1706 | 6852 | 79.15 | 30.16 | 1.31 |
| 10 | 39 | 2218 | 7157 | 98.05 | 37.58 | 1.53 |
End-to-end benchmark results
This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.
Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.
Generated at 2026-01-06 00:16:42.165077015 UTC
Baseline Scenario
| Number of nodes | 1 |
|---|---|
| Number of txs | 300 |
| Avg. Confirmation Time (ms) | 5.473456956 |
| P99 | 7.609304379999963ms |
| P95 | 6.68419725ms |
| P50 | 5.2490615ms |
| Number of Invalid txs | 0 |
Three local nodes
| Number of nodes | 3 |
|---|---|
| Number of txs | 900 |
| Avg. Confirmation Time (ms) | 38.691192806 |
| P99 | 70.48709549999998ms |
| P95 | 57.64521705ms |
| P50 | 35.9569545ms |
| Number of Invalid txs | 0 |
Transaction cost differencesNo cost or size differences found |
…oss under high load Under high transaction load, snapshot signatures were being lost because: - All messages shared a single FIFO queue - ReqSn/AckSn protocol messages got buried behind ReqTx transactions - When AckSn arrived before local ReqSn was processed, it got re-enqueued to the back of the queue, causing signature collection to fail Solution: Dual-queue system that processes protocol messages before transactions - HighPriority: ReqSn, AckSn, ChainInput, ClientInput, ConnectivityEvent - LowPriority: ReqTx, ReqDec This ensures protocol state machine messages are never starved by transaction load.
d34ea28 to
4459947
Compare
GitHub Actions runners have limited disk space (~14GB available). When building uncached Nix derivations (like our modified hydra-node), the build can exhaust disk space during compilation. This adds a cleanup step that removes unused tools before the build: - .NET SDK (~1.8GB) - Android SDK (~9GB) - GHC (~5GB) - CodeQL (~2.5GB) - Unused Docker images This frees up ~20GB of disk space, ensuring builds complete successfully.
Summary
Under high transaction load, snapshot signatures were being lost because all messages shared a single FIFO queue. ReqSn/AckSn protocol messages got buried behind ReqTx transactions, causing signature collection to fail.
Solution
Dual-queue system that processes protocol messages before transactions:
This ensures protocol state machine messages are never starved by transaction load.
Changes
MessagePrioritytype andclassifyPriorityfunction inHydra.HeadLogic.InputInputQueueto use dual queues (high/low priority)Testing