Skip to content

feat: add priority queue for protocol messages to prevent signature loss under high load#19

Open
awcjack wants to merge 2 commits intomasterfrom
priority-queue
Open

feat: add priority queue for protocol messages to prevent signature loss under high load#19
awcjack wants to merge 2 commits intomasterfrom
priority-queue

Conversation

@awcjack
Copy link
Owner

@awcjack awcjack commented Jan 5, 2026

Summary

Under high transaction load, snapshot signatures were being lost because all messages shared a single FIFO queue. ReqSn/AckSn protocol messages got buried behind ReqTx transactions, causing signature collection to fail.

Solution

Dual-queue system that processes protocol messages before transactions:

  • HighPriority: ReqSn, AckSn, ChainInput, ClientInput, ConnectivityEvent
  • LowPriority: ReqTx, ReqDec

This ensures protocol state machine messages are never starved by transaction load.

Changes

  • Added MessagePriority type and classifyPriority function in Hydra.HeadLogic.Input
  • Updated InputQueue to use dual queues (high/low priority)
  • High priority queue is always drained before processing low priority messages
  • Updated tests to verify priority queue behavior

Testing

  • All HeadLogic tests pass (69 tests)
  • All Behavior tests pass (45 tests)
  • InputQueue tests updated and pass

@github-actions
Copy link

github-actions bot commented Jan 5, 2026

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2026-01-06 00:13:17.641452035 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial c8a101a5c8ac4816b0dceb59ce31fc2258e387de828f02961d2f2045 2652
νCommit 61458bc2f297fff3cc5df6ac7ab57cefd87763b0b7bd722146a1035c 685
νHead a1442faf26d4ec409e2f62a685c1d4893f8d6bcbaf7bcb59d6fa1340 14599
μHead fd173b993e12103cd734ca6710d364e17120a5eb37a224c64ab2b188* 5284
νDeposit ae01dade3a9c346d5c93ae3ce339412b90a0b8f83f94ec6baa24e30c 1102
  • The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

Init transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 5840 10.57 3.36 0.52
2 6038 12.99 4.13 0.55
3 6238 14.71 4.65 0.58
5 6640 18.43 5.81 0.63
10 7647 29.31 9.25 0.79
43 14282 98.94 30.92 1.80

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 561 2.44 1.16 0.20
2 739 3.38 1.73 0.22
3 924 4.36 2.33 0.24
5 1282 6.41 3.60 0.28
10 2173 12.13 7.25 0.40
54 10048 98.61 68.52 1.88

CollectCom transaction costs

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 525 25.24 7.32 0.43
2 114 636 34.23 9.85 0.53
3 170 747 42.25 12.15 0.61
4 226 858 49.83 14.41 0.69
5 280 969 61.05 17.49 0.81
6 340 1081 71.10 20.29 0.92
7 393 1192 85.98 24.25 1.07
8 451 1303 86.83 24.81 1.09
9 504 1414 94.26 27.21 1.17

Cost of Increment Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 1807 24.37 7.71 0.48
2 1922 25.88 8.79 0.51
3 2132 28.47 10.18 0.55
5 2327 29.88 11.93 0.58
10 3230 43.14 18.95 0.78
39 7622 97.62 53.47 1.66

Cost of Decrement Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 638 22.54 7.30 0.41
2 700 22.55 7.95 0.42
3 884 25.47 9.47 0.46
5 1193 29.97 12.05 0.53
10 2049 39.63 18.05 0.69
43 6783 98.65 56.48 1.65

Close transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 678 29.13 8.90 0.48
2 824 31.54 10.25 0.52
3 952 33.47 11.46 0.55
5 1201 36.35 13.57 0.59
10 2228 47.15 20.05 0.77
37 6000 97.81 52.18 1.58

Contest transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 700 33.87 10.16 0.53
2 804 35.88 11.39 0.56
3 896 37.20 12.40 0.58
5 1223 41.90 15.05 0.65
10 2136 55.43 22.23 0.85
29 4779 97.69 46.61 1.49

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 5789 27.16 9.11 0.69
2 5966 35.89 12.06 0.79
3 6084 44.77 15.03 0.89
4 6213 51.68 17.35 0.97
5 6303 58.48 19.61 1.04
6 6606 74.66 25.13 1.23
7 6837 84.86 28.60 1.34
8 6838 88.46 29.77 1.38

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
10 0 0 5835 19.63 6.56 0.61
10 5 285 6004 30.67 10.88 0.74
10 10 570 6175 39.69 14.52 0.85
10 20 1139 6513 59.98 22.53 1.08
10 30 1706 6852 79.15 30.16 1.31
10 39 2218 7157 98.05 37.58 1.53

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2026-01-06 00:16:42.165077015 UTC

Baseline Scenario

Number of nodes 1
Number of txs 300
Avg. Confirmation Time (ms) 5.473456956
P99 7.609304379999963ms
P95 6.68419725ms
P50 5.2490615ms
Number of Invalid txs 0

Three local nodes

Number of nodes 3
Number of txs 900
Avg. Confirmation Time (ms) 38.691192806
P99 70.48709549999998ms
P95 57.64521705ms
P50 35.9569545ms
Number of Invalid txs 0

@github-actions
Copy link

github-actions bot commented Jan 5, 2026

Transaction cost differences

No cost or size differences found

…oss under high load

Under high transaction load, snapshot signatures were being lost because:
- All messages shared a single FIFO queue
- ReqSn/AckSn protocol messages got buried behind ReqTx transactions
- When AckSn arrived before local ReqSn was processed, it got re-enqueued
  to the back of the queue, causing signature collection to fail

Solution: Dual-queue system that processes protocol messages before transactions
- HighPriority: ReqSn, AckSn, ChainInput, ClientInput, ConnectivityEvent
- LowPriority: ReqTx, ReqDec

This ensures protocol state machine messages are never starved by transaction load.
GitHub Actions runners have limited disk space (~14GB available).
When building uncached Nix derivations (like our modified hydra-node),
the build can exhaust disk space during compilation.

This adds a cleanup step that removes unused tools before the build:
- .NET SDK (~1.8GB)
- Android SDK (~9GB)
- GHC (~5GB)
- CodeQL (~2.5GB)
- Unused Docker images

This frees up ~20GB of disk space, ensuring builds complete successfully.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant