Skip to content

Sim tests refactoring#4918

Merged
g11tech merged 5 commits intounstablefrom
nh/sim-refactor
Dec 19, 2022
Merged

Sim tests refactoring#4918
g11tech merged 5 commits intounstablefrom
nh/sim-refactor

Conversation

@nazarhussain
Copy link
Contributor

@nazarhussain nazarhussain commented Dec 17, 2022

Motivation

Some pending refactoring, suggested in previous discussions.

Description

Sim tests refactoring based on previous suggestions/feedback.

Steps to test or reproduce

Run all tests.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 17, 2022 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 278cea8 Previous: 2ece180 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.4508 ms/op 1.1539 ms/op 1.26
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 94.961 us/op 67.962 us/op 1.40
BLS verify - blst-native 2.4722 ms/op 1.8708 ms/op 1.32
BLS verifyMultipleSignatures 3 - blst-native 5.0815 ms/op 3.8168 ms/op 1.33
BLS verifyMultipleSignatures 8 - blst-native 10.809 ms/op 8.2030 ms/op 1.32
BLS verifyMultipleSignatures 32 - blst-native 40.096 ms/op 29.699 ms/op 1.35
BLS aggregatePubkeys 32 - blst-native 54.343 us/op 39.331 us/op 1.38
BLS aggregatePubkeys 128 - blst-native 198.82 us/op 153.56 us/op 1.29
getAttestationsForBlock 123.86 ms/op 86.234 ms/op 1.44
isKnown best case - 1 super set check 589.00 ns/op 418.00 ns/op 1.41
isKnown normal case - 2 super set checks 544.00 ns/op 407.00 ns/op 1.34
isKnown worse case - 16 super set checks 563.00 ns/op 409.00 ns/op 1.38
CheckpointStateCache - add get delete 11.174 us/op 8.0350 us/op 1.39
validate gossip signedAggregateAndProof - struct 5.7793 ms/op 4.2710 ms/op 1.35
validate gossip attestation - struct 2.7106 ms/op 2.0269 ms/op 1.34
pickEth1Vote - no votes 2.8950 ms/op 2.1299 ms/op 1.36
pickEth1Vote - max votes 22.347 ms/op 16.536 ms/op 1.35
pickEth1Vote - Eth1Data hashTreeRoot value x2048 16.400 ms/op 12.042 ms/op 1.36
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 27.847 ms/op 21.062 ms/op 1.32
pickEth1Vote - Eth1Data fastSerialize value x2048 1.6497 ms/op 1.1914 ms/op 1.38
pickEth1Vote - Eth1Data fastSerialize tree x2048 12.818 ms/op 9.9918 ms/op 1.28
bytes32 toHexString 1.2670 us/op 1.0100 us/op 1.25
bytes32 Buffer.toString(hex) 758.00 ns/op 554.00 ns/op 1.37
bytes32 Buffer.toString(hex) from Uint8Array 1.2670 us/op 921.00 ns/op 1.38
bytes32 Buffer.toString(hex) + 0x 768.00 ns/op 564.00 ns/op 1.36
Object access 1 prop 0.41500 ns/op 0.28000 ns/op 1.48
Map access 1 prop 0.35100 ns/op 0.25800 ns/op 1.36
Object get x1000 17.061 ns/op 16.007 ns/op 1.07
Map get x1000 1.1030 ns/op 0.99500 ns/op 1.11
Object set x1000 122.80 ns/op 109.46 ns/op 1.12
Map set x1000 80.586 ns/op 66.500 ns/op 1.21
Return object 10000 times 0.45400 ns/op 0.37420 ns/op 1.21
Throw Error 10000 times 8.5554 us/op 6.7567 us/op 1.27
fastMsgIdFn sha256 / 200 bytes 5.7210 us/op 4.3520 us/op 1.31
fastMsgIdFn h32 xxhash / 200 bytes 623.00 ns/op 440.00 ns/op 1.42
fastMsgIdFn h64 xxhash / 200 bytes 842.00 ns/op 631.00 ns/op 1.33
fastMsgIdFn sha256 / 1000 bytes 18.448 us/op 13.596 us/op 1.36
fastMsgIdFn h32 xxhash / 1000 bytes 807.00 ns/op 631.00 ns/op 1.28
fastMsgIdFn h64 xxhash / 1000 bytes 1.0230 us/op 700.00 ns/op 1.46
fastMsgIdFn sha256 / 10000 bytes 159.13 us/op 116.94 us/op 1.36
fastMsgIdFn h32 xxhash / 10000 bytes 2.9420 us/op 2.3190 us/op 1.27
fastMsgIdFn h64 xxhash / 10000 bytes 2.3010 us/op 1.6720 us/op 1.38
enrSubnets - fastDeserialize 64 bits 3.0310 us/op 2.1610 us/op 1.40
enrSubnets - ssz BitVector 64 bits 1.0420 us/op 760.00 ns/op 1.37
enrSubnets - fastDeserialize 4 bits 384.00 ns/op 279.00 ns/op 1.38
enrSubnets - ssz BitVector 4 bits 1.0310 us/op 777.00 ns/op 1.33
prioritizePeers score -10:0 att 32-0.1 sync 2-0 124.00 us/op 92.180 us/op 1.35
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 166.42 us/op 136.00 us/op 1.22
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 308.83 us/op 245.06 us/op 1.26
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 484.33 us/op 400.89 us/op 1.21
prioritizePeers score 0:0 att 64-1 sync 4-1 628.62 us/op 497.06 us/op 1.26
array of 16000 items push then shift 5.7331 us/op 2.8011 us/op 2.05
LinkedList of 16000 items push then shift 20.541 ns/op 17.260 ns/op 1.19
array of 16000 items push then pop 263.60 ns/op 211.99 ns/op 1.24
LinkedList of 16000 items push then pop 20.533 ns/op 16.528 ns/op 1.24
array of 24000 items push then shift 8.5889 us/op 3.9667 us/op 2.17
LinkedList of 24000 items push then shift 21.847 ns/op 19.886 ns/op 1.10
array of 24000 items push then pop 272.05 ns/op 212.41 ns/op 1.28
LinkedList of 24000 items push then pop 19.273 ns/op 17.829 ns/op 1.08
intersect bitArray bitLen 8 28.961 ns/op 29.410 ns/op 0.98
intersect array and set length 8 216.59 ns/op 155.05 ns/op 1.40
intersect bitArray bitLen 128 104.42 ns/op 108.08 ns/op 0.97
intersect array and set length 128 2.6902 us/op 2.2863 us/op 1.18
Buffer.concat 32 items 6.6680 us/op 5.1750 us/op 1.29
Uint8Array.set 32 items 5.1840 us/op 4.1410 us/op 1.25
pass gossip attestations to forkchoice per slot 8.0767 ms/op 6.2763 ms/op 1.29
computeDeltas 6.1545 ms/op 5.6342 ms/op 1.09
computeProposerBoostScoreFromBalances 2.9790 ms/op 2.8365 ms/op 1.05
altair processAttestation - 250000 vs - 7PWei normalcase 5.4997 ms/op 3.5855 ms/op 1.53
altair processAttestation - 250000 vs - 7PWei worstcase 9.2658 ms/op 7.0176 ms/op 1.32
altair processAttestation - setStatus - 1/6 committees join 262.08 us/op 205.04 us/op 1.28
altair processAttestation - setStatus - 1/3 committees join 523.57 us/op 392.20 us/op 1.33
altair processAttestation - setStatus - 1/2 committees join 755.37 us/op 548.25 us/op 1.38
altair processAttestation - setStatus - 2/3 committees join 988.82 us/op 706.63 us/op 1.40
altair processAttestation - setStatus - 4/5 committees join 1.3456 ms/op 987.82 us/op 1.36
altair processAttestation - setStatus - 100% committees join 1.5812 ms/op 1.1623 ms/op 1.36
altair processBlock - 250000 vs - 7PWei normalcase 35.207 ms/op 25.769 ms/op 1.37
altair processBlock - 250000 vs - 7PWei normalcase hashState 47.114 ms/op 39.204 ms/op 1.20
altair processBlock - 250000 vs - 7PWei worstcase 122.65 ms/op 78.656 ms/op 1.56
altair processBlock - 250000 vs - 7PWei worstcase hashState 129.04 ms/op 106.78 ms/op 1.21
phase0 processBlock - 250000 vs - 7PWei normalcase 4.8320 ms/op 3.4938 ms/op 1.38
phase0 processBlock - 250000 vs - 7PWei worstcase 59.961 ms/op 45.026 ms/op 1.33
altair processEth1Data - 250000 vs - 7PWei normalcase 1.1309 ms/op 729.53 us/op 1.55
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 23.646 us/op 9.8880 us/op 2.39
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 47.228 us/op 28.440 us/op 1.66
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 28.782 us/op 11.628 us/op 2.48
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 24.856 us/op 8.9710 us/op 2.77
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 155.63 us/op 99.359 us/op 1.57
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.3485 ms/op 1.0066 ms/op 1.34
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 16384 1.8381 ms/op 1.3440 ms/op 1.37
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 16384 1.7423 ms/op 1.3056 ms/op 1.33
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 16384 5.7832 ms/op 4.3179 ms/op 1.34
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 16384 3.8660 ms/op 2.8428 ms/op 1.36
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 16384 9.1389 ms/op 6.9202 ms/op 1.32
Tree 40 250000 create 1.0454 s/op 629.17 ms/op 1.66
Tree 40 250000 get(125000) 402.15 ns/op 282.10 ns/op 1.43
Tree 40 250000 set(125000) 2.7300 us/op 2.0914 us/op 1.31
Tree 40 250000 toArray() 39.598 ms/op 31.242 ms/op 1.27
Tree 40 250000 iterate all - toArray() + loop 38.524 ms/op 30.681 ms/op 1.26
Tree 40 250000 iterate all - get(i) 139.45 ms/op 108.84 ms/op 1.28
MutableVector 250000 create 17.480 ms/op 15.099 ms/op 1.16
MutableVector 250000 get(125000) 15.776 ns/op 13.124 ns/op 1.20
MutableVector 250000 set(125000) 809.79 ns/op 609.28 ns/op 1.33
MutableVector 250000 toArray() 7.3644 ms/op 6.1955 ms/op 1.19
MutableVector 250000 iterate all - toArray() + loop 7.4921 ms/op 7.2551 ms/op 1.03
MutableVector 250000 iterate all - get(i) 3.6944 ms/op 3.5261 ms/op 1.05
Array 250000 create 7.0054 ms/op 6.0561 ms/op 1.16
Array 250000 clone - spread 2.6195 ms/op 2.1183 ms/op 1.24
Array 250000 get(125000) 1.1500 ns/op 0.90700 ns/op 1.27
Array 250000 set(125000) 1.2340 ns/op 0.90300 ns/op 1.37
Array 250000 iterate all - loop 155.63 us/op 167.74 us/op 0.93
effectiveBalanceIncrements clone Uint8Array 300000 89.759 us/op 60.525 us/op 1.48
effectiveBalanceIncrements clone MutableVector 300000 710.00 ns/op 555.00 ns/op 1.28
effectiveBalanceIncrements rw all Uint8Array 300000 320.11 us/op 252.85 us/op 1.27
effectiveBalanceIncrements rw all MutableVector 300000 236.43 ms/op 163.19 ms/op 1.45
phase0 afterProcessEpoch - 250000 vs - 7PWei 218.58 ms/op 184.77 ms/op 1.18
phase0 beforeProcessEpoch - 250000 vs - 7PWei 109.92 ms/op 72.265 ms/op 1.52
altair processEpoch - mainnet_e81889 700.72 ms/op 492.80 ms/op 1.42
mainnet_e81889 - altair beforeProcessEpoch 188.97 ms/op 132.55 ms/op 1.43
mainnet_e81889 - altair processJustificationAndFinalization 80.850 us/op 21.737 us/op 3.72
mainnet_e81889 - altair processInactivityUpdates 12.468 ms/op 11.510 ms/op 1.08
mainnet_e81889 - altair processRewardsAndPenalties 120.06 ms/op 89.430 ms/op 1.34
mainnet_e81889 - altair processRegistryUpdates 13.044 us/op 2.9230 us/op 4.46
mainnet_e81889 - altair processSlashings 2.9000 us/op 724.00 ns/op 4.01
mainnet_e81889 - altair processEth1DataReset 2.8330 us/op 792.00 ns/op 3.58
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.6944 ms/op 2.3087 ms/op 1.17
mainnet_e81889 - altair processSlashingsReset 27.416 us/op 6.1460 us/op 4.46
mainnet_e81889 - altair processRandaoMixesReset 25.162 us/op 5.4640 us/op 4.61
mainnet_e81889 - altair processHistoricalRootsUpdate 4.0100 us/op 779.00 ns/op 5.15
mainnet_e81889 - altair processParticipationFlagUpdates 14.342 us/op 3.0260 us/op 4.74
mainnet_e81889 - altair processSyncCommitteeUpdates 3.1640 us/op 746.00 ns/op 4.24
mainnet_e81889 - altair afterProcessEpoch 228.70 ms/op 193.90 ms/op 1.18
phase0 processEpoch - mainnet_e58758 668.35 ms/op 517.34 ms/op 1.29
mainnet_e58758 - phase0 beforeProcessEpoch 309.51 ms/op 222.76 ms/op 1.39
mainnet_e58758 - phase0 processJustificationAndFinalization 67.134 us/op 26.668 us/op 2.52
mainnet_e58758 - phase0 processRewardsAndPenalties 164.57 ms/op 132.01 ms/op 1.25
mainnet_e58758 - phase0 processRegistryUpdates 34.387 us/op 9.9380 us/op 3.46
mainnet_e58758 - phase0 processSlashings 3.0500 us/op 678.00 ns/op 4.50
mainnet_e58758 - phase0 processEth1DataReset 2.9200 us/op 722.00 ns/op 4.04
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.8000 ms/op 1.9230 ms/op 1.46
mainnet_e58758 - phase0 processSlashingsReset 16.330 us/op 4.7330 us/op 3.45
mainnet_e58758 - phase0 processRandaoMixesReset 26.388 us/op 5.2880 us/op 4.99
mainnet_e58758 - phase0 processHistoricalRootsUpdate 3.6490 us/op 739.00 ns/op 4.94
mainnet_e58758 - phase0 processParticipationRecordUpdates 24.375 us/op 4.6140 us/op 5.28
mainnet_e58758 - phase0 afterProcessEpoch 190.01 ms/op 159.59 ms/op 1.19
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.5499 ms/op 2.6549 ms/op 0.96
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.9633 ms/op 3.4107 ms/op 0.87
altair processInactivityUpdates - 250000 normalcase 54.133 ms/op 37.709 ms/op 1.44
altair processInactivityUpdates - 250000 worstcase 54.726 ms/op 46.451 ms/op 1.18
phase0 processRegistryUpdates - 250000 normalcase 31.427 us/op 12.243 us/op 2.57
phase0 processRegistryUpdates - 250000 badcase_full_deposits 715.15 us/op 572.40 us/op 1.25
phase0 processRegistryUpdates - 250000 worstcase 0.5 258.46 ms/op 207.33 ms/op 1.25
altair processRewardsAndPenalties - 250000 normalcase 110.35 ms/op 130.69 ms/op 0.84
altair processRewardsAndPenalties - 250000 worstcase 160.77 ms/op 81.183 ms/op 1.98
phase0 getAttestationDeltas - 250000 normalcase 14.815 ms/op 12.143 ms/op 1.22
phase0 getAttestationDeltas - 250000 worstcase 15.468 ms/op 12.249 ms/op 1.26
phase0 processSlashings - 250000 worstcase 7.7418 ms/op 5.9141 ms/op 1.31
altair processSyncCommitteeUpdates - 250000 352.73 ms/op 261.94 ms/op 1.35
BeaconState.hashTreeRoot - No change 695.00 ns/op 456.00 ns/op 1.52
BeaconState.hashTreeRoot - 1 full validator 91.680 us/op 55.647 us/op 1.65
BeaconState.hashTreeRoot - 32 full validator 903.84 us/op 652.61 us/op 1.38
BeaconState.hashTreeRoot - 512 full validator 8.9224 ms/op 5.9322 ms/op 1.50
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 114.56 us/op 75.505 us/op 1.52
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.6713 ms/op 1.3102 ms/op 1.28
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 21.576 ms/op 15.370 ms/op 1.40
BeaconState.hashTreeRoot - 1 balances 81.862 us/op 58.514 us/op 1.40
BeaconState.hashTreeRoot - 32 balances 824.76 us/op 580.06 us/op 1.42
BeaconState.hashTreeRoot - 512 balances 7.8146 ms/op 5.7850 ms/op 1.35
BeaconState.hashTreeRoot - 250000 balances 111.68 ms/op 94.364 ms/op 1.18
aggregationBits - 2048 els - zipIndexesInBitList 31.697 us/op 28.554 us/op 1.11
regular array get 100000 times 65.225 us/op 67.370 us/op 0.97
wrappedArray get 100000 times 63.869 us/op 67.382 us/op 0.95
arrayWithProxy get 100000 times 36.359 ms/op 28.051 ms/op 1.30
ssz.Root.equals 1.3300 us/op 1.0790 us/op 1.23
byteArrayEquals 1.3080 us/op 1.0610 us/op 1.23
shuffle list - 16384 els 13.149 ms/op 12.572 ms/op 1.05
shuffle list - 250000 els 194.12 ms/op 165.54 ms/op 1.17
processSlot - 1 slots 18.503 us/op 11.511 us/op 1.61
processSlot - 32 slots 2.5516 ms/op 1.6784 ms/op 1.52
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 422.01 us/op 332.36 us/op 1.27
getCommitteeAssignments - req 1 vs - 250000 vc 6.2061 ms/op 5.6217 ms/op 1.10
getCommitteeAssignments - req 100 vs - 250000 vc 8.6201 ms/op 7.5874 ms/op 1.14
getCommitteeAssignments - req 1000 vs - 250000 vc 9.4482 ms/op 8.1159 ms/op 1.16
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 10.570 ns/op 9.6500 ns/op 1.10
state getBlockRootAtSlot - 250000 vs - 7PWei 1.6171 us/op 1.1349 us/op 1.42
computeProposers - vc 250000 21.444 ms/op 15.687 ms/op 1.37
computeEpochShuffling - vc 250000 199.22 ms/op 170.15 ms/op 1.17
getNextSyncCommittee - vc 250000 360.98 ms/op 260.60 ms/op 1.39

by benchmarkbot/action

@nazarhussain nazarhussain marked this pull request as ready for review December 17, 2022 18:42
@nazarhussain nazarhussain requested a review from a team as a code owner December 17, 2022 18:42
g11tech
g11tech reviewed Dec 17, 2022
View reviewed changes
packages/cli/test/utils/simulation/interfaces.ts
readonly keyManager: KeyManagerApi;
readonly localKeys: SecretKey[];
readonly remoteKeys: SecretKey[];
readonly keys: CLClientKeys;
Copy link
Contributor

@g11tech g11tech Dec 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but there can be both local and remote kets in the CL

Copy link
Contributor Author

@nazarhussain nazarhussain Dec 18, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes there can be, but as per @dapplion we don't want to make SIM tests complex and mix up the keys. You can skill have local or remote keys in multiple CL nodes.

Copy link
Contributor

@dapplion dapplion Dec 19, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLClientKeys represents both as an enum

@nazarhussain nazarhussain requested a review from g11tech December 18, 2022 19:44
dapplion
dapplion reviewed Dec 19, 2022
View reviewed changes
dapplion
dapplion reviewed Dec 19, 2022
View reviewed changes
Copy link
Contributor

@dapplion dapplion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, some minor comments

g11tech
g11tech previously approved these changes Dec 19, 2022
View reviewed changes
Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@g11tech g11tech enabled auto-merge (squash) December 19, 2022 07:34
g11tech
g11tech approved these changes Dec 19, 2022
View reviewed changes
Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@g11tech g11tech merged commit c337f0e into unstable Dec 19, 2022
@g11tech g11tech deleted the nh/sim-refactor branch December 19, 2022 08:34
@dapplion
Copy link
Contributor

dapplion commented Dec 19, 2022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers

@dapplion dapplion dapplion left review comments

@g11tech g11tech g11tech approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nazarhussain @dapplion @g11tech