Skip to content

EIP-4844: Add blobsSidecar db repos#4847

Merged
g11tech merged 1 commit intounstablefrom
dapplion/blobs-db
Dec 5, 2022
Merged

EIP-4844: Add blobsSidecar db repos#4847
g11tech merged 1 commit intounstablefrom
dapplion/blobs-db

Conversation

@dapplion
Copy link
Contributor

@dapplion dapplion commented Dec 5, 2022

Motivation

Description

BlobsSidecar have to be stored, served and prune with identical logic to blocks. So this two new repos mimic existing for blocks. See #4774 for how this repos will be used specifically

@dapplion dapplion requested a review from a team as a code owner December 5, 2022 08:43
@dapplion dapplion changed the title Add blobsSidecar db repos EIP-4844: Add blobsSidecar db repos Dec 5, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2022

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: f554e02 Previous: 1fde79d Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 3.1168 ms/op 2.3952 ms/op 1.30
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 100.69 us/op 79.363 us/op 1.27
BLS verify - blst-native 2.4172 ms/op 1.9986 ms/op 1.21
BLS verifyMultipleSignatures 3 - blst-native 4.9076 ms/op 4.2332 ms/op 1.16
BLS verifyMultipleSignatures 8 - blst-native 10.466 ms/op 8.8027 ms/op 1.19
BLS verifyMultipleSignatures 32 - blst-native 39.210 ms/op 34.314 ms/op 1.14
BLS aggregatePubkeys 32 - blst-native 51.097 us/op 48.821 us/op 1.05
BLS aggregatePubkeys 128 - blst-native 202.99 us/op 163.45 us/op 1.24
getAttestationsForBlock 134.31 ms/op 102.00 ms/op 1.32
isKnown best case - 1 super set check 567.00 ns/op 436.00 ns/op 1.30
isKnown normal case - 2 super set checks 578.00 ns/op 421.00 ns/op 1.37
isKnown worse case - 16 super set checks 572.00 ns/op 442.00 ns/op 1.29
CheckpointStateCache - add get delete 12.571 us/op 9.4070 us/op 1.34
validate gossip signedAggregateAndProof - struct 5.7662 ms/op 4.7969 ms/op 1.20
validate gossip attestation - struct 2.7420 ms/op 2.0977 ms/op 1.31
pickEth1Vote - no votes 3.0421 ms/op 2.0058 ms/op 1.52
pickEth1Vote - max votes 28.783 ms/op 21.064 ms/op 1.37
pickEth1Vote - Eth1Data hashTreeRoot value x2048 16.248 ms/op 11.254 ms/op 1.44
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 29.775 ms/op 21.036 ms/op 1.42
pickEth1Vote - Eth1Data fastSerialize value x2048 2.0369 ms/op 1.6766 ms/op 1.21
pickEth1Vote - Eth1Data fastSerialize tree x2048 19.982 ms/op 15.477 ms/op 1.29
bytes32 toHexString 1.6560 us/op 1.1290 us/op 1.47
bytes32 Buffer.toString(hex) 943.00 ns/op 746.00 ns/op 1.26
bytes32 Buffer.toString(hex) from Uint8Array 1.2740 us/op 1.0470 us/op 1.22
bytes32 Buffer.toString(hex) + 0x 938.00 ns/op 688.00 ns/op 1.36
Object access 1 prop 0.51700 ns/op 0.34100 ns/op 1.52
Map access 1 prop 0.38800 ns/op 0.27500 ns/op 1.41
Object get x1000 18.701 ns/op 13.368 ns/op 1.40
Map get x1000 1.0930 ns/op 0.75700 ns/op 1.44
Object set x1000 145.50 ns/op 85.592 ns/op 1.70
Map set x1000 93.475 ns/op 63.109 ns/op 1.48
Return object 10000 times 0.45410 ns/op 0.39670 ns/op 1.14
Throw Error 10000 times 7.6982 us/op 6.4669 us/op 1.19
fastMsgIdFn sha256 / 200 bytes 5.8130 us/op 5.1520 us/op 1.13
fastMsgIdFn h32 xxhash / 200 bytes 721.00 ns/op 579.00 ns/op 1.25
fastMsgIdFn h64 xxhash / 200 bytes 1.0280 us/op 721.00 ns/op 1.43
fastMsgIdFn sha256 / 1000 bytes 18.037 us/op 14.306 us/op 1.26
fastMsgIdFn h32 xxhash / 1000 bytes 891.00 ns/op 667.00 ns/op 1.34
fastMsgIdFn h64 xxhash / 1000 bytes 1.0190 us/op 881.00 ns/op 1.16
fastMsgIdFn sha256 / 10000 bytes 153.55 us/op 120.15 us/op 1.28
fastMsgIdFn h32 xxhash / 10000 bytes 3.1030 us/op 2.2780 us/op 1.36
fastMsgIdFn h64 xxhash / 10000 bytes 2.4460 us/op 1.7690 us/op 1.38
enrSubnets - fastDeserialize 64 bits 3.8130 us/op 2.7610 us/op 1.38
enrSubnets - ssz BitVector 64 bits 1.0540 us/op 861.00 ns/op 1.22
enrSubnets - fastDeserialize 4 bits 533.00 ns/op 364.00 ns/op 1.46
enrSubnets - ssz BitVector 4 bits 995.00 ns/op 695.00 ns/op 1.43
prioritizePeers score -10:0 att 32-0.1 sync 2-0 126.48 us/op 90.494 us/op 1.40
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 165.22 us/op 122.51 us/op 1.35
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 318.74 us/op 228.55 us/op 1.39
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 669.83 us/op 498.16 us/op 1.34
prioritizePeers score 0:0 att 64-1 sync 4-1 612.24 us/op 482.54 us/op 1.27
RateTracker 1000000 limit, 1 obj count per request 233.80 ns/op 163.35 ns/op 1.43
RateTracker 1000000 limit, 2 obj count per request 180.40 ns/op 116.16 ns/op 1.55
RateTracker 1000000 limit, 4 obj count per request 153.30 ns/op 97.823 ns/op 1.57
RateTracker 1000000 limit, 8 obj count per request 137.89 ns/op 89.142 ns/op 1.55
RateTracker with prune 7.5170 us/op 4.0900 us/op 1.84
array of 16000 items push then shift 6.0122 us/op 4.2745 us/op 1.41
LinkedList of 16000 items push then shift 23.303 ns/op 14.993 ns/op 1.55
array of 16000 items push then pop 313.22 ns/op 212.11 ns/op 1.48
LinkedList of 16000 items push then pop 21.254 ns/op 13.661 ns/op 1.56
array of 24000 items push then shift 8.7881 us/op 6.3622 us/op 1.38
LinkedList of 24000 items push then shift 23.378 ns/op 16.140 ns/op 1.45
array of 24000 items push then pop 296.31 ns/op 203.90 ns/op 1.45
LinkedList of 24000 items push then pop 22.499 ns/op 14.571 ns/op 1.54
intersect bitArray bitLen 8 13.576 ns/op 9.7590 ns/op 1.39
intersect array and set length 8 252.66 ns/op 152.06 ns/op 1.66
intersect bitArray bitLen 128 78.189 ns/op 57.488 ns/op 1.36
intersect array and set length 128 2.7819 us/op 1.7687 us/op 1.57
Buffer.concat 32 items 2.7240 ns/op 1.8720 ns/op 1.46
pass gossip attestations to forkchoice per slot 5.2855 ms/op 3.5583 ms/op 1.49
computeDeltas 6.7863 ms/op 5.1914 ms/op 1.31
computeProposerBoostScoreFromBalances 957.05 us/op 712.25 us/op 1.34
altair processAttestation - 250000 vs - 7PWei normalcase 6.0011 ms/op 3.7455 ms/op 1.60
altair processAttestation - 250000 vs - 7PWei worstcase 9.1169 ms/op 5.8901 ms/op 1.55
altair processAttestation - setStatus - 1/6 committees join 300.07 us/op 200.75 us/op 1.49
altair processAttestation - setStatus - 1/3 committees join 578.24 us/op 402.99 us/op 1.43
altair processAttestation - setStatus - 1/2 committees join 773.50 us/op 575.29 us/op 1.34
altair processAttestation - setStatus - 2/3 committees join 1.0080 ms/op 757.32 us/op 1.33
altair processAttestation - setStatus - 4/5 committees join 1.4103 ms/op 1.0110 ms/op 1.39
altair processAttestation - setStatus - 100% committees join 1.6917 ms/op 1.3092 ms/op 1.29
altair processBlock - 250000 vs - 7PWei normalcase 36.896 ms/op 26.738 ms/op 1.38
altair processBlock - 250000 vs - 7PWei normalcase hashState 48.493 ms/op 43.711 ms/op 1.11
altair processBlock - 250000 vs - 7PWei worstcase 118.46 ms/op 79.531 ms/op 1.49
altair processBlock - 250000 vs - 7PWei worstcase hashState 130.86 ms/op 98.296 ms/op 1.33
phase0 processBlock - 250000 vs - 7PWei normalcase 5.0696 ms/op 3.5202 ms/op 1.44
phase0 processBlock - 250000 vs - 7PWei worstcase 62.206 ms/op 43.190 ms/op 1.44
altair processEth1Data - 250000 vs - 7PWei normalcase 1.2524 ms/op 734.92 us/op 1.70
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 18.670 us/op 20.075 us/op 0.93
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 44.084 us/op 45.087 us/op 0.98
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 23.811 us/op 25.600 us/op 0.93
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 18.942 us/op 21.210 us/op 0.89
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 146.03 us/op 134.01 us/op 1.09
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.3726 ms/op 1.1139 ms/op 1.23
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 141069 19.107 ms/op 13.075 ms/op 1.46
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 250000 30.274 ms/op 23.415 ms/op 1.29
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 250000 102.82 ms/op 76.872 ms/op 1.34
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 250000 59.738 ms/op 44.208 ms/op 1.35
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 250000 161.53 ms/op 123.31 ms/op 1.31
Tree 40 250000 create 1.3949 s/op 757.50 ms/op 1.84
Tree 40 250000 get(125000) 363.66 ns/op 257.99 ns/op 1.41
Tree 40 250000 set(125000) 4.1225 us/op 2.1617 us/op 1.91
Tree 40 250000 toArray() 40.866 ms/op 32.374 ms/op 1.26
Tree 40 250000 iterate all - toArray() + loop 41.810 ms/op 32.960 ms/op 1.27
Tree 40 250000 iterate all - get(i) 148.02 ms/op 108.83 ms/op 1.36
MutableVector 250000 create 19.528 ms/op 13.369 ms/op 1.46
MutableVector 250000 get(125000) 15.499 ns/op 11.575 ns/op 1.34
MutableVector 250000 set(125000) 1.1465 us/op 545.94 ns/op 2.10
MutableVector 250000 toArray() 8.8149 ms/op 6.2579 ms/op 1.41
MutableVector 250000 iterate all - toArray() + loop 9.3833 ms/op 6.2043 ms/op 1.51
MutableVector 250000 iterate all - get(i) 3.8198 ms/op 2.6749 ms/op 1.43
Array 250000 create 8.8204 ms/op 5.6554 ms/op 1.56
Array 250000 clone - spread 5.1053 ms/op 3.1076 ms/op 1.64
Array 250000 get(125000) 2.3540 ns/op 1.3990 ns/op 1.68
Array 250000 set(125000) 2.4920 ns/op 1.3240 ns/op 1.88
Array 250000 iterate all - loop 158.73 us/op 119.56 us/op 1.33
effectiveBalanceIncrements clone Uint8Array 300000 150.46 us/op 64.298 us/op 2.34
effectiveBalanceIncrements clone MutableVector 300000 1.7730 us/op 934.00 ns/op 1.90
effectiveBalanceIncrements rw all Uint8Array 300000 321.57 us/op 242.30 us/op 1.33
effectiveBalanceIncrements rw all MutableVector 300000 385.47 ms/op 192.29 ms/op 2.00
phase0 afterProcessEpoch - 250000 vs - 7PWei 217.83 ms/op 165.34 ms/op 1.32
phase0 beforeProcessEpoch - 250000 vs - 7PWei 140.10 ms/op 99.949 ms/op 1.40
altair processEpoch - mainnet_e81889 711.46 ms/op 538.90 ms/op 1.32
mainnet_e81889 - altair beforeProcessEpoch 179.85 ms/op 130.98 ms/op 1.37
mainnet_e81889 - altair processJustificationAndFinalization 66.587 us/op 69.371 us/op 0.96
mainnet_e81889 - altair processInactivityUpdates 13.199 ms/op 9.3178 ms/op 1.42
mainnet_e81889 - altair processRewardsAndPenalties 115.12 ms/op 89.595 ms/op 1.28
mainnet_e81889 - altair processRegistryUpdates 13.154 us/op 14.945 us/op 0.88
mainnet_e81889 - altair processSlashings 3.9970 us/op 4.1410 us/op 0.97
mainnet_e81889 - altair processEth1DataReset 4.1630 us/op 4.3760 us/op 0.95
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.8057 ms/op 1.9670 ms/op 1.43
mainnet_e81889 - altair processSlashingsReset 26.652 us/op 27.487 us/op 0.97
mainnet_e81889 - altair processRandaoMixesReset 24.556 us/op 25.159 us/op 0.98
mainnet_e81889 - altair processHistoricalRootsUpdate 4.2320 us/op 3.9940 us/op 1.06
mainnet_e81889 - altair processParticipationFlagUpdates 14.159 us/op 15.278 us/op 0.93
mainnet_e81889 - altair processSyncCommitteeUpdates 2.9680 us/op 3.5080 us/op 0.85
mainnet_e81889 - altair afterProcessEpoch 224.29 ms/op 172.46 ms/op 1.30
phase0 processEpoch - mainnet_e58758 685.36 ms/op 493.85 ms/op 1.39
mainnet_e58758 - phase0 beforeProcessEpoch 323.42 ms/op 260.13 ms/op 1.24
mainnet_e58758 - phase0 processJustificationAndFinalization 65.398 us/op 63.352 us/op 1.03
mainnet_e58758 - phase0 processRewardsAndPenalties 164.16 ms/op 112.07 ms/op 1.46
mainnet_e58758 - phase0 processRegistryUpdates 34.602 us/op 34.207 us/op 1.01
mainnet_e58758 - phase0 processSlashings 3.1290 us/op 3.9930 us/op 0.78
mainnet_e58758 - phase0 processEth1DataReset 3.2120 us/op 3.8360 us/op 0.84
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 2.8351 ms/op 1.9498 ms/op 1.45
mainnet_e58758 - phase0 processSlashingsReset 15.558 us/op 19.124 us/op 0.81
mainnet_e58758 - phase0 processRandaoMixesReset 24.835 us/op 26.058 us/op 0.95
mainnet_e58758 - phase0 processHistoricalRootsUpdate 3.9720 us/op 4.3150 us/op 0.92
mainnet_e58758 - phase0 processParticipationRecordUpdates 23.791 us/op 24.125 us/op 0.99
mainnet_e58758 - phase0 afterProcessEpoch 181.50 ms/op 134.75 ms/op 1.35
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.8022 ms/op 1.9822 ms/op 1.41
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 3.0748 ms/op 2.2041 ms/op 1.40
altair processInactivityUpdates - 250000 normalcase 55.583 ms/op 47.645 ms/op 1.17
altair processInactivityUpdates - 250000 worstcase 68.651 ms/op 55.703 ms/op 1.23
phase0 processRegistryUpdates - 250000 normalcase 28.883 us/op 29.695 us/op 0.97
phase0 processRegistryUpdates - 250000 badcase_full_deposits 553.56 us/op 448.64 us/op 1.23
phase0 processRegistryUpdates - 250000 worstcase 0.5 284.08 ms/op 228.74 ms/op 1.24
altair processRewardsAndPenalties - 250000 normalcase 172.58 ms/op 135.80 ms/op 1.27
altair processRewardsAndPenalties - 250000 worstcase 105.74 ms/op 80.118 ms/op 1.32
phase0 getAttestationDeltas - 250000 normalcase 17.402 ms/op 11.512 ms/op 1.51
phase0 getAttestationDeltas - 250000 worstcase 16.529 ms/op 11.690 ms/op 1.41
phase0 processSlashings - 250000 worstcase 8.6769 ms/op 5.8091 ms/op 1.49
altair processSyncCommitteeUpdates - 250000 397.84 ms/op 267.40 ms/op 1.49
BeaconState.hashTreeRoot - No change 885.00 ns/op 460.00 ns/op 1.92
BeaconState.hashTreeRoot - 1 full validator 75.750 us/op 61.382 us/op 1.23
BeaconState.hashTreeRoot - 32 full validator 756.91 us/op 624.11 us/op 1.21
BeaconState.hashTreeRoot - 512 full validator 8.6843 ms/op 6.7505 ms/op 1.29
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 103.17 us/op 84.040 us/op 1.23
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.4794 ms/op 1.1719 ms/op 1.26
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 20.412 ms/op 14.657 ms/op 1.39
BeaconState.hashTreeRoot - 1 balances 77.197 us/op 58.336 us/op 1.32
BeaconState.hashTreeRoot - 32 balances 680.18 us/op 561.78 us/op 1.21
BeaconState.hashTreeRoot - 512 balances 6.9903 ms/op 5.7629 ms/op 1.21
BeaconState.hashTreeRoot - 250000 balances 117.78 ms/op 88.042 ms/op 1.34
aggregationBits - 2048 els - zipIndexesInBitList 50.191 us/op 21.679 us/op 2.32
regular array get 100000 times 60.912 us/op 45.457 us/op 1.34
wrappedArray get 100000 times 60.885 us/op 46.295 us/op 1.32
arrayWithProxy get 100000 times 39.050 ms/op 27.715 ms/op 1.41
ssz.Root.equals 672.00 ns/op 466.00 ns/op 1.44
byteArrayEquals 645.00 ns/op 413.00 ns/op 1.56
shuffle list - 16384 els 12.907 ms/op 9.1569 ms/op 1.41
shuffle list - 250000 els 187.51 ms/op 137.88 ms/op 1.36
processSlot - 1 slots 18.683 us/op 13.045 us/op 1.43
processSlot - 32 slots 2.4395 ms/op 1.8176 ms/op 1.34
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 497.19 us/op 357.18 us/op 1.39
getCommitteeAssignments - req 1 vs - 250000 vc 5.9347 ms/op 4.2755 ms/op 1.39
getCommitteeAssignments - req 100 vs - 250000 vc 8.5799 ms/op 6.1270 ms/op 1.40
getCommitteeAssignments - req 1000 vs - 250000 vc 9.2685 ms/op 6.4460 ms/op 1.44
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 10.180 ns/op 7.8900 ns/op 1.29
state getBlockRootAtSlot - 250000 vs - 7PWei 1.7530 us/op 924.44 ns/op 1.90
computeProposers - vc 250000 23.182 ms/op 15.780 ms/op 1.47
computeEpochShuffling - vc 250000 192.51 ms/op 144.12 ms/op 1.34
getNextSyncCommittee - vc 250000 383.22 ms/op 268.06 ms/op 1.43

by benchmarkbot/action

@dapplion dapplion added the spec-deneb 🐡 Issues targeting the Deneb spec version label Dec 5, 2022
g11tech
g11tech approved these changes Dec 5, 2022
View reviewed changes
Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@g11tech g11tech merged commit 0be7b28 into unstable Dec 5, 2022
@g11tech g11tech deleted the dapplion/blobs-db branch December 5, 2022 12:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@g11tech g11tech g11tech approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

spec-deneb 🐡 Issues targeting the Deneb spec version

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dapplion @g11tech