Skip to content

EIP-4844: Update produce block flow#4864

Merged
g11tech merged 2 commits intounstablefrom
dapplion/eip-4844-blockproduction
Dec 6, 2022
Merged

EIP-4844: Update produce block flow#4864
g11tech merged 2 commits intounstablefrom
dapplion/eip-4844-blockproduction

Conversation

@dapplion
Copy link
Contributor

@dapplion dapplion commented Dec 6, 2022

Motivation

Description

  • Update block production flow to produce blobs if necessary. Since validator does not have to sign the blobsSidecar, it stays cached in the beacon node until the validator calls publishBlock back.
  • Uses EIP-4844: Add engine API getBlobsBundle method #4843 to create blobs

@dapplion dapplion requested a review from a team as a code owner December 6, 2022 09:39
@github-actions
Copy link
Contributor

github-actions bot commented Dec 6, 2022

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: ebe2be9 Previous: e0fe73c Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 1.5458 ms/op 3.0705 ms/op 0.50
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 59.359 us/op 127.40 us/op 0.47
BLS verify - blst-native 2.1704 ms/op 2.5454 ms/op 0.85
BLS verifyMultipleSignatures 3 - blst-native 4.4882 ms/op 5.3547 ms/op 0.84
BLS verifyMultipleSignatures 8 - blst-native 9.6995 ms/op 11.285 ms/op 0.86
BLS verifyMultipleSignatures 32 - blst-native 35.262 ms/op 40.225 ms/op 0.88
BLS aggregatePubkeys 32 - blst-native 46.505 us/op 53.379 us/op 0.87
BLS aggregatePubkeys 128 - blst-native 181.66 us/op 200.48 us/op 0.91
getAttestationsForBlock 76.276 ms/op 137.19 ms/op 0.56
isKnown best case - 1 super set check 467.00 ns/op 629.00 ns/op 0.74
isKnown normal case - 2 super set checks 457.00 ns/op 564.00 ns/op 0.81
isKnown worse case - 16 super set checks 460.00 ns/op 592.00 ns/op 0.78
CheckpointStateCache - add get delete 8.6770 us/op 11.993 us/op 0.72
validate gossip signedAggregateAndProof - struct 5.0422 ms/op 5.6547 ms/op 0.89
validate gossip attestation - struct 2.3810 ms/op 2.7430 ms/op 0.87
pickEth1Vote - no votes 2.3924 ms/op 2.7590 ms/op 0.87
pickEth1Vote - max votes 17.382 ms/op 27.686 ms/op 0.63
pickEth1Vote - Eth1Data hashTreeRoot value x2048 12.061 ms/op 15.450 ms/op 0.78
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 18.839 ms/op 27.126 ms/op 0.69
pickEth1Vote - Eth1Data fastSerialize value x2048 1.3726 ms/op 1.9311 ms/op 0.71
pickEth1Vote - Eth1Data fastSerialize tree x2048 11.193 ms/op 18.468 ms/op 0.61
bytes32 toHexString 897.00 ns/op 1.4510 us/op 0.62
bytes32 Buffer.toString(hex) 709.00 ns/op 960.00 ns/op 0.74
bytes32 Buffer.toString(hex) from Uint8Array 941.00 ns/op 1.3380 us/op 0.70
bytes32 Buffer.toString(hex) + 0x 697.00 ns/op 964.00 ns/op 0.72
Object access 1 prop 0.33900 ns/op 0.51000 ns/op 0.66
Map access 1 prop 0.30200 ns/op 0.38000 ns/op 0.79
Object get x1000 11.598 ns/op 18.744 ns/op 0.62
Map get x1000 0.99300 ns/op 1.2100 ns/op 0.82
Object set x1000 69.871 ns/op 133.45 ns/op 0.52
Map set x1000 45.370 ns/op 89.013 ns/op 0.51
Return object 10000 times 0.43850 ns/op 0.44940 ns/op 0.98
Throw Error 10000 times 6.1396 us/op 7.7545 us/op 0.79
fastMsgIdFn sha256 / 200 bytes 4.7830 us/op 5.4940 us/op 0.87
fastMsgIdFn h32 xxhash / 200 bytes 547.00 ns/op 712.00 ns/op 0.77
fastMsgIdFn h64 xxhash / 200 bytes 775.00 ns/op 920.00 ns/op 0.84
fastMsgIdFn sha256 / 1000 bytes 15.352 us/op 17.388 us/op 0.88
fastMsgIdFn h32 xxhash / 1000 bytes 703.00 ns/op 890.00 ns/op 0.79
fastMsgIdFn h64 xxhash / 1000 bytes 863.00 ns/op 997.00 ns/op 0.87
fastMsgIdFn sha256 / 10000 bytes 133.67 us/op 151.99 us/op 0.88
fastMsgIdFn h32 xxhash / 10000 bytes 2.5740 us/op 2.7490 us/op 0.94
fastMsgIdFn h64 xxhash / 10000 bytes 1.7850 us/op 2.1270 us/op 0.84
enrSubnets - fastDeserialize 64 bits 2.6590 us/op 3.3070 us/op 0.80
enrSubnets - ssz BitVector 64 bits 798.00 ns/op 951.00 ns/op 0.84
enrSubnets - fastDeserialize 4 bits 360.00 ns/op 536.00 ns/op 0.67
enrSubnets - ssz BitVector 4 bits 798.00 ns/op 956.00 ns/op 0.83
prioritizePeers score -10:0 att 32-0.1 sync 2-0 81.124 us/op 123.62 us/op 0.66
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 134.22 us/op 175.44 us/op 0.77
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 196.42 us/op 314.75 us/op 0.62
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 345.69 us/op 566.13 us/op 0.61
prioritizePeers score 0:0 att 64-1 sync 4-1 445.43 us/op 609.56 us/op 0.73
RateTracker 1000000 limit, 1 obj count per request 180.37 ns/op 221.20 ns/op 0.82
RateTracker 1000000 limit, 2 obj count per request 131.54 ns/op 166.66 ns/op 0.79
RateTracker 1000000 limit, 4 obj count per request 110.54 ns/op 139.47 ns/op 0.79
RateTracker 1000000 limit, 8 obj count per request 95.348 ns/op 132.13 ns/op 0.72
RateTracker with prune 3.7320 us/op 6.2060 us/op 0.60
array of 16000 items push then shift 51.576 us/op 5.4048 us/op 9.54
LinkedList of 16000 items push then shift 12.112 ns/op 21.639 ns/op 0.56
array of 16000 items push then pop 200.17 ns/op 287.00 ns/op 0.70
LinkedList of 16000 items push then pop 11.694 ns/op 21.092 ns/op 0.55
array of 24000 items push then shift 77.336 us/op 8.3539 us/op 9.26
LinkedList of 24000 items push then shift 12.225 ns/op 22.879 ns/op 0.53
array of 24000 items push then pop 191.56 ns/op 265.54 ns/op 0.72
LinkedList of 24000 items push then pop 11.801 ns/op 21.325 ns/op 0.55
intersect bitArray bitLen 8 10.594 ns/op 13.118 ns/op 0.81
intersect array and set length 8 127.39 ns/op 219.71 ns/op 0.58
intersect bitArray bitLen 128 55.994 ns/op 73.912 ns/op 0.76
intersect array and set length 128 1.7008 us/op 2.6630 us/op 0.64
Buffer.concat 32 items 1.7280 ns/op 2.5140 ns/op 0.69
pass gossip attestations to forkchoice per slot 3.5505 ms/op 5.0981 ms/op 0.70
computeDeltas 4.4160 ms/op 6.2576 ms/op 0.71
computeProposerBoostScoreFromBalances 805.02 us/op 942.80 us/op 0.85
altair processAttestation - 250000 vs - 7PWei normalcase 3.2106 ms/op 5.9154 ms/op 0.54
altair processAttestation - 250000 vs - 7PWei worstcase 4.9495 ms/op 8.7822 ms/op 0.56
altair processAttestation - setStatus - 1/6 committees join 179.54 us/op 266.73 us/op 0.67
altair processAttestation - setStatus - 1/3 committees join 354.69 us/op 540.46 us/op 0.66
altair processAttestation - setStatus - 1/2 committees join 511.81 us/op 726.65 us/op 0.70
altair processAttestation - setStatus - 2/3 committees join 668.77 us/op 944.17 us/op 0.71
altair processAttestation - setStatus - 4/5 committees join 931.60 us/op 1.3446 ms/op 0.69
altair processAttestation - setStatus - 100% committees join 1.1249 ms/op 1.6569 ms/op 0.68
altair processBlock - 250000 vs - 7PWei normalcase 24.626 ms/op 35.204 ms/op 0.70
altair processBlock - 250000 vs - 7PWei normalcase hashState 37.629 ms/op 52.066 ms/op 0.72
altair processBlock - 250000 vs - 7PWei worstcase 73.893 ms/op 124.59 ms/op 0.59
altair processBlock - 250000 vs - 7PWei worstcase hashState 100.77 ms/op 133.49 ms/op 0.75
phase0 processBlock - 250000 vs - 7PWei normalcase 3.1638 ms/op 5.0844 ms/op 0.62
phase0 processBlock - 250000 vs - 7PWei worstcase 53.306 ms/op 61.568 ms/op 0.87
altair processEth1Data - 250000 vs - 7PWei normalcase 645.99 us/op 1.2245 ms/op 0.53
vc - 250000 eb 1 eth1 1 we 0 wn 0 - smpl 15 13.987 us/op 18.545 us/op 0.75
vc - 250000 eb 0.95 eth1 0.1 we 0.05 wn 0 - smpl 219 39.585 us/op 43.729 us/op 0.91
vc - 250000 eb 0.95 eth1 0.3 we 0.05 wn 0 - smpl 42 18.058 us/op 22.737 us/op 0.79
vc - 250000 eb 0.95 eth1 0.7 we 0.05 wn 0 - smpl 18 14.233 us/op 21.744 us/op 0.65
vc - 250000 eb 0.1 eth1 0.1 we 0 wn 0 - smpl 1020 157.25 us/op 148.51 us/op 1.06
vc - 250000 eb 0.03 eth1 0.03 we 0 wn 0 - smpl 11777 1.1011 ms/op 1.2987 ms/op 0.85
vc - 250000 eb 0.01 eth1 0.01 we 0 wn 0 - smpl 141069 11.022 ms/op 17.494 ms/op 0.63
vc - 250000 eb 0 eth1 0 we 0 wn 0 - smpl 250000 20.268 ms/op 29.194 ms/op 0.69
vc - 250000 eb 0 eth1 0 we 0 wn 0 nocache - smpl 250000 74.159 ms/op 97.022 ms/op 0.76
vc - 250000 eb 0 eth1 1 we 0 wn 0 - smpl 250000 37.173 ms/op 57.481 ms/op 0.65
vc - 250000 eb 0 eth1 1 we 0 wn 0 nocache - smpl 250000 119.08 ms/op 154.32 ms/op 0.77
Tree 40 250000 create 664.37 ms/op 1.1907 s/op 0.56
Tree 40 250000 get(125000) 225.73 ns/op 341.93 ns/op 0.66
Tree 40 250000 set(125000) 1.9436 us/op 3.4676 us/op 0.56
Tree 40 250000 toArray() 26.193 ms/op 39.476 ms/op 0.66
Tree 40 250000 iterate all - toArray() + loop 25.092 ms/op 39.519 ms/op 0.63
Tree 40 250000 iterate all - get(i) 108.53 ms/op 144.62 ms/op 0.75
MutableVector 250000 create 11.841 ms/op 18.776 ms/op 0.63
MutableVector 250000 get(125000) 11.179 ns/op 15.032 ns/op 0.74
MutableVector 250000 set(125000) 514.39 ns/op 986.38 ns/op 0.52
MutableVector 250000 toArray() 5.8767 ms/op 8.6819 ms/op 0.68
MutableVector 250000 iterate all - toArray() + loop 5.9064 ms/op 8.8894 ms/op 0.66
MutableVector 250000 iterate all - get(i) 2.6851 ms/op 4.4292 ms/op 0.61
Array 250000 create 5.9041 ms/op 8.3101 ms/op 0.71
Array 250000 clone - spread 3.2174 ms/op 5.0895 ms/op 0.63
Array 250000 get(125000) 1.4720 ns/op 2.1130 ns/op 0.70
Array 250000 set(125000) 1.4570 ns/op 2.1860 ns/op 0.67
Array 250000 iterate all - loop 154.42 us/op 154.97 us/op 1.00
effectiveBalanceIncrements clone Uint8Array 300000 204.17 us/op 139.94 us/op 1.46
effectiveBalanceIncrements clone MutableVector 300000 700.00 ns/op 1.6890 us/op 0.41
effectiveBalanceIncrements rw all Uint8Array 300000 246.50 us/op 315.21 us/op 0.78
effectiveBalanceIncrements rw all MutableVector 300000 138.99 ms/op 334.45 ms/op 0.42
phase0 afterProcessEpoch - 250000 vs - 7PWei 185.82 ms/op 214.19 ms/op 0.87
phase0 beforeProcessEpoch - 250000 vs - 7PWei 57.172 ms/op 124.29 ms/op 0.46
altair processEpoch - mainnet_e81889 542.38 ms/op 733.32 ms/op 0.74
mainnet_e81889 - altair beforeProcessEpoch 107.51 ms/op 194.32 ms/op 0.55
mainnet_e81889 - altair processJustificationAndFinalization 31.729 us/op 70.690 us/op 0.45
mainnet_e81889 - altair processInactivityUpdates 8.7096 ms/op 12.757 ms/op 0.68
mainnet_e81889 - altair processRewardsAndPenalties 78.422 ms/op 116.22 ms/op 0.67
mainnet_e81889 - altair processRegistryUpdates 4.6400 us/op 15.079 us/op 0.31
mainnet_e81889 - altair processSlashings 1.1730 us/op 4.4670 us/op 0.26
mainnet_e81889 - altair processEth1DataReset 1.2460 us/op 4.0610 us/op 0.31
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.1471 ms/op 2.9166 ms/op 0.74
mainnet_e81889 - altair processSlashingsReset 8.0020 us/op 28.167 us/op 0.28
mainnet_e81889 - altair processRandaoMixesReset 8.3020 us/op 23.731 us/op 0.35
mainnet_e81889 - altair processHistoricalRootsUpdate 1.4490 us/op 4.2590 us/op 0.34
mainnet_e81889 - altair processParticipationFlagUpdates 3.8230 us/op 14.933 us/op 0.26
mainnet_e81889 - altair processSyncCommitteeUpdates 1.3270 us/op 3.4140 us/op 0.39
mainnet_e81889 - altair afterProcessEpoch 219.50 ms/op 228.34 ms/op 0.96
phase0 processEpoch - mainnet_e58758 612.67 ms/op 664.75 ms/op 0.92
mainnet_e58758 - phase0 beforeProcessEpoch 229.12 ms/op 320.36 ms/op 0.72
mainnet_e58758 - phase0 processJustificationAndFinalization 31.648 us/op 64.146 us/op 0.49
mainnet_e58758 - phase0 processRewardsAndPenalties 134.18 ms/op 165.39 ms/op 0.81
mainnet_e58758 - phase0 processRegistryUpdates 13.745 us/op 31.615 us/op 0.43
mainnet_e58758 - phase0 processSlashings 1.1130 us/op 3.1860 us/op 0.35
mainnet_e58758 - phase0 processEth1DataReset 667.00 ns/op 3.1390 us/op 0.21
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.9204 ms/op 2.7634 ms/op 0.69
mainnet_e58758 - phase0 processSlashingsReset 7.7070 us/op 17.367 us/op 0.44
mainnet_e58758 - phase0 processRandaoMixesReset 7.0660 us/op 24.055 us/op 0.29
mainnet_e58758 - phase0 processHistoricalRootsUpdate 1.5000 us/op 3.9570 us/op 0.38
mainnet_e58758 - phase0 processParticipationRecordUpdates 7.4050 us/op 22.746 us/op 0.33
mainnet_e58758 - phase0 afterProcessEpoch 162.27 ms/op 193.31 ms/op 0.84
phase0 processEffectiveBalanceUpdates - 250000 normalcase 2.3686 ms/op 2.7531 ms/op 0.86
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.4642 ms/op 3.1613 ms/op 0.78
altair processInactivityUpdates - 250000 normalcase 52.016 ms/op 60.025 ms/op 0.87
altair processInactivityUpdates - 250000 worstcase 51.009 ms/op 75.399 ms/op 0.68
phase0 processRegistryUpdates - 250000 normalcase 12.962 us/op 29.915 us/op 0.43
phase0 processRegistryUpdates - 250000 badcase_full_deposits 605.10 us/op 641.46 us/op 0.94
phase0 processRegistryUpdates - 250000 worstcase 0.5 223.02 ms/op 288.52 ms/op 0.77
altair processRewardsAndPenalties - 250000 normalcase 136.04 ms/op 140.72 ms/op 0.97
altair processRewardsAndPenalties - 250000 worstcase 131.23 ms/op 171.74 ms/op 0.76
phase0 getAttestationDeltas - 250000 normalcase 10.514 ms/op 17.162 ms/op 0.61
phase0 getAttestationDeltas - 250000 worstcase 11.406 ms/op 16.481 ms/op 0.69
phase0 processSlashings - 250000 worstcase 5.7262 ms/op 8.1792 ms/op 0.70
altair processSyncCommitteeUpdates - 250000 276.26 ms/op 395.07 ms/op 0.70
BeaconState.hashTreeRoot - No change 673.00 ns/op 891.00 ns/op 0.76
BeaconState.hashTreeRoot - 1 full validator 72.250 us/op 77.036 us/op 0.94
BeaconState.hashTreeRoot - 32 full validator 709.48 us/op 764.84 us/op 0.93
BeaconState.hashTreeRoot - 512 full validator 8.4278 ms/op 8.6901 ms/op 0.97
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 98.800 us/op 102.60 us/op 0.96
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.2645 ms/op 1.4601 ms/op 0.87
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 16.187 ms/op 18.323 ms/op 0.88
BeaconState.hashTreeRoot - 1 balances 76.323 us/op 73.435 us/op 1.04
BeaconState.hashTreeRoot - 32 balances 700.54 us/op 679.48 us/op 1.03
BeaconState.hashTreeRoot - 512 balances 6.0514 ms/op 6.1113 ms/op 0.99
BeaconState.hashTreeRoot - 250000 balances 105.02 ms/op 126.10 ms/op 0.83
aggregationBits - 2048 els - zipIndexesInBitList 22.392 us/op 41.313 us/op 0.54
regular array get 100000 times 62.617 us/op 61.832 us/op 1.01
wrappedArray get 100000 times 60.903 us/op 61.477 us/op 0.99
arrayWithProxy get 100000 times 34.042 ms/op 38.282 ms/op 0.89
ssz.Root.equals 444.00 ns/op 621.00 ns/op 0.71
byteArrayEquals 434.00 ns/op 652.00 ns/op 0.67
shuffle list - 16384 els 11.334 ms/op 13.367 ms/op 0.85
shuffle list - 250000 els 169.04 ms/op 191.64 ms/op 0.88
processSlot - 1 slots 12.643 us/op 19.227 us/op 0.66
processSlot - 32 slots 1.9585 ms/op 2.4530 ms/op 0.80
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 354.63 us/op 753.20 us/op 0.47
getCommitteeAssignments - req 1 vs - 250000 vc 5.3841 ms/op 6.0708 ms/op 0.89
getCommitteeAssignments - req 100 vs - 250000 vc 7.9025 ms/op 8.7047 ms/op 0.91
getCommitteeAssignments - req 1000 vs - 250000 vc 8.4244 ms/op 9.5983 ms/op 0.88
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 7.9800 ns/op 11.730 ns/op 0.68
state getBlockRootAtSlot - 250000 vs - 7PWei 958.65 ns/op 1.5876 us/op 0.60
computeProposers - vc 250000 16.449 ms/op 23.534 ms/op 0.70
computeEpochShuffling - vc 250000 172.04 ms/op 197.63 ms/op 0.87
getNextSyncCommittee - vc 250000 274.40 ms/op 390.85 ms/op 0.70

by benchmarkbot/action

@dapplion dapplion changed the title Update produce block flow for EIP-4844 EIP-4844: Update produce block flow Dec 6, 2022
@dapplion dapplion added the spec-deneb 🐡 Issues targeting the Deneb spec version label Dec 6, 2022
// Capella and later forks have withdrawals on their ExecutionPayload
if (fork === ForkName.capella || fork === ForkName.eip4844) {
// TODO EIP-4844 Remove this when the EC includes `withdrawals`
(blockBody as capella.BeaconBlockBody).executionPayload.withdrawals = [];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will break the withdrawals, so we should do this only in an unmerged PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing from here, so that we can only keep this in the unmerged eip-4844 parent PR

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, we should merge withdrawal logic here ASAP. Is it there already on unstable or no-where yet?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merge withdrawal logic is already in the executin/http where withdrawals are bundeled on execution payload depending on fork

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we set withdrawal to attributes here: https://github.com/ChainSafe/lodestar/pull/4864/files#diff-c331f7b8e7121839c55f5f20f21d1efe1375b69c6fbbe5db12a4baf1ed4acf5dR350, and rest execution apis handle it appropriately

@g11tech g11tech force-pushed the dapplion/eip-4844-blockproduction branch from 86b4b94 to 95739bd Compare December 6, 2022 12:26
*/
getBlobsSidecar(beaconBlock: eip4844.BeaconBlock): eip4844.BlobsSidecar {
const blockHash = toHex(beaconBlock.body.executionPayload.blockHash);
const blobsSidecar = this.producedBlobsSidecarCache.get(blockHash);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems here we get from beacon block hash while we cache by payload block hash

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

const blockHash = toHex(beaconBlock.body.executionPayload.blockHash); is payload blockHash

const blobsBundle = await this.executionEngine.getBlobsBundle(payloadId);

// Sanity check consistency between getPayload() and getBlobsBundle()
const blockHash = toHex(payload.blockHash);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here we return payload block hash but seems later on we get by beacon block hash

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Get by execution payload blockHash

const blockHash = toHex(beaconBlock.body.executionPayload.blockHash);
const blobsSidecar = this.producedBlobsSidecarCache.get(blockHash);

Set by execution payload blockHash

this.producedBlobsSidecarCache.set(blobs.blockHash, {

Which is computed here from the payload

const blockHash = toHex(payload.blockHash);

Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

@g11tech g11tech merged commit a27f675 into unstable Dec 6, 2022
@g11tech g11tech deleted the dapplion/eip-4844-blockproduction branch December 6, 2022 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spec-deneb 🐡 Issues targeting the Deneb spec version

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants