feat: use positions based on a forest with 63 rows#104
feat: use positions based on a forest with 63 rows#104Davidson-Souza merged 1 commit intomit-dci:mainfrom
Conversation
Cargo.toml
Outdated
| # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html | ||
| [dependencies] | ||
| bitcoin_hashes = { version = "0.19", default-features = false } | ||
| bitcoin_hashes = { version = "0.20.0", default-features = false } |
There was a problem hiding this comment.
| bitcoin_hashes = { version = "0.20.0", default-features = false } | |
| bitcoin_hashes = { version = "0.20", default-features = false } |
src/lib.rs
Outdated
| /// dividing this by 90,000 we get 102,481,911,520,608 blocks | ||
| /// it would take 3,249,680 years to mine that many blocks... | ||
| /// | ||
| /// For the poor soul in 3,249,682 who need to fix this hard-fork, here's what you gotta do: |
There was a problem hiding this comment.
lmao what is this documentation
f9d7b84 to
854f4f3
Compare
|
rebased after #102 |
You mean, 2^63. Increasing the number would only require a local remap and serialization change? All historic proofs and roots stay the same? |
I meant to say rows, not leaves. 63 rows = 2ˆ63 leaves
You don't need to change it on-disk, but you do need to translate before sending it to someone. |
src/lib.rs
Outdated
| /// dividing this by 90,000 we get 102,481,911,520,608 blocks | ||
| /// it would take 3,249,680 years to mine that many blocks... | ||
| /// | ||
| /// For the poor soul in 3,249,682 A.D, who need to fix this hard-fork, here's what you gotta do: |
There was a problem hiding this comment.
You meant adding the 2,000 years that have already passed, but this is just a pessimistic estimate anyway
| /// For the poor soul in 3,249,682 A.D, who need to fix this hard-fork, here's what you gotta do: | |
| /// For the poor soul in 3,251,680 A.D., who need to fix this hard-fork, here's what you gotta do: |
There was a problem hiding this comment.
I think the punch gets even funnier if I add the two there (this is obviously an Easter egg and not meant to be taken seriously).
There was a problem hiding this comment.
The guy reading this in 3,251,680 will actually take this seriously
There was a problem hiding this comment.
3,251,682*, in 3,251,680 they will be fighting whether the bug is real
src/util/mod.rs
Outdated
| /// Translates targets from a forest with `from_rows` to a forest with `to_rows`. | ||
| /// | ||
| /// When we compute the position of a node, for any node not in the 0th row, their position depends | ||
| /// on how many leaves there are. This happens because the 0th row's size is allocated to the | ||
| /// nearest power of two that can fit that many leaves. Therefore, in a forest with 6 leaves, the | ||
| /// bottom row goes from zero through 7, the row 1 from 8 through 11 (the size of each row | ||
| /// halves as you move up). If you add three extra UTXOs, growing the forest to nine leaves, adding | ||
| /// the 9th will require allocating 16 0-row leaves, row 1 therefore goes from 16 to 23 and so on. | ||
| /// | ||
| /// If leaves always stay at the bottom, that fine. Nothing at the bottom ever needs to care about | ||
| /// this, because there's no row before it to grow and shift their positions. However, leaves | ||
| /// **do** move up during deletions. For that reason, whenever the forest grow, all targets that | ||
| /// aren't at the bottom needs to be updated. | ||
| /// | ||
| /// Now imagine that we want to keep a leaf map that maps leaf_hash -> position within the forest: | ||
| /// this works fine, we know where a node must go when deleting, by calling [`parent`] with their | ||
| /// current position and `num_leaves`. But now imagine the forest has to grow: we need to go through | ||
| /// the map and update all non-row 0 leaves. This could potentially involve going through millions | ||
| /// of UTXOs and update one-by-one. Note that we can find the next position, it's not super | ||
| /// efficient but works (see [`crate::proof::Proof::maybe_remap`] for more details), but doing this | ||
| /// for every UTXO that isn't at the bottom is too expensive, even though it happens exponentially | ||
| /// less frequently, when it happens, it's going to take an absurd amount of time and potentially | ||
| /// stall the Utreexo network for hours. | ||
| /// | ||
| /// For that reason, we communicate positions as if the forest is always filled with the maximum | ||
| /// amount of leaves we can possibly have, which is 63. Therefore, those positions never need to be | ||
| /// remapped. Internally, we still use the dynamic size, and use this function to translate between | ||
| /// the two. | ||
| /// | ||
| /// # Implementation | ||
| /// | ||
| /// This function simply computes how far away from the start of the row this leaf is, then use it | ||
| /// to offset the same amount in the new structure. |
There was a problem hiding this comment.
Nit: small text style modification
| /// Translates targets from a forest with `from_rows` to a forest with `to_rows`. | |
| /// | |
| /// When we compute the position of a node, for any node not in the 0th row, their position depends | |
| /// on how many leaves there are. This happens because the 0th row's size is allocated to the | |
| /// nearest power of two that can fit that many leaves. Therefore, in a forest with 6 leaves, the | |
| /// bottom row goes from zero through 7, the row 1 from 8 through 11 (the size of each row | |
| /// halves as you move up). If you add three extra UTXOs, growing the forest to nine leaves, adding | |
| /// the 9th will require allocating 16 0-row leaves, row 1 therefore goes from 16 to 23 and so on. | |
| /// | |
| /// If leaves always stay at the bottom, that fine. Nothing at the bottom ever needs to care about | |
| /// this, because there's no row before it to grow and shift their positions. However, leaves | |
| /// **do** move up during deletions. For that reason, whenever the forest grow, all targets that | |
| /// aren't at the bottom needs to be updated. | |
| /// | |
| /// Now imagine that we want to keep a leaf map that maps leaf_hash -> position within the forest: | |
| /// this works fine, we know where a node must go when deleting, by calling [`parent`] with their | |
| /// current position and `num_leaves`. But now imagine the forest has to grow: we need to go through | |
| /// the map and update all non-row 0 leaves. This could potentially involve going through millions | |
| /// of UTXOs and update one-by-one. Note that we can find the next position, it's not super | |
| /// efficient but works (see [`crate::proof::Proof::maybe_remap`] for more details), but doing this | |
| /// for every UTXO that isn't at the bottom is too expensive, even though it happens exponentially | |
| /// less frequently, when it happens, it's going to take an absurd amount of time and potentially | |
| /// stall the Utreexo network for hours. | |
| /// | |
| /// For that reason, we communicate positions as if the forest is always filled with the maximum | |
| /// amount of leaves we can possibly have, which is 63. Therefore, those positions never need to be | |
| /// remapped. Internally, we still use the dynamic size, and use this function to translate between | |
| /// the two. | |
| /// | |
| /// # Implementation | |
| /// | |
| /// This function simply computes how far away from the start of the row this leaf is, then use it | |
| /// to offset the same amount in the new structure. | |
| /// Translates targets from a forest with `from_rows` to a forest with `to_rows`. | |
| /// | |
| /// When we compute the position of a node, any node not in row 0 has a position that depends | |
| /// on how many leaves there are. This happens because row 0 is allocated to the nearest power of | |
| /// two that can fit that many leaves. Therefore, in a forest with 6 leaves, the bottom row goes | |
| /// from 0 through 7, and row 1 goes from 8 through 11 (the size of each row halves as you move | |
| /// up). If you add three extra UTXOs, growing the forest to 9 leaves, adding the 9th will require | |
| /// allocating 16 row-0 leaves; row 1 therefore goes from 16 through 23, and so on. | |
| /// | |
| /// If leaves always stayed at the bottom, that's fine. Nothing at the bottom ever needs to care | |
| /// about this, because there is no row below it whose growth would shift its position. However, | |
| /// leaves **do** move up during deletions. For that reason, whenever the forest grows, all targets | |
| /// that are not at the bottom need to be updated. | |
| /// | |
| /// Now imagine that we want to keep a leaf map from `leaf_hash` to position within the forest: | |
| /// this works fine, and we know where a node must go when deleting by calling [`parent`] with its | |
| /// current position and `num_leaves`. But now imagine the forest has to grow: we need to go | |
| /// through the map and update all non-row-0 leaves. This could potentially involve going through | |
| /// millions of UTXOs and updating them one by one. Note that we can find the next position; it is | |
| /// not super efficient, but it works (see [`crate::proof::Proof::maybe_remap`] for more details). | |
| /// But doing this for every UTXO that is not at the bottom is too expensive. Even though it | |
| /// happens exponentially less frequently, when it does happen, it is going to take an absurd | |
| /// amount of time and could potentially stall the Utreexo network for hours. | |
| /// | |
| /// For that reason, we communicate positions as if the forest was always filled with the maximum | |
| /// number of leaves we can possibly have, which is 63. Therefore, those positions never need to be | |
| /// remapped. Internally, we still use the dynamic size, and use this function to translate between | |
| /// the two. | |
| /// | |
| /// # Implementation | |
| /// | |
| /// This function simply computes how far from the start of the row this leaf is, then uses that | |
| /// offset in the new structure. |
src/lib.rs
Outdated
| /// | ||
| /// # Calculations | ||
| /// | ||
| /// If you think: "but... is 63 enough space"? Well... Assuming there's around 999,000 WU |
There was a problem hiding this comment.
| /// If you think: "but... is 63 enough space"? Well... Assuming there's around 999,000 WU | |
| /// If you think: "but... is 63 enough space"? Well... assuming there's around 999,000 WUs |
src/lib.rs
Outdated
| /// If you think: "but... is 63 enough space"? Well... Assuming there's around 999,000 WU | ||
| /// available on each block (let's account for header and coinbase), a non-segwit transaction's | ||
| /// size is: | ||
| /// 4 (version) + 1 (vin count) + 41 (input) + 5 (vout for a large number of inputs) + 10N + 4 |
There was a problem hiding this comment.
You mean a large number of outputs?
src/lib.rs
Outdated
| /// - Change the leaf_data type to a u128 or something q128 if Quantum Bits are the fashionable | ||
| /// standard |
There was a problem hiding this comment.
| /// - Change the leaf_data type to a u128 or something q128 if Quantum Bits are the fashionable | |
| /// standard | |
| /// - Change the `leaf_data` type to u128, or q128 if Quantum Bits are the fashionable standard |
There was a problem hiding this comment.
Noo you just made this one liner without the changeee @Davidson-Souza (anyway you need to re-push to fix MSRV)
854f4f3 to
95384ad
Compare
|
Pushed 95384ad applying docs improvements suggested by @JoseSK999 |
src/util/mod.rs
Outdated
|
|
||
| /// Translates targets from a forest with `from_rows` to a forest with `to_rows`. | ||
| /// | ||
| /// When we compute the position of a node, for any node not in row 0 has a position that depends |
There was a problem hiding this comment.
Missing from my prev comment
| /// When we compute the position of a node, for any node not in row 0 has a position that depends | |
| /// When we compute the position of a node, any node not in row 0 has a position that depends |
src/util/mod.rs
Outdated
| /// from 0 through 7, and row 1 goes from 8 through 11 (the size of each row halves as you | ||
| /// move up). If you add three extra UTXOs, growing the forest to 9 leaves, adding the 9th | ||
| /// will require allocating 16 row-0 leaves; row 1 therefore goes from 16 though 23, and so on. |
There was a problem hiding this comment.
Also missing
| /// from 0 through 7, and row 1 goes from 8 through 11 (the size of each row halves as you | |
| /// move up). If you add three extra UTXOs, growing the forest to 9 leaves, adding the 9th | |
| /// will require allocating 16 row-0 leaves; row 1 therefore goes from 16 though 23, and so on. | |
| /// from 0 through 7, and row 1 goes from 8 through 11 (the size of each row halves as you move | |
| /// up). If you add three extra UTXOs, growing the forest to 9 leaves, adding the 9th will require | |
| /// allocating 16 row-0 leaves; row 1 therefore goes from 16 through 23, and so on. |
src/util/mod.rs
Outdated
| /// If leaves always stayed at the bottom, that fine. Nothing at the bottom ever needs to care about | ||
| /// this, because there is no row below it whose growth would shift its positions. However, leaves | ||
| /// **do** move up during deletions. For that reason, whenever the forest grows, all targets that | ||
| /// are not at the bottom needs to be updated. |
There was a problem hiding this comment.
| /// If leaves always stayed at the bottom, that fine. Nothing at the bottom ever needs to care about | |
| /// this, because there is no row below it whose growth would shift its positions. However, leaves | |
| /// **do** move up during deletions. For that reason, whenever the forest grows, all targets that | |
| /// are not at the bottom needs to be updated. | |
| /// If leaves always stayed at the bottom, that's fine. Nothing at the bottom ever needs to care | |
| /// about this, because there is no row below it whose growth would shift its position. However, | |
| /// leaves **do** move up during deletions. For that reason, whenever the forest grows, all targets | |
| /// that are not at the bottom need to be updated. |
src/util/mod.rs
Outdated
| /// Now, imagine that we want to keep a leaf map from `leaf_hash` to position within the forest: | ||
| /// this works fine, and we know where a node must go when deleting, by calling [`parent`] with their | ||
| /// current position and `num_leaves`. But now imagine the forest has to grow: we need to go through | ||
| /// the map and update all non-row-0 leaves. This could potentially involve going through millions | ||
| /// of UTXOs and update them one by one. Note that we can find the next position, it is not super | ||
| /// efficient, but it works (see [`crate::proof::Proof::maybe_remap`] for more details). But doing this | ||
| /// for every UTXO that are not at the bottom is too expensive. Even though it happens exponentially | ||
| /// less frequently, when it happens, it is going to take an absurd amount of time and potentially | ||
| /// stall the Utreexo network for hours. |
There was a problem hiding this comment.
| /// Now, imagine that we want to keep a leaf map from `leaf_hash` to position within the forest: | |
| /// this works fine, and we know where a node must go when deleting, by calling [`parent`] with their | |
| /// current position and `num_leaves`. But now imagine the forest has to grow: we need to go through | |
| /// the map and update all non-row-0 leaves. This could potentially involve going through millions | |
| /// of UTXOs and update them one by one. Note that we can find the next position, it is not super | |
| /// efficient, but it works (see [`crate::proof::Proof::maybe_remap`] for more details). But doing this | |
| /// for every UTXO that are not at the bottom is too expensive. Even though it happens exponentially | |
| /// less frequently, when it happens, it is going to take an absurd amount of time and potentially | |
| /// stall the Utreexo network for hours. | |
| /// Now imagine that we want to keep a leaf map from `leaf_hash` to position within the forest: | |
| /// this works fine, and we know where a node must go when deleting by calling [`parent`] with its | |
| /// current position and `num_leaves`. But now imagine the forest has to grow: we need to go | |
| /// through the map and update all non-row-0 leaves. This could potentially involve going through | |
| /// millions of UTXOs and updating them one by one. Note that we can find the next position; it is | |
| /// not super efficient, but it works (see [`crate::proof::Proof::maybe_remap`] for more details). | |
| /// But doing this for every UTXO that is not at the bottom is too expensive. Even though it | |
| /// happens exponentially less frequently, when it does happen, it is going to take an absurd | |
| /// amount of time and could potentially stall the Utreexo network for hours. |
src/lib.rs
Outdated
| /// 4 (version) + 1 (vin count) + 41 (input) + 5 (vout for a large number of outputs) + 10N + 4 | ||
| /// (locktime) | ||
| /// | ||
| /// N is how many outputs we have (we are considering outputs with amount and a zero-sized | ||
| /// script), for 999,000 WU we can fit | ||
| /// 55 + 10N <= 999,000 | ||
| /// N ~= 90k outputs (a little over) | ||
| /// | ||
| /// 2^63 = 9,223,372,036,854,775,808 | ||
| /// dividing this by 90,000 we get 102,481,911,520,608 blocks | ||
| /// it would take 3,249,680 years to mine that many blocks... | ||
| /// | ||
| /// For the poor soul in 3,249,682 A.D., who need to fix this hard-fork, here's what you gotta do: | ||
| /// - Change the leaf_data type to a u128 or something q128 if Quantum Bits are the fashionable standard | ||
| /// - Change `MAX_FOREST_ROWS` to 128 or higher in `lib.rs` | ||
| /// - Modify [`start_position_at_row`] to avoid overflows. |
There was a problem hiding this comment.
| /// 4 (version) + 1 (vin count) + 41 (input) + 5 (vout for a large number of outputs) + 10N + 4 | |
| /// (locktime) | |
| /// | |
| /// N is how many outputs we have (we are considering outputs with amount and a zero-sized | |
| /// script), for 999,000 WU we can fit | |
| /// 55 + 10N <= 999,000 | |
| /// N ~= 90k outputs (a little over) | |
| /// | |
| /// 2^63 = 9,223,372,036,854,775,808 | |
| /// dividing this by 90,000 we get 102,481,911,520,608 blocks | |
| /// it would take 3,249,680 years to mine that many blocks... | |
| /// | |
| /// For the poor soul in 3,249,682 A.D., who need to fix this hard-fork, here's what you gotta do: | |
| /// - Change the leaf_data type to a u128 or something q128 if Quantum Bits are the fashionable standard | |
| /// - Change `MAX_FOREST_ROWS` to 128 or higher in `lib.rs` | |
| /// - Modify [`start_position_at_row`] to avoid overflows. | |
| /// `4 (version) + 1 (vin count) + 41 (input) + 5 (vout for many outputs) + 10N + 4 (locktime)` | |
| /// | |
| /// `N` is how many outputs we have (we are considering outputs with amount and a zero-sized | |
| /// script), for 999,000 WUs we can fit: | |
| /// - `55 + 10N <= 999,000` | |
| /// - `N ~= 90k` outputs (a little over) | |
| /// | |
| /// Since `2^63 = 9,223,372,036,854,775,808`, if you divide this by 90,000 we get | |
| /// 102,481,911,520,608 blocks. It would take us 3,249,680 years to mine that many blocks. | |
| /// | |
| /// For the poor soul in 3,249,682 A.D., who needs to fix this hard-fork, here's what you gotta do: | |
| /// - Change the `leaf_data` type to u128, or q128 if Quantum Bits are the fashionable standard. | |
| /// - Change `MAX_FOREST_ROWS` to 128 or higher in `lib.rs` | |
| /// - Modify [`start_position_at_row`] to avoid overflows. |
95384ad to
1ad58f1
Compare
|
Should add a typo finder on CI |
|
Fixed the MSRV error and applied docs changes suggestions by @JoseSK999 |
src/lib.rs
Outdated
| /// - Change `MAX_FOREST_ROWS` to 128 or higher in `lib.rs` | ||
| /// - Modify [`start_position_at_row`] to avoid overflows. |
There was a problem hiding this comment.
Optional nit, broken link
| /// - Change `MAX_FOREST_ROWS` to 128 or higher in `lib.rs` | |
| /// - Modify [`start_position_at_row`] to avoid overflows. | |
| /// - Change `MAX_FOREST_ROWS` to 128 or higher in `lib.rs`. | |
| /// - Modify [`util::start_position_at_row`] to avoid overflows. |
src/proof/mod.rs
Outdated
| .targets | ||
| .iter() | ||
| .copied() | ||
| .map(|pos| translate(pos, 63, total_rows)) |
There was a problem hiding this comment.
Why don't u use the constant here
src/proof/mod.rs
Outdated
| .targets | ||
| .iter() | ||
| .copied() | ||
| .map(|pos| translate(pos, 63, total_rows)) |
Hmm, |
|
That was my mistake (I made it public), that's why I removed the comment 😂 only the review comments are relevant |
df10b0b to
2774072
Compare
|
@JoseSK999 used the constants where applicable. |
|
I'll review this one later today. |
2774072 to
005ed44
Compare
|
Rebased, since 2774072 is already on main |
|
|
||
| #[test] | ||
| fn test_start_position_at_row() { | ||
| assert_eq!(start_position_at_row(1, 12), 4096); |
There was a problem hiding this comment.
Last nit: I feel like we could add a few more test cases here, with different number of rows and row positions.
When we compute the position of a node, for any node not in the 0th row, their position depends on how many leaves there are. This happens because the 0th row's size is allocated to the nearest power of two that can fit that many leaves. Therefore, in a forest with 6 leaves, the bottom row goes from zero through 7, the row 1 from 8 through 11 (the size of each row halves as you move up). If you add three extra UTXOs, growing the forest to nine leaves, adding the 9th will require allocating 16 0-row leaves, row 1 therefore goes from 16 to 23 and so on. If leaves always stay at the bottom, that fine. Nothing at the bottom ever needs to care about this, because there's no row before it to grow and shift their positions. However, leaves **do** move up during deletions. For that reason, whenever the forest grow, all targets that aren't at the bottom needs to be updated. Now imagine that we want to keep a leaf map that maps leaf_hash -> position within the forest: this works fine, we know where a node must go when deleting, by calling [`parent`] with their current position and `num_leaves`. But now imagine the forest has to grow: we need to go through the map and update all non-row 0 leaves. This could potentially involve going through millions of UTXOs and update one-by-one. Note that we can find the next position, it's not super efficient but works (see [`crate::proof::Proof::maybe_remap`] for more details), but doing this for every UTXO that isn't at the bottom is too expensive, even though it happens exponentially less frequently, when it happens, it's going to take an absurd amount of time and potentially stall the Utreexo network for hours. For that reason, we communicate positions as if the forest is always filled with the maximum amount of leaves we can possibly have, which is 63. Therefore, those positions never need to be remapped. Internally, we still use the dynamic size, and use this function to translate between the two.
005ed44 to
627664a
Compare
|
@luisschwab ur turn |
3c3a939 chore: bump rustreexo (Davidson Souza) Pull request description: ### Description and Notes ~~This PR updates rustreexo to `main` + mit-dci/rustreexo#104 ~~I'll leave this as draft until we ship the next version, but this should help testing mit-dci/rustreexo#104 This PR bumps rustreexo v0.4.0 -> v0.5.0 ACKs for top commit: moisesPompilio: ACK 3c3a939 luisschwab: tACK 3c3a939 JoseSK999: ACK 3c3a939; I had previously synced signet in this branch and currently it is still working for me (given I connect a good bridge) Tree-SHA512: eb941c27b7d9de0e4a47e9f65912904d7cb6df3fc93fd114a412424140d94820eda58e1e2b71bb6ecc85a86612981589dc1269ca32743ac8fc1f4bf9247d3a74

When we compute the position of a node, for any node not in the 0th row, their position depends on how many leaves there are. This happens because the 0th row's size is allocated to the nearest power of two that can fit that many leaves. Therefore, in a forest with 6 leaves, the bottom row goes from zero through 7, the row 1 from 8 through 11 (the size of each row halves as you move up). If you add three extra UTXOs, growing the forest to nine leaves, adding the 9th will require allocating 16 0-row leaves, row 1 therefore goes from 16 to 23 and so on.
If leaves always stay at the bottom, that fine. Nothing at the bottom ever needs to care about this, because there's no row before it to grow and shift
their positions. However, leaves do move up during deletions. For that
reason, whenever the forest grow, all targets that aren't at the bottom needs to be updated.
Now imagine that we want to keep a leaf map that maps leaf_hash -> position within the forest: this works fine, we know where a node must go when deleting, by calling [
parent] with their current position andnum_leaves. But now imagine the forest has to grow: we need to go through the map and update all non-row 0 leaves. This could potentially involve going through millions of UTXOs and update one-by-one. Note that we can find the next position, it's not super efficient but works (see [crate::proof::Proof::maybe_remap] for more details), but doing this for every UTXO that isn't at the bottom is too expensive, even though it happens exponentially less frequently, when it happens, it's going to take an absurd amount of time and potentially stall the Utreexo network for hours.For that reason, we communicate positions as if the forest is always filled with the maximum amount of rows we can possibly have, which is 63. Therefore, those positions never need to be remapped. Internally, we still use the dynamic size, and use this function to translate between the two.