add transparent data to compact block serialization by LarryRuane · Pull Request #534 · zcash/lightwalletd

LarryRuane · 2025-10-24T05:54:56Z

This PR includes changing the proto files directly, which is incorrect and is actually being done by zcash/lightwallet-protocol#1, so these changes will have to be removed before merging. There is also probably some documentation that should be added / changed here. But it's in a good enough state to be reviewed and tested now.

This PR does not provide the value of each transparent input, as suggested as a possible enhancement in the PR referenced above. It will probably increase the cache build time too much. But I think I'll implement as (potentially) part of this PR, and see if that's the case. It is pretty easy to code.

LarryRuane · 2025-10-24T15:04:18Z

On mainnet, to recreate the entire compact block cache (resync) took about 8.5 hours on my mid-level desktop, using zebrad as the backend. The cache size is 33 GB, about double the pre-this-PR size. This is expected for two reasons: first, we're now storing some parts of every transparent transaction, instead of (parts of) only shielded transactions. Secondly, we're storing blocks between height 0 and 419200 (mainnet sapling activation height).

UPDATE: I should also mention that the way this is implemented, the lightwalletd memory usage has not increased (and it is very light, there are no data structures that scale with the number of blocks, transactions, etc.). Only the on-disk storage is larger.

pacu · 2025-10-24T16:05:01Z

On mainnet, to recreate the entire compact block cache (resync) took about 8.5 hours on my mid-level desktop, using zebrad as the backend. The cache size is 33 GB, about double the pre-this-PR size. This is expected for two reasons: first, we're now storing some parts of every transparent transaction, instead of (parts of) only shielded transactions. Secondly, we're storing blocks between height 0 and 419200 (mainnet sapling activation height).

This close to what the statistical analysis indicated, but we didn't run it since genesis because we didn't consider that use case of people importing a pre-sapling wallet into a shielded light-client

LarryRuane · 2025-10-24T16:53:58Z

... but we didn't run it since genesis because we didn't consider that use case of people importing a pre-sapling wallet into a shielded light-client

I asked @nuttycom and he said the cache should go back to genesis. It's not much more space (I didn't calculate how much), so I think it's okay to leave it even if it might not be needed.

pacu

utACK modulo documentation suggestions

pacu · 2025-10-27T21:18:27Z

lightwallet-protocol/walletrpc/compact_formats.proto

    repeated CompactSaplingSpend spends = 4;
    repeated CompactSaplingOutput outputs = 5;
    repeated CompactOrchardAction actions = 6;
+    // CompactTxIn values  corresponding to the `vin` entries of the full transaction.


Suggested change

// CompactTxIn values corresponding to the `vin` entries of the full transaction.

// CompactTxIn values corresponding to the `vin` entries of the full transaction.

lightwallet-protocol/walletrpc/compact_formats.proto

CHANGELOG.md

nuttycom

Reviewed 427ccbd

Overall this looks good, but I think there are problems with a number of the transaction version checks in parsing in that I don't believe that they function correctly for V5 transactions as currently written.

nuttycom · 2025-10-24T13:43:12Z

CHANGELOG.md


 ### Added

+- Add block header to GetBlock and GetBlockRange output (issue 527)


This should mention the .proto update (once it has been brought in via subtree merge and once we've sorted out the tagging on the zcash/lightwallet-protocol repo.)

common/cache.go

common/common.go

parser/transaction.go

smoke-test.bash

nuttycom · 2025-10-27T23:06:00Z

testdata/compact_blocks.json

How is this test data generated? It would be useful to document this (though maybe it's documented elsewhere, since .json files can't contain comments?).

How is this test data generated?

This data dates back to the earliest versions of the project, when George Tankersley maintained it, so I'm unsure. The more recent changes that I've made were sort of cheating; we don't have an independent way to generate compact blocks and transactions, so I just let the code tell me what it should be. This does at least catch regressions — if the tests that use this data start failing when we don't expect them to, then we know something has inadvertently broken in either the deserialization of the full block, conversion to its compact format, or in the serialization of the compact block.

LarryRuane

I left comments on most of your comments, I'll push changes, thanks!

parser/transaction.go

LarryRuane · 2025-10-28T18:11:22Z

parser/transaction.go


-	if !s.Skip(192) {
-		return nil, errors.New("could not skip Groth16 proof")
+	if isSaplingV4 {


Yes, I think you're correct. There must not be any joinSplits after V5 activated on mainnet, because the entire chain parses correctly. But this must be fixed (for the future). One test we definitely should do (and I will) is verify that lightwalletd parses testnet correctly; there may be transaction formats there that don't exist on mainnet.

We may still want to introduce the enum, but for now, I'll add a boolean that indicates if the joinsplit proof is Groth16 (Sapling or later) or not.

parser/transaction.go

common/common.go

frontend/service.go

hash32/hash32.go

LarryRuane · 2025-10-28T19:09:39Z

parser/transaction.go

+	if !s.ReadBytes(&b32, 32) {
+		return nil, errors.New("could not read HashPrevBlock")
 	}
+	tx.Outpoint.PrevTxHash = hash32.T(b32)


I think it should be FromSlice, will fix it.

common/common.go

LarryRuane · 2025-11-07T18:41:11Z

Force pushed to

address review comments
distribute changes correctly across commits
get all tests to pass on each commit
make sure code is formatted on each commit.

I think this is ready to merge, except I'm still unclear on the proto files with respect to the separate repo (git subtree), so this PR probably isn't mergeable exactly as is.

Working on these changes also led to creating zcash/lightwallet-protocol#5 (documentation only change).

nuttycom · 2025-11-07T19:00:31Z

@LarryRuane the way to handle updating the protobuf files is just to do the following:

git subtree -P lightwallet-protocol/ pull git@github.com:zcash/lightwallet-protocol.git main --squash

I would create a temporary branch from main where you go through this process, then rebase this PR on that branch. zcash/lightwallet-protocol#5 has now been merged, so using main will work.

LarryRuane · 2025-11-10T22:46:30Z

the way to handle updating the protobuf files is just to do the following [...]

Thanks, did that, force pushed, please review. Ready to merge, AFAIK.

pacu

reviewed 0bbda6026ce99f3f3b032257b6469bc8b21af2cb

+1 to @nuttycom's comments plus a comment on empty struct

parser/transaction.go

lightwallet-protocol/walletrpc/compact_formats.proto

LarryRuane · 2025-11-13T17:30:10Z

To check for regressions, I ran the following command against this PR and again against the master branch:

grpcurl -plaintext -d '{"start":{"height":419200}, "end":{"height":3133574}}' localhost:9067 cash.z.wallet.sdk.rpc.CompactTxStreamer/GetBlockRange

This fetches all blocks between Sapling activation and (near) the current chain tip. The results are identical. This shows that the following actions are correct for each block using the PR branch (or at least that the PR doesn't introduce any new bugs):

deserializing (parsing) the raw (binary, actually hex) block from zebrad
converting the in-memory block to compact format
serializing the compact format and writing it to the disk cache
later, reading and deserializing (parsing) the compact block from disk cache
expressing the compact block as JSON and writing to stdout

This doesn't verify transparent transaction elements, because the point of the test is to compare the PR with master, which doesn't support transparent elements.

This was with the local zebrad running mainnet. (The starting height is Sapling activation, which is the lowest height allowed until this PR.) This took about 19 minutes on a mid-range desktop (for anyone interested).

I did the same test on testnet (since there may be some unusual transaction formats that aren't on mainnet), and the results were also identical.

LarryRuane · 2026-01-08T16:07:33Z

Rebased, and also fix commit "add header to GetBlock result (issue 527) if requested" by adding the "requested" part: don't unconditionally include the full block header in the compact blocks returned by GetBlock (and similar), but only if the BlockID.fullHeader flag is set in the request. See zcash/lightwallet-protocol#15

AFAIK, this is ready to be merged (no other changes planned), just needs review.

LarryRuane · 2026-01-08T16:44:36Z

me:

AFAIK, this is ready to be merged (no other changes planned), just needs review.

Correction -- there are still a few things left to address, don't merge yet.

LarryRuane · 2026-01-10T02:07:09Z

Rebased onto the latest lightwallet-protocol zcash/lightwallet-protocol/pull/16 to remove BlockID.fullHeader from the protobuf definition (and corresponding code). Let's not merge just yet, want to check over a few more things, but this is ready for re-review.

nuttycom · 2026-01-11T19:14:57Z

Rebased onto the latest lightwallet-protocol zcash/lightwallet-protocol/pull/16 to remove BlockID.fullHeader from the protobuf definition (and corresponding code). Let's not merge just yet, want to check over a few more things, but this is ready for re-review.

Shouldn't this be just based on the released 0.4.0 tag? We don't want to end up releasing a lightwalletd that's based on a version of the lightwallet-protocol API that is not tagged as a release, and I don't think that the delta between 0.4.0 and the unreleased version that this is based on is meaningful.

The main work is adding the ability to parse pre-V4 (Sapling) transactions. I used zcash: src/primitives/transaction.h to see how the serialization is done.

LarryRuane · 2026-01-12T21:07:05Z

Shouldn't this be just based on the released 0.4.0 tag?

That's a good point. I just force pushed to remove the subtree commit, since lightwallet-protocol version 0.4.0 is already in lightwalletd's master branch. So this PR just got simpler.

LarryRuane · 2026-01-13T08:13:48Z

Another force push (d2de8f5) to address all remaining review comments; I think this is finally ready to merge (after re-review).

str4d

Reviewed d2de8f5

CHANGELOG.md

parser/transaction.go

str4d · 2026-01-26T21:19:46Z

frontend/service.go

+		return status.Errorf(codes.InvalidArgument, "invalid pool type requested")
+	}
+	if len(exclude.PoolTypes) == 0 {
+		// legacy behavior: return only blocks containing sheilded components.


This was marked as resolved but has not been resolved.

frontend/service.go

This is a space and network optimization; the coinbase input doesn't contain relevant information for us (our compact block inputs are just an outpoint, which is a txid and index, and for the coinbase, these aren't meaningful).

No functional difference, just clarity. Change calls such as: hash32.T(someslice) to: hash32.FromSlice(someslice)

The first time a release with this PR runs, it will find a compact block disk cache that doesn't begin at height zero, doesn't include transparent transaction components, and doesn't contain the full block header. The way this will be fixed is to re-create the cache from scratch. To detect that this needs to be done, we attempt to deserialize (only) the first block. If that fails, rebuild the cache. As part of this, simplify the recovery when disk cache corruption is detected. Instead of backing up 10,000 blocks and trying again (which caused an unintentional function call sequence recursion), rebuild the entire cache. If there is any corruption detected, it's best to rebuild the entire cache, rather than trying to optimize and rebuild only part of it.

This uses the recently-added BlockRange.poolTypes argument; previously, GetBlockRange ignored this argument.

The PoolTypes function argument is already in service.proto. So the change in behavior here is that instead of GetMempoolTx returning all transactions (and all parts of those transactions), ignoring the exclude.PoolType argument; now it will (1) return only the requested parts of transactions (rather than the entire transaction), and (2) only return transactions that have at least one part after pool filtering. This is the same behavior as GetBlockRange.

LarryRuane · 2026-01-28T06:58:51Z

Addressed all review comments, thanks @str4d

nuttycom

re-utACK 1e58b2a

str4d

utACK 1e58b2a

LarryRuane requested review from daira and nuttycom October 24, 2025 05:54

LarryRuane added the enhancement New feature or request label Oct 24, 2025

LarryRuane mentioned this pull request Oct 24, 2025

Add transparent data to the CompactBlock format. zcash/lightwallet-protocol#1

Merged

nuttycom requested review from arya2, pacu and str4d October 24, 2025 13:41

LarryRuane force-pushed the 2025-10-add-transparent branch from 1a2759d to 427ccbd Compare October 27, 2025 05:20

pacu approved these changes Oct 27, 2025

View reviewed changes

nuttycom requested changes Oct 27, 2025

View reviewed changes

pacu mentioned this pull request Oct 27, 2025

PR Reviews pacu/zcash-dev-rel-engineer#345

Closed

LarryRuane commented Oct 28, 2025

View reviewed changes

nuttycom force-pushed the 2025-10-add-transparent branch from 427ccbd to 4185b7c Compare November 1, 2025 03:02

LarryRuane changed the title ~~2025 10 add transparent~~ add transparent data to compact block serialization Nov 2, 2025

pacu requested changes Nov 6, 2025

View reviewed changes

common/common.go Outdated Show resolved Hide resolved

pacu mentioned this pull request Nov 6, 2025

PR Reviews pacu/zcash-dev-rel-engineer#369

Closed

LarryRuane force-pushed the 2025-10-add-transparent branch from 4185b7c to 151268a Compare November 7, 2025 18:23

nuttycom requested review from nuttycom and pacu November 10, 2025 19:22

LarryRuane force-pushed the 2025-10-add-transparent branch from 151268a to 1f4eb17 Compare November 10, 2025 22:41

pacu reviewed Nov 10, 2025

View reviewed changes

parser/transaction.go Outdated Show resolved Hide resolved

lightwallet-protocol/walletrpc/compact_formats.proto Outdated Show resolved Hide resolved

LarryRuane force-pushed the 2025-10-add-transparent branch from 1f4eb17 to 2b6c547 Compare November 10, 2025 23:15

LarryRuane mentioned this pull request Nov 14, 2025

add header to compact block #532

Closed

LarryRuane force-pushed the 2025-10-add-transparent branch from 37e8ca7 to 5f48fc4 Compare January 8, 2026 00:58

LarryRuane force-pushed the 2025-10-add-transparent branch from 5f48fc4 to 401e540 Compare January 10, 2026 02:03

start cache at height 0 instead of sapling activation

62d74de

The main work is adding the ability to parse pre-V4 (Sapling) transactions. I used zcash: src/primitives/transaction.h to see how the serialization is done.

LarryRuane force-pushed the 2025-10-add-transparent branch 2 times, most recently from 1d59069 to a41a2de Compare January 12, 2026 21:05

LarryRuane force-pushed the 2025-10-add-transparent branch from a41a2de to d2de8f5 Compare January 13, 2026 07:59

idky137 mentioned this pull request Jan 19, 2026

Add compact transparent tx data representation to compact blocks and update relevant gRPC services. zingolabs/zaino#800

Closed

str4d requested changes Jan 26, 2026

View reviewed changes

nuttycom mentioned this pull request Jan 27, 2026

Define GetBlock behavior in relation to transparent data zcash/lightwallet-protocol#18

Closed

Larry Ruane added 7 commits January 27, 2026 23:36

add transparent inputs and outputs to compact transactions

b529f76

Don't write coinbase tx transparent input to compact block

b1b17af

This is a space and network optimization; the coinbase input doesn't contain relevant information for us (our compact block inputs are just an outpoint, which is a txid and index, and for the coinbase, these aren't meaningful).

improve readability converting byte slice to hash32

a5e6488

No functional difference, just clarity. Change calls such as: hash32.T(someslice) to: hash32.FromSlice(someslice)

GetBlockRange filtering (poolTypes)

b5cf6bd

This uses the recently-added BlockRange.poolTypes argument; previously, GetBlockRange ignored this argument.

update CHANGELOG.md

1e58b2a

LarryRuane force-pushed the 2025-10-add-transparent branch from d2de8f5 to 1e58b2a Compare January 28, 2026 06:48

nuttycom approved these changes Jan 29, 2026

View reviewed changes

LarryRuane dismissed pacu’s stale review via 1e58b2a January 29, 2026 17:14

pacu mentioned this pull request Jan 29, 2026

Adopt lightclient-protocol v0.4.0 zingolabs/zaino#808

Closed

str4d approved these changes Jan 29, 2026

View reviewed changes

nuttycom merged commit e17bfec into zcash:master Jan 29, 2026
6 checks passed

LarryRuane deleted the 2025-10-add-transparent branch January 29, 2026 23:49

	// CompactTxIn values corresponding to the `vin` entries of the full transaction.
	// CompactTxIn values corresponding to the `vin` entries of the full transaction.


		### Added

		- Add block header to GetBlock and GetBlockRange output (issue 527)

Conversation

LarryRuane commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LarryRuane commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pacu commented Oct 24, 2025

Uh oh!

LarryRuane commented Oct 24, 2025

Uh oh!

pacu left a comment

Choose a reason for hiding this comment

Uh oh!

pacu Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nuttycom left a comment

Choose a reason for hiding this comment

Uh oh!

nuttycom Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nuttycom Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

LarryRuane Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

LarryRuane left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LarryRuane Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LarryRuane Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

LarryRuane commented Nov 7, 2025

Uh oh!

nuttycom commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LarryRuane commented Nov 10, 2025

Uh oh!

pacu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

LarryRuane commented Nov 13, 2025

Uh oh!

LarryRuane commented Jan 8, 2026

Uh oh!

LarryRuane commented Jan 8, 2026

Uh oh!

LarryRuane commented Jan 10, 2026

Uh oh!

nuttycom commented Jan 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LarryRuane commented Oct 24, 2025 •

edited

Loading

LarryRuane commented Oct 24, 2025 •

edited

Loading

nuttycom commented Nov 7, 2025 •

edited

Loading

nuttycom commented Jan 11, 2026 •

edited

Loading