Receive & record "subpiece" information for AddPieces operations

[rvagg]

Currently: addPieces(uint256 setId, Cids.Cid[] calldata pieceData, bytes calldata extraData)

Curio has support for building a single piece from multiple "subpieces", similar to the Filecoin miner actor sector onboarding that takes multiple pieces and combines them in to one root CommD. With a manifest subpiece PieceCIDs (including size, so CommPv2), you can verify that they resolve to a parent PieceCID, but you can't go the other way obviously.

Now that we use CommPv2 everywhere, we have a problem with this subgraph mechanism because the aggregate piece size is now going to be larger (>=) than the sum of the raw sizes of the subpieces, because we have to arrange them in such a way that they align properly into the tree needed to form the aggregate piece. This has downstream effects because of CommP being a source of piece size:

• PDPVerifier thinks its verifying the full aggregate piece size, not the combined raw size of the subpieces
• PDP Explorer will be seeing this size too and reporting it as the raw size of the piece
• SDK & user will have a CommP that will report that it references a piece that is larger (potentially significantly larger) than the actual combined raw sizes of the pieces

Curio should be able to deal with this, I think. I haven't verified but Magik built this and I trust his ability to get this right.

I think this should be OK as long as we're clear with users and limit the uses of it to where it's necessary. In order to have nice overlap with PoRep, I think we should treat 32G as the boundary (minus fr32), and have Curio refuse to accept piece upload of larger than this, as well as the SDK. If you want to onboard a PDP piece larger than this, then you have to use the subpiece mechanism up to whatever size you need. But, we should record it somewhere so this isn't just ephemeral data between the SDK and Curio.

So my proposal is:

• Extend addPieces to take a "PieceManifest", similarly mapping to the concept in the miner actor except in this case we will use (SubPieceCID, Offset) pairs (by using offsets, we give ourselves additional flexibility to arrange content and potentially insert meta content (e.g. PoDSI data) -- these pieces combined will produce my aggregate PieceCID. I'm not sure what the call signature for this should be, maybe it needs a separate one for when you want to do this.
• For now, due to the gas cost of verifying this in the contract, it will just trust that this information is correct. All we know here is that Curio reported this, so we're essentially trusting Curio. We also have the option of passing this manifest through extraData into the service contract and verifying the client signature there, in which case we can then have assurance that (a) Curio reported this and thinks it's correct, but more importantly, (b) the user is asserting this is what they want recorded in the deal. We make these trust assumptions very clear in the contract.
• We emit this information as an event, don't store it, but we now have it on record and an indexer could pick it up and use it as needed.
• The SDK has a mechanism for uploading very large data that breaks the 32G boundary, but in return you don't just get a single CommP (with the wrong raw size in it), you get that and a PieceManifest to go along with it that lists your subpieces that made up the entire piece.
• The PDP Explorer is going to report the wrong size, but it can pick up the piece manifest and report that along with the piece.
• Linking with PoRep (archive()) will happen either at the sub-32G level (putting together multiple pieces into a larger manifest for a single sector) or at the above-32G level by committing subpieces separately and using the piece manifest to do that and be able to verify that your aggregate piece is available.

This assumes that Curio can (a) prove these large aggregate pieces (probably can now), and (b) can allow /piece/ retrieval of either aggregate or subpieces (apparently mkv2 is supposed to support this).

[rvagg]

I was thinking that we should cap piece size for PDP to 32Gib (minus fr32) but it may be more ideal to go smaller, but there's a trade-off: smaller chunks means you have smaller verifiable units (good) but also mean you have more of them for large data and your manifest is larger and more costly/complicated to report, emit, retain (bad). 4GiB could be a good choice - you can verify your data in 4GiB chunks, 8 of them make up a full PoRep sector and we lower the upload connection-loss risk. Curio's default hard limit is ~256MiB, which would be 128 sub-pieces for a PoRep sector, and quite a large blob to pass to a contract and have emitted and collected by a subgraph, even more problematic when you get up to much larger size pieces.

From a user perspective:

1. Transparent Large File Support

// Works the same for 100MB or 100GB
const result = await storage.upload(largeBlobStream)
// Returns: { commP, manifest? } 
// manifest only present if >4GiB

2. User Choices:

• Self-custody: Store CommP + manifest locally
• Trust subgraph: Store only aggregate CommP, retrieve manifest later
• Verify correctness either way: verifyManifest(manifest, aggregateCommP) → true/false

3. Archive to PoRep:

// Small data: direct archive
await archive(commP)

// Large data: manifest-based archive
await archiveWithManifest(commP, manifest)
// SDK handles subpiece → sector mapping

// Large data: abstracted subpiece handling with subgraph
await archive(commP)
// SDK handles CommP → manifest fetch & verify
// SDK handles subpiece → sector mapping

[bajtos]

To me, 4 GiB still seems a pretty large chunk to upload at once - it means ~6 minutes on a 100MBit/s broadband connection. It will be impossible to successfully upload such a large chunk on a less reliable connection, unless there is a resume mechanism in place.

How do you envision the retrieval story?

For example, let's say I upload a 10GB video file via storage.upload(largeBlobStream) and get back { commP, manifest }. How can I retrieve the video file from the SP, preferably as a single stream?

[rvagg]

Retrieval is straightforward as Curio will just stitch it back together for us when we ask for that CommP, should work with byte ranges too. At least that's the promise (and if it doesn't work, it should be able to). So retrieval is transparent here.

We can go lower than 4 GiB, but we're trading off on manifest size and complexity. The current size in Curio is capped at 256 MiB, partly for the reason that you're highlighting, but as I said above, it's 128 subpieces for a full 32 GiB sector, and even more if you have a piece that's larger than that. That's not impossible, we're not going to store those on chain but we do need to get them in a message and emitted from an event (or events). So that's going to put a cap on our maximum piece size unless we come up with a batching message mechanism for this.

[bajtos]

I see your points about the tradeoffs between the chunk size, manifest size and complexity.

It makes me wonder if it would be easier to bite the bullet and figure out resumable uploads to Curio? Isn't that the real need we are trying to solve by introducing subpieces?

[rvagg]

It's not the main one I care about, although it is why subpieces exist in Curio in the first place from what I understand. In here I'm mostly concerned about capping out at the PoRep sector size so we have a nice 1:1 (ish) correspondence between PDP and PoRep when we get there. If we end up with proving units larger than a PoRep sector then we have to slice it up to fit it into a sector and the hash will be different - better to have subpieces with a manifest that can be shoved into sectors and you can see using chain data that your original "piece" (the large, aggregate one), is properly sectioned up into units that provably combine into your singular CommP but then also can be shunted into different sectors and proven there and still maintain that relationship for you.

So finding the right subpiece size is about trading off around having a usable manifest that we can send to the chain and emit and have on record somewhere for later reference when we need to archive() and also prove that some aggregate CommP equals the combined manifest.

[rvagg]

Storacha are (still) doing 1GiB pieces, maybe we just unify on that as our piece sizes, that's 32 pieces per sector/manifest, gets quite big for uploads larger than that, we'd just want to test what the limits are of pushing such a manifest through a message to emit, or come up with some other strategy to get it "recorded".

[rvagg]

Thinking about this further in #224; given the blow-out in extraData size to get a maxxed out metadata message in to Curio, I think that we should consider having a completely separate contract who's job it is to "index" subpieces. Probably just to emit events, not even store anything interesting, although it could do some basic validation against PDPVerifier and/or WarmStorage perhaps. Curio would just call that contract to have the records emitted.