Add field to store/add to include commP

[Gozala]

We should add a field to store/add capability in order to include commP proof with every write

@ribasushi is there js code somewhere we could use to compute it on client side ?

@mikeal computing it on the client side is fine, but we can’t trust client & would need to verify, which makes me wonder if doing it on a client is even worth it, as we may only complicate things by having to handle case where client provides incorrect value.

[ribasushi]

@Gozala the client-side calculation is so that a client has a full chain-of-custody, that starts on their own machine.

The only thing I am aware of as prior-art is https://github.com/rvagg/js-fil-utils. @rvagg can you chime in wrt completeness / whether you'd recommend it to be used today as-is?

A test corpus is available here: https://github.com/filecoin-project/go-fil-commp-hashhash/tree/master/testdata

[Gozala]

@Gozala the client-side calculation is so that a client has a full chain-of-custody, that starts on their own machine.

Wouldn't same exact CAR file produce same exact commP ? If so I'd argue user providing CAR CID does provide full chain-of-custody. We will capture computed commP in the receipt and sign it so we can be hold accountable if computed incorrectly.

[ribasushi]

commP is a stream-hash, a bit weird one internally, but for an end-user behaving exactly like say sha256. What is special about commP is that a user can plug it into a Fil oracle directly, without going through w3s at all, and verify that indeed you guys did deliver the exact bytes that were uploaded, all the way to the SP. You can not do this with a CAR CID, nor with sha256 nor with any other hash aside from the fil-native commP.

Hence you want to train your users ( at least these that care about Fil ) to key everything they upload off commP. Hence the user-side calculation.

[mikeal]

if the client lies about commp, then the data won’t land in a deal.

when the miner notices it doesn’t match they invalidate the deal and can notify the broker (spade) why and they can create a new deal that just drops that data.

the purpose of the cryptography being used here is to give trustless guarantees between actors who don’t already have the underlying data to calculate the proof. the moment you accept that some actor is “responsible” for validation that isn’t the miner or the client you’ve accept the permanent need for an actor to sit between them with a full copy of the data.

there may be some communication necessary between actors to repair the deal, but it must be accomplished using the proofs to avoid actors between the client and the miner having to validate each proof themselves against the deal data because this will always necessitate an additional copy (additional infra) between the client and filecoin forever, or as long as you use that means of deal creation.

[ribasushi]

when the miner notices it doesn’t match they invalidate the deal

Better - the chain consensus itself ( specifically the PoRep part ) does this. CommD in the linked verification function is the "CommP of the entire sector", with all deals and free space. Every single node in the network ensures that the storage claim is legit when it is made.

[Gozala]

@mikeal so if I understand you correctly we basically don't do anything with the CommP other than store it in the invocation, correct ?

What do we do if we have two users uploading same CAR but specify different CommP ? Do we not care and just store ?

P.S.: I'm less worried about someone lying, and more about a bug in the client and deploying updates to clients.

[Gozala]

@yusefnapora found this relevant repo https://github.com/rvagg/rust-fil-commp-generate

[ribasushi]

@Gozala @yusefnapora note: this contains a pretty old copy of the currently used rust code: https://github.com/filecoin-project/rust-fil-proofs/blob/128f7209/filecoin-proofs/src/api/mod.rs#L322-L353
It will work, but possibly missing a lot of speedups since.

[Gozala]

@ribasushi thanks for the pointer! I think creating a wasm wrapper library from there is probably a best way to go about it. Also probably a good self-contained project for hacking on rust/wasm

[rvagg]

So https://github.com/rvagg/js-fil-utils does commp, it's written for Node.js files but that's just cause that's all I cared about at the time. It's written for uint8array iterables so you'd just rip out the Node.js imports and make it take a plain asynciterable and replace the crypto with (probably) native browser sha2-256. I wouldn't want to be doing this in the browser over very large assets, but I assume we're talking relatively small things.

WASM might be overkill? It's really not that complicated a process, maybe WASM gives you a perf boost but the most expensive thing is calling out to sha2-256. Have a look at the JS code, there's 3 main components to it - zero padding to get up to a ^2, fr32 padding to space out 2 bits every 254, and a merkle tree using a slightly truncated/zeroed sha2-256 result.