[Project] GA durability (2 copies; 1 with endorsed SP)

[BigLep]

This is a tracking issue for all the work to fully land our minimum-bar durability story for GA to make sure we don't lose track of any of the details. This tracks the project work across multiple repos. We're sticking this in filecoin-services, because we don't currently have a better over-arching place.

The expectation is that Synapse will store two copies by default:

• One copy to an "endorsed" SP (which is also an "approved" SP)
• One copy to another "approved" SP

Tasks

Contracts

• (Endorsement Providers Contract) Create contract with deployment script: Endorsed Providers Contract #382
• (Endorsement Providers Contract) Deploy contract with multisig to calibration and mainnet

Synapse

• (Endorsement Certificates) Store a copy with an "endorsed" SP by default: feat: Endorsement Certificates synapse-sdk#443
• (Endorsement Certificates) Have tooling for "endorsing" an SP: feat: Endorsement Certificates synapse-sdk#443
• (Endorsement Providers Contract) Use endorsement providers contract: feat: Endorsements Service synapse-sdk#553
• (@rvagg) Design desired API for exposing multiple copies. Happening in https://www.notion.so/filecoindev/Multi-copy-Synapse-Upload-2bddc41950c1803d9fb6c8830c8a7a28
• GA DURABILITY: Multi-copy upload via SP-to-SP pull synapse-sdk#494
• Simplify provider & data set resolution logic & API synapse-sdk#495
• Do a release so that filecoin-pin can use it

Curio

• feat!(ServiceProviderRegistry): Hex Capabilities filecoin-project/curio#816
• pdpv0: Add SP-to-SP piece fetch endpoint filecoin-project/curio#828
• fix(pdp): register pulled pieces in pdp_piecerefs for service ownership filecoin-project/curio#943

foc-devnet

• (Endorsement Providers Contract) Add endorsements contract to devnet foc-devnet#41
• (Endorsement Providers Contract) Ability to set an "endorsed SP" foc-devnet#50
• Inter-SP communication setup foc-devnet#39
• Do a release that uses this new version of synapse that does 2-copies

User Documentation

• Synapse docs cover durability story.

SP Community

• Endorsement Certificates) Onboard at least X SPs with endorsement certificates

Synapse Demo App

• Do 2 copies by default

filecoin-pin

• Do 2 copies by default

Operations

• (Endorsement Certificates) Runbook/process for how to endorse an SP
  • Happening in https://www.notion.so/filecoindev/FOC-Operational-Excellence-2b7dc41950c1802aa432fff8ecb801cc

Notes

Past Issues and Docs

https://www.notion.so/filecoindev/GTM-Replication-280dc41950c180c78684d3eda77c3164
FilOzone/synapse-sdk#275
#250

Endorsement Certificates

As a short term measure to endorse reliable "Approved" SPs, we'll be adding endorsement certifications, as this doesn't rely on any addition contract deployments.

Ideally FOC would enable storing arbitrary key/value pairs about SPs. We have an ApproveList in the FilecoinWarmStorageService, but it would have been ideal if this was more generic to allow key value pairs. That would allow us to say an SP is "Approve" and in the "endorsed inner circle".

[BigLep]

@rjan90 @rvagg : let me know if this has been covered elsewhere, but I wanted to make sure we have a place that covers all the details for fully landing our "GA durability" story. Fully completing this touches quite a few ares if I understand correctly and wanted to make none of the details are lost. Feel free to close, correct, or add to it.

[BigLep]

Design desired API for exposing multiple copies is happening in https://www.notion.so/filecoindev/Multi-copy-Synapse-Upload-2bddc41950c1803d9fb6c8830c8a7a28

[rvagg]

Added curio and synapse issues above