Feature: "children" pinning mode

[hsanjuan]

Version information: master

Type: Feature

Description:

I have discussed this with someone but apparently it was not written down. IPFS Cluster needs IPFS to pin partial trees, that is, a root CID and its immediate children (this is part of the sharding feature). Thus, I'll be working to add this feature to go-ipfs. I will reference this issue in upcoming PRs.

[kevina]

Moved from #5142:

Rather than complicate our pinner one potential solution is to enhance our notation of best-effort pins and fetch the needed subtrees separately.

[Stebalien]

So, I would like some kind of "download manager" to track download progress in MFS/best-effort pins. The main issue here would be accidentally downloading nodes we don't want to keep but keeping them anyways. @kevina what did you have in mind?

---

Also, talking this over with @whyrusleeping, I'd like to do the following:

1. Switch to the refmt based go-ipld-cbor
2. Finalize go-ipld-hamt (requires 1).
3. Store the pinset in go-ipld-hamt.

This would allow us to easily add extra metadata to pins and git rid of 90% of our current logic.

---

What can be done now:

• Depth-limited traversal
• Depth-limited ipfs refs -r

[hsanjuan]

I'll take on depth limited traversal and refs -r.

[0zAND1z]

Hi all,

I have been redirected to this issue, cited as a blocking item, to enable the DAG Sharding support in the ipfs-cluster.

Hoping that the information is still relevant, I would like to seek some clarity on the following questions:

Question 1: Is the purpose of depth-limit to enable the user to modulate the number of shards to be generated for a given file?

Consider a scenario where a file bearing a multihash X is to be pinned across the cluster be divided into an S number of shards(say x1, x2, x3 ... xS). Does the depth-limit support the creation of these number of shards? Is it necessary that all the S number of shards be persisted in minimum one node, or could these unique shards be distributed across the cluster?

Question 2: Are the shards subjected to distribution and overlapping for fault tolerance?

Consider the same file bearing a multihash X is to be pinned across the cluster be divided into an S number of shards(say x1, x2, x3 ... xS) in an overlapping manner across the M number of nodes in the cluster? Is this planned & made possible under the scope of the current issue?

All help in answering these questions are appreciated. Thanks.

[hsanjuan]

hi,

while this issue is a blocker to enable sharding in cluster, these are Cluster questions so they would better be answered in discuss.ipfs.io or the cluster repository, as they are not specific to the issue here.

Question 1: Is the purpose of depth-limit to enable the user to modulate the number of shards to be generated for a given file?

Consider a scenario where a file bearing a multihash X is to be pinned across the cluster be divided into an S number of shards(say x1, x2, x3 ... xS). Does the depth-limit support the creation of these number of shards?

As sharding is approached, it builds a parallel DAG referencing the original DAG nodes but with a different Layout. In this approach, we need to pin the shard CID (x1, x2 ...) and their children, which are the DAG nodes from the original object.

Is it necessary that all the S number of shards be persisted in minimum one node, or could these unique shards be distributed across the cluster?

Each shard needs to be fully pin in at least one node. But shard size can be configured.

Question 2: Are the shards subjected to distribution and overlapping for fault tolerance?

Consider the same file bearing a multihash X is to be pinned across the cluster be divided into an S number of shards(say x1, x2, x3 ... xS) in an overlapping manner across the M number of nodes in the cluster? Is this planned & made possible under the scope of the current issue?

you could pin each shard in several ipfs nodes. The number of shards depends on the size chosen for them, and then each shard works like an independent pin item.

Please follow up in discuss.ipfs.io if you have more questions about sharding.