Index off-chain data with "File Data Source"

[alexandre-abrioux]

Solving this issue would also solve #20

Issue

The current logic that fetches IPFS files makes the overall indexing depend on whether or not the Graph Node was able to find a file when ipfs.cat was called. This is not deterministic and can lead to incomplete data when the IPFS node is down for some time, or is overloaded, and the network request times out.

Solution

Last year, TheGraph introduced the notion of "File Data Source", available on Graph Nodes since version 0.30, which exactly solves that issue. More details available on their blog post: https://thegraph.com/blog/file-data-sources-tutorial/

Proposal: migrate our existing subgraph to implement the "File Data Source" framework.

References

• File data source blog post: https://thegraph.com/blog/file-data-sources-tutorial/
• File data source documentation: https://thegraph.com/docs/en/subgraphs/developing/creating/advanced/#ipfsarweave-file-data-sources
• [graph-node] IPFS availability issue: IPFS data sources graphprotocol/graph-node#1017
• [graph-node] File data source implementation: File data sources graphprotocol/graph-node#3072

[MantisClone]

Thanks @alexandre-abrioux! This looks interesting. I'll discuss it with the team.