Stateless dagstore

[dirkmc]

Refactor DAG store to be stateless

Questions:

• Should we remove the RecoverOnStart configuration?
  It's not going to scale, so suggest we just remove it

[raulk]

We discussed this synchronously. This isn't a direction we'll want to pursue generally. It's only possible to remove these states for Boost because Lotus mounts are fully remote and never require local management of resources. However, the DagStore was conceived to be mount-agnostic and many mounts will require local caching of resources which, in turn, requires tracking usage and refcounts.

See #65 for an example. There are in fact 4-5 new usages of DagStore that I've recently became aware of, many of which hinge on GC.

That said, there is a key learning to absorb. In a horizontally scalable DagStore, not all states are equal.

• New, Available are common cluster-wide states.
• Initializing, Serving are local states.
• Recovering, Errored are somewhere in between, depending on the kind of error.

I would consider a refactor of the DagStore that modelled this scoping. But generally removing the local states to support Boost at the expense of other users and use cases is not something I would advise.