Proposal: Replace existing overlay API with one based around io.FS (or just get rid of it altogether)

[snuxoll]

The overlay support in cue/load is undoubtedly useful in many applications, but the design of intercepting absolute file paths causes some problems not only in usage of the overlay feature itself; but also in other places within the code (a large issue I have had myself is custom modregistries cannot just return an io.FS backed in memory, in a ZIP file, embedded with go:embed, or other virtual file system that does not have a reasonable way to implement OSRootFS to make cue/load happy).

I know #607 has been open for years, and I’m sure I’m not the only person who’s silently hung their head dealing with this - so I’m happy to contribute a patch if we can come up with a solution that makes everyone happy.

There’s a few major questions to answer here:

1. Do we really need absolute paths anywhere but when trying to dump errors? Given a cue package cannot access anything outside of its directory, it seems scoping the overlay to the root of a local module or package would be sufficient.
2. If relative paths are indeed sufficient for an overlay API change, what would the changes to the context and build APIs need to look like.
3. Given paths are indeed useful for debugging, an alternative to OSRootFS to expose a base URI for virtual file systems instead of assuming an absolute file system path would seem wise.

Alternatively, if we can simply accept anything that implements io.FS in a context do we even need to implement overlay support in cue anymore? Any user could simply implement a simple overlay (as had already been done internally) that implements io.FS themselves.

[myitcv]

Thanks for raising this for discussion, @snuxoll. @rogpeppe did some work in this space ahead of our work on CUE modules. In short (but I will let @rogpeppe expand) this change has implications deep within CUE.

As an additional comment, as part of work on cue lsp (a more recent update on the most recent community call) we will ultimately be addressing this, to move towards io.FS.