[epic] `rv sync` to install full bundles

[indirect]

Today, we have rv clean-install, which is roughly equivalent to bundle install --frozen. The next major step for managing projects is rv sync, which works like bundle install.

The first pass at the sync mechanism looks something like:

• Get a list of dependencies from the Gemfile via ruby -e 'YAML.dump(Bundler.definition.dependencies)'
• Fetch the latest information about requested gems from the gem server /info/ files
• Recursively build a list of every possible requested dependency by reading /info/ files
• Feed the full list of requested dependencies and available packages into pubgrub to get a resolution
• write out a new Gemfile.lock
• Install the resolved gems just like clean-install does
• use the existing Gemfile.lock as a basis for resolving the Gemfile

Once that is working, full functionality has a few additional steps:

• Serialize the resolution as gem.lock (which is still a kdl file, but not a human-editable one)
• If Gemfile.lock or gem.lock exist, use that as the basis for the new resolution to minimize churn
• Once this is all working, automatically sync as a step during rv run
• Best-effort translation of Gemfile into gem.kdl on demand when rv init is run
  • we could potentially try to parse the Gemfile ourselves looking for common patterns
  • we could also try to run Bundler to eval the Gemfile and then print out a data structure of the result
  • either way, it's impractical to run Ruby and Bundler to eval the entire Gemfile every time, so we need some translator
• Read the list of requested dependencies from gem.kdl's section named dependencies

@deivid-rodriguez am I missing anything here that we should add to the first pass?

[deivid-rodriguez]

@indirect Yes, that sounds like a plan. I think we could consider moving the fifth bullet point to an additional step because we already know how to build a Gemfile.lock file, so we may as well start generating that.

Regarding additional steps, we'll need to figure out how to resolve sources, too. I.e. when fetching the list of all possible dependencies, assign the single most relevant source for each of them, or abort with an error if the source is ambiguous. But for an initial pass, we can consider there's a single global source.

Also, since we'll now have an static manifest, we may want to save the manifest checksum in the lockfile, so we have a quick way of knowing if we need a re-resolve (the checksum manifest in the lockfile no longer matches the manifest). But we'll still want to do something similar to Bundler anyways to keep locked gems in places as much as possible.

Regarding the translation phase, using Bundler to evaluate the Gemfile seems the most bulletproof way, but yeah, agreed we definitely don't want to do that every time.

[adamchalmers]

So this would require building the gem.kdl machinery first, right? So for release 0.6 we can plan to have gem.kdl stuff, and rv sync.

[indirect]

@adamchalmers I think there are two reasonable ways to approach this: we can either tackle only gem.kdl and reading/installing/locking from gem.kdl, or we can write a shim that runs bundler to eval and then serialize and print a Gemfile, and use that as the input for our installing/locking system. After more pondering, I am leaning towards the second one, where we basically layer gem.kdl on top as an optimization that people can opt in to once they are satisfied with using rv to install their gems.

[adamchalmers]

If we can agree on a Rust structure like struct ListOfDeps { ... }, then we can each work on our half of the contract:

• You implement some way to build ListOfDeps from Gemfile, or from gem.kdl, or after running some Ruby script or whatever
• I write code that takes a ListOfDeps and installs from it.

in your original post, basically, you'd take the first checkbox item, and I'd take all the rest. Most of the work on my side is already done, right? This is basically just rv tool install except of resolving the deps for a single gem, we're finding all deps for a list of gems, then resolving them all together.

My proposal is

struct DepsToSync {
  deps: Vec<DepToSync>
}

struct DepToSync {
  gem_name: String,
  gem_version: rv_version::Version,
}

[indirect]

@adamchalmers so the good news and the bad news is that your proposal is completely reasonable but doesn't contain information about things like "does this gem come from a git repo, or gem.coop, or avohq.io, or etc". here's a first pass at just your list of deps:

rv run ruby -rbundler -rpsych -e 'puts Bundler.definition.dependencies.to_yaml'

Here's the much more terrifying command that will dump the entire internal data structure, including all the information we will eventually need about sources, the current lockfile, etc, etc:

rv run ruby -rbundler -rpsych -e 'puts Bundler.definition.to_yaml'

[deivid-rodriguez]

I agree with starting with the slow approach that needs shelling out to Bundler 👍.

For an initial iteration, we could consider only Gemfiles with a single global source, and add some check in the ruby code to fetch dependencies to reject multiple source Gemfiles as not yet supported. As long as the data structure we choose can be easily modified later to account for the extra complexity, it sounds like a reasonable first step?

[adamchalmers]

I think handling multiple gem sources should be fine actually. I'm not worried about that.

[adamchalmers]

OK, new proposal:

struct GemsToSync {
    gems: Vec<GemToSync>,
}

struct GemToSync {
    gem_name: String,
    // this VersionConstraints type is already defined in tool/install/gemserver.rs,
    // it's basically the information in ['~> 0.2', '> 0.1']
    gem_version: VersionConstraints,
    source: Url,
}

[indirect]

Sounds good! Here's an example YAML output of dependencies from Bundler after parsing the Discourse Gemfile:

❯ rv run ruby -rbundler -rpsych -e 'puts Bundler.definition.dependencies.to_yaml' | gist -p
https://gist.github.com/indirect/0c990eb83b6b3aeee98adc06d0bc683e

https://gist.github.com/indirect/0c990eb83b6b3aeee98adc06d0bc683e

[adamchalmers]

OK so then actually we'd just want a Vec<Dependency> where Dependency is defined here?

[indirect]

We can probably start there, yeah! We'll need to build up a more elaborate representation as we add features, but we should be able to get going from "here's a list of requirements" by parsing the YAML into a list of gems. 👍🏻

[adamchalmers]

Wait yeah I didn't read it properly, I see the list of information you've got now, sgtm. We'd need fields like Source at least.

[deivid-rodriguez]

I started working on the first step here: shelling out to Bundler to get the information we need from the Gemfile.