[Discussion] bundler-shard

[christhekeele]

Wanted to lead with giving you major props for shard. It perfectly fills a void in the GitHub ecosystem!

It also would be a major component for filling a void I've felt in the gem ecosystem: patch management.

Semantic Versioning, Gems, and Bundler enable Rubyists to construct 3rd-party dependency stacks beyond the wildest dreams of other languages. However, even with the best practices in place, stacks can grow sufficiently complicated that patching is necessary. Ruby makes this trivial, so in a way everything balances out. But at the end of the day a lot of developers end up maintaining the same patches individually, since there's no tool for managing their development, distribution, or deprecation.

A great example would be my patch for ActiveRecord and Kaminari, which I need for Rails 4 with RailsAdmin. Kaminari doesn't work with ActiveRecord in certain contexts, which is fine and inevitable. There's an open issue for it, the resolution of which is tied to an open issue in ActiveRecord. I had to build this monkey patch out of the discussion threads on both issues, which is also reasonable.

The problem is that now I have to check in and remember to monitor this patch, potentially for forever. The reason why my patch is so extensively documented, and located in my source code under config/initializers/DANGER_ZONE/ is to encourage me to regularly check in on the issues (rails/rails#10658 and amatsuda/kaminari#421) in progress and search for resolution, alternatives, or even discovery of major problems that these patches have been discovered to introduce. And I can only hope every other person who's had to apply them as well makes the same attempts at vigilance.

I'd like to consolidate this burden of patch maintenance through an extension to Bundler, adding a shard method to their DSL, that allows you to pull in and apply patches in much the same way the gem method works. This would allow the community to collaborate on the daunting but necessary task of patch management instead of going code cowboy on it the way we have now.

Originally I had a patch DSL method in mind that would hook into Gists, but the complexities of this problem paired with the lack of a good formalized mechanism for pulling code down from Gists similar to gem kept me from wanting to address it.

With shard, though, I'm feeling bolder. I think tackling this challenge is more in reach and I was hoping to gauge your interest and get your input on such an undertaking, as well as start a discussion anywhere in public that might help bend the minds of the Ruby community to implement such a tool.

[mark]

That's not a bad idea, although I'm a bit nervous about using shards in production deploys. I've been going back and forth on adding versioning to shards (via sha of the commit to start with, I think semantic versions are a lot heavier than I want shard to be, and adds a lot of overhead to creating / maintaining them). But I think I'll ultimately have to, since without any sort of versioning, they'd be too unpredictable to depend on, and so won't be usable for much of anything.

So what do you think the chances are of such a pull request being accepted into bundler?

Although: given the fact that shards auto download the first time you use them, do they need a separate gem command? Wouldn't calling shard in your gem file be sufficient?

[mark]

Oh, and: Thanks! :-)

[christhekeele]

I think simple sha versioning makes sense in this context. Patches are meant to have a short lifecycle, and my idea of bundler-shard is to avoid having to update them manually during that lifecycle. So specifying a version should be a rare action–you'll want to be running the latest of that patch most of the time. You'd have to pass in the sha if you want to change that, no need for symantic versioning.

I think as long as shard has a way of grabbing versions of gist histories by sha, the version-finding logic could be handled in bundler-shard. I don't think it makes sense to have something like a .shardspec that extends dependency graph, at least at first.

With those capabilities, I do think just calling shard in your Gemfile should suffice.

I doubt such a thing would be accepted into bundler; the idea would be to make a gem that extends vanilla bundler in this direction.

So on a new ruby install, your first actions might be

$ gem install bundler shard bundler-shard

instead of just bundler. Then it'd be able to handle the shard directive in any Gemfiles it finds.