Proposal: Always get the latest buildpack from heroku

[tolstenko]

Goal:

When downloading buildpacks from heroku, always download the latest version.

Overview

Instead of editing the file install-buildpacks everytime heroku creates a new version, I think it is safe to assume the new version is stable enough for us to use it. This approach will reduce maintenance burden.
If a given user wants to lock the buildpack version, it could do it by setting the BUILD_PACK env with the version:
deis config:set BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-php#v149
The same way I had to do in order to make example-php work again for me.
I don't know when this code runs. If it runs once, this approach wont be useful. But if it runs every time, so we will always be updated with heroku builds.

Code Changes

The script install-buildpacks would fetch the latest tag and use it. It be something similar to this code instead of hardcoding the version:
https://github.com/gr1d-io/slugbuilder/blob/master/rootfs/builder/install-buildpacks

Testing

I am running the make tests right now

[kingdonb]

Thank you for creating a design document! RFC

We are generally in favor of changes that reduce the maintenance burden, although this one reduces traceability and build reproducibility.

I agree that "Heroku's latest buildpack" is probably always preferable to whatever version we have pinned, we are not maintaining any buildpacks on our own, and it's not a particularly heavy burden for users that know they need a specific buildpack version to be expected to require it themselves in a BUILDPACK_URL var. Maybe there are some users who are not going to like this change. Buildpack upgrades should be backwards compatible, so it would be good to have a reference document with some text supporting that, to cite from Heroku in our documentation.

This PR should come with a docs update with that reference in it, if possible, and a paragraph that explains for users, just how to do that (buildpack version locking with BUILDPACK_URL, something like http://dokku.viewdocs.io/dokku~v0.4.14/deployment/buildpacks/#specifying-a-custom-buildpack).

I spoke with @tolstenko offline and we agreed that in general traceability and reproducible builds are a good thing, but in this case you found that the "heroku-buildpack-php" was no longer functioning with the example-php, even though I vaguely remember that all of the buildpack e2e tests passed when Hephy v2.20 was released. So, something must have changed upstream and caused our build to break?

That should ideally not have been possible, but here we are. So the facts appear to show that we do not have a purely reproducible environment in buildpacks now, and I guess maybe it's somewhere in the tagged Heroku buildpacks where there are unspecified things, or upstream changes leaking in? Or perhaps there was a release that was removed from upstreams due to security issues, and that's why it failed. I think there is generally an expectation that you should upgrade more frequently than we are upgrading. It would be good to improve that with a sweeping change like you're proposing.

Maybe have a way that users can globally "opt-out" their Workflow clusters from always pulling the buildpack latest, by pinning a configured upstream version of one or all built-in buildpacks. Nice to have, a backup plan documented for cluster admins, for when Heroku breaks something in the latest release and then goes on holiday, doomsday scenario where a distributed team is affected and all new builds are breaking.

Something for the cluster admin who needs to apply a consistent setting across the entire workflow instance in builder. Maybe that's too much. One could always just build your own "builder" image. Just spitballing.

We don't really have precedent in Team Hephy for how long design documents are to stay in review, but if there are issues with some of the buildpacks in the current release, then we will need to do something to address it in the next release. This proposal does not seem like a bad option.

[Cryptophobia]

Let's all discuss. The fact that we increment buildpack versions I think is a good design because we are showing which versions we support at a minimum.

However, when things break, it's not usually our fault but the buildpack might be at fault too. Backwards compatibility with the slugs is also a factor here.

If we do not hard-code buildpack versions then do we run the risk of breaking backwards compatibility with older slugs?

[till]

I agree, we should not auto-update the buildpacks. E.g. the Christmas release of Ruby and the release of the buildpack (on the same day) are notorious for breaking applications because people don't pin them. ;) E.g. breaking BC in patch releases.

I don't know how much other language maintainers vet their buildpacks, but I would prefer not to auto-update them as it comes back to us and doesn't allow for repeatable or immutable runs.

---

Having said all this, should hephy provide a flag (disabled by default) to auto-update? I think then it's a conscious if you turn it on/off, etc..

[duanhongyi]

Is it better to support Zip or GZ compressed package formats? just like this:

deis config:set BUILDPACK_URL=https://github.com/user/repository/archive/branch.zip

or

deis config:set BUILDPACK_URL=https://my-php-buildbacks.s3.amazon.com/php.tar.gz

I think it's more controllable and flexible.

[Cryptophobia]

Having said all this, should hephy provide a flag (disabled by default) to auto-update? I think then it's a conscious if you turn it on/off, etc..

I like this idea. We could make it so it always pulls the latest tag on the buildpack if a specific variable is defined like BUILDPACK_LATEST=true. Should be an easy PR.

Anybody, interest in making this simple PR? Should be a small change on slugbuilder and controller to add the variable and pass to slugbuilder and a new list of latest tag for the buildpack urls.

Is it better to support Zip or GZ compressed package formats? just like this:

Could be. Have not really tested the pros vs. the cons of that approach. For one thing, you can't clearly see from the URL what version of the buildpack you are actually using which is making things more obscure and unsafe as you could be using a malicious BUILDPACK_URL unknowingly.