From 4a2d228aa9cab377933f6d969d35f719d56bd978 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 10 Apr 2025 16:31:38 -0400 Subject: [PATCH 1/5] Decouple extend phase from kaniko Signed-off-by: Natalie Arellano --- text/0000-extract-extender.md | 256 ++++++++++++++++++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 text/0000-extract-extender.md diff --git a/text/0000-extract-extender.md b/text/0000-extract-extender.md new file mode 100644 index 000000000..d38b3b3e8 --- /dev/null +++ b/text/0000-extract-extender.md @@ -0,0 +1,256 @@ +# Meta +[meta]: #meta +- Name: Decouple extend phase from kaniko +- Start Date: 2025-04-10 +- Author(s): natalieparellano +- Status: Draft +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: (put "N/A" unless this replaces an existing RFC, then link to that RFC) + +# Summary +[summary]: #summary + +The lifecycle currently uses [kaniko](https://github.com/GoogleContainerTools/kaniko) as a library to perform build- and run-image extension, but kaniko has fallen into [unmaintained](https://github.com/GoogleContainerTools/kaniko/issues/3348) status, making it impractical and undesirable to rely on this dependency. We should remove the kaniko dependency from the lifecycle. + +We should do this by: + +Phase 1 +* Separating the extender binary from the lifecycle +* Delivering the extender separately from the lifecycle + +Phase 2 +* Providing an alternative implementation of the extender that uses a different "dockerfile applier" - e.g., buildah or buildkit +* Removing kaniko-specific references from the platform spec + +# Definitions +[definitions]: #definitions + +## Extender binary + +The extender, according to the [platform spec](https://github.com/buildpacks/spec/blob/main/platform.md#outputs-3): + +- For each extension in `` in order, if a Dockerfile exists in `//.Dockerfile`, the lifecycle: + - SHALL apply the Dockerfile to the environment according to the process outlined in the [Image Extension Specification](image-extension.md). + - SHALL set the build context to the folder according to the process outlined in the [Image Extension Specification](image-extension.md). +- The extended image MUST be an extension of: + - The `build-image` in `` when `` is `build`, or + - The `run-image` in `` when `` is `run` +- When extending the build image, after all `build.Dockerfile`s are applied, the lifecycle: + - SHALL proceed with the `build` phase using the provided `` and `` +- When extending the run image, after all `run.Dockerfile`s are applied, the lifecycle: + - **If** any `run.Dockerfile` set the label `io.buildpacks.rebasable` to `false` or left the label unset: + - SHALL set the label `io.buildpacks.rebasable` to `false` on the extended run image + - **If** after the final `run.Dockerfile` the run image user is `root`, + - SHALL fail + - SHALL copy the manifest and config for the extended run image, along with any new layers, to ``/run + +**Kaniko currently does this piece**: `SHALL apply the Dockerfile to the environment according to the process outlined in the Image Extension Specification` + +Where the [image extension spec](https://github.com/buildpacks/spec/blob/main/image_extension.md#phase-extension) says: + +- The lifecycle MUST provide each Dockerfile with: +- A `base_image` build arg + - For the first Dockerfile, the value MUST be the original base image. + - When there are multiple Dockerfiles, the value MUST be the intermediate image generated from the application of the previous Dockerfile. +- A `build_id` build arg + - The value MUST be a UUID +- `user_id` and `group_id` build args + - For the first Dockerfile, the values MUST be the original `uid` and `gid` from the `User` field of the config for the original base image. + - When there are multiple Dockerfiles, the values MUST be the `uid` and `gid` from the `User` field of the config for the intermediate image generated from the application of the previous Dockerfile. + +Additionally: + +When extending the build image: + +- In addition to the outputs enumerated below, outputs produced by `extender` include those produced by `builder` - as the lifecycle will run the `build` phase after extending the build image. +- Platforms MUST skip the `builder` and proceed to the `exporter`. + +## Lifecycle delivery + +The lifecycle is delivered as a tarball on the GitHub releases page, and as an image at `buildpacksio/lifecycle:`. In both cases, the directory structure looks like the following: + +``` +Permission UID:GID Size Filetree +drwxr-xr-x 0:0 32 MB └── cnb +drwxr-xr-x 0:0 32 MB ├── lifecycle +-rwxrwxrwx 0:0 0 B │ ├── analyzer → lifecycle +-rwxrwxrwx 0:0 0 B │ ├── builder → lifecycle +-rwxrwxrwx 0:0 0 B │ ├── creator → lifecycle +-rwxrwxrwx 0:0 0 B │ ├── detector → lifecycle +-rwxrwxrwx 0:0 0 B │ ├── exporter → lifecycle +-rwxrwxrwx 0:0 0 B │ ├── extender → lifecycle +-rwxr-xr-x 0:0 2.8 MB │ ├── launcher +-rw-r--r-- 0:0 10 kB │ ├── launcher.sbom.cdx.json +-rw-r--r-- 0:0 16 kB │ ├── launcher.sbom.spdx.json +-rw-r--r-- 0:0 15 kB │ ├── launcher.sbom.syft.json +-rwxr-xr-x 0:0 29 MB │ ├── lifecycle +-rw-r--r-- 0:0 121 kB │ ├── lifecycle.sbom.cdx.json +-rw-r--r-- 0:0 187 kB │ ├── lifecycle.sbom.spdx.json +-rw-r--r-- 0:0 126 kB │ ├── lifecycle.sbom.syft.json +-rwxrwxrwx 0:0 0 B │ ├── rebaser → lifecycle +-rwxrwxrwx 0:0 0 B │ └── restorer → lifecycle +-rw-r--r-- 0:0 560 B └── lifecycle.toml +``` + +Notice how the extender is just a symlink pointing back to the lifecycle. + +## Kaniko-specific references in the platform spec + +### restorer + +* `` is a required input when using image extensions +* `/cache` is an output + +Expected behavior: +- When `` is provided (optional), the lifecycle: + - MUST record the digest reference to the provided `` in `` + - MUST copy the OCI manifest and config file for `` to `/cache` +- The lifecycle: + - **If** `` has `run-image.extend = true`, the lifecycle: + - MUST download from the registry and save in OCI layout format the `run-image` in `` to `/cache` + +E.g., after the restorer has run, `/cache/base//oci-layout` should exist. If we are not using kaniko then this may no longer be necessary! + +### extender + +* `` is a required input +* `` is an optional input +* `/cache` is an output (the changes here are specific to kaniko) +* The ability to extend the build image in-container and drop down to the CNB user to run the build phase could potentially be a kaniko-specific behavior that we might not be able to replicate with an alternative implementation. + +# Motivation +[motivation]: #motivation + +- Why should we do this? + - It is insecure to rely on an unsupported dependency + - We're currently blocked on upgrading the docker dependency in the lifecycle because of kaniko +- What use cases does it support? + - More flexibility for platforms to "bring your own" extender +- What is the expected outcome? + - It should still be possible to use image extensions by using CNB-provided tooling + +# What it is +[what-it-is]: #what-it-is + +- Target persona: platform operator, platform implementor, and/or project contributor. + +* Starting from lifecycle 0.21.0 (the next minor as of this writing), we should remove the extender from [cmd/lifecycle](https://github.com/buildpacks/lifecycle/tree/main/cmd/lifecycle) + * We could create [cmd/extender](https://github.com/buildpacks/lifecycle/tree/main/cmd) with its own go.mod + * This will allow us to upgrade the docker version used by the lifecycle +* We should deliver the extender via a separate tarball on the GitHub releases page, and as a separate image at `buildpacksio/extender:`. In both cases, the directory structure would look like the following: + +``` +Permission UID:GID Size Filetree +drwxr-xr-x 0:0 32 MB └── cnb +drwxr-xr-x 0:0 32 MB ├── lifecycle +-rwxrwxrwx 0:0 0 B │ ├── extender +-rw-r--r-- 0:0 10 kB │ ├── extender.sbom.cdx.json +-rw-r--r-- 0:0 16 kB │ ├── extender.sbom.spdx.json +-rw-r--r-- 0:0 15 kB │ ├── extender.sbom.syft.json +-rw-r--r-- 0:0 560 B └── lifecycle.toml +``` + +* When creating builders, pack will inspect the lifecycle version provided and pull the extender with the same version from the GitHub releases page. + * This mimics the [behavior](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/create_builder.go#L437) when NO lifecycle version is specified - pack will pull the latest lifecycle that it knows about from the GitHub releases page. + * Alternatively, we could add a `[lifecycle.extender]` table in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-optional), but this is more cumbersome to implement. + * Alternatively, we could look for a sibling tarball or image for the lifecycle provided (e.g., if provided `buildpacksio/lifecycle:` we could look for `buildpacksio/extender:`), but this seems risky as it is non-declarative. +* pack should put `/cnb/lifecycle/lifecycle` and `/cnb/lifecycle/extender` within the SAME "lifecycle layer" in the builder image. + * From a platform's perspective, nothing about the builder would have really changed, except that the extender is no longer a symlink. + +* When running builds, platforms would launch extend-build and extend-run containers + * Using the builder image (as before) when doing build image extension + * Using the run image + extender binary from the _extender_ image (this is different) when doing run image extension + * See [here](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/build.go#L662) for how this is working today in pack + * Other platforms would need to make a similar change in order to upgrade to a newer lifecycle version + + + +* We will need to explore alternatives to kaniko for Dockerfile application. + * A spike using the Docker Daemon was already completed, see here: https://github.com/buildpacks/pack/pull/1791. We were able to get it working and saw improved performance when extending the build image, but in the end we felt that the additional complexity was too much to support. + * Note that instead of extending the build image in-container and dropping down to the CNB user to run the build phase, we simply extended the build image and saved it back to the Docker Daemon with a new reference. We then provided the new reference as the builder to use going forward. We could have done something similar when extending the run image (i.e., updated the reference in analyzed.toml). + * Other alternatives considered are buildah and buildkit. + +* Once we have an alternative extender working, we can decide which spec changes (if any) are needed to make it work. + * We might need to remove the stipulation that we extend the build image in-container and drop down to the CNB user to run the build phase if that is not technically feasible. This was always only an optimization, so we should be okay here. + +* For cosmetic reasons, we should at least change: + * `` -> `` + * `` -> `` + + + +# How it Works +[how-it-works]: #how-it-works + +Some code that will probably need to change: +* TODO + +# Migration +[migration]: #migration + +Phase 1 +* Platforms that are performing run image extension must take care to use the run image + extender binary from the _extender_ image (instead of the lifecycle image), as noted above. + +Phase 2 +* There may be a Platform API version bump for spec changes. + +# Drawbacks +[drawbacks]: #drawbacks + +Why should we *not* do this? +* It's more work! +* Platforms that are performing run image extension will have to make changes in order to upgrade the lifecycle to the version that uses the new delivery mechanism. This could be surprising. We should take care to socialize this change in advance of its deployment. + +# Alternatives +[alternatives]: #alternatives + +- What other designs have been considered? + - We thought about just shoving the new extender binary into the existing tarball & image, but this would + - Probably increase the size of the image (currently about 30 MB) + - Not solve the problem of the lifecycle image getting flagged by vulnerability scanners (at least for the period of time when we are relying on the old kaniko implementation) + - Though, if we're still packing the extender into builders, builders are still going to get flagged + - Delivering the new extender binary as a separate tarball & image would make it easier for platforms to prohibit the use of extensions by excluding the extender binary from builders, but that is beyond the scope of this RFC + - This would actually be the quickest and easiest to deliver, and would require no changes from platforms, so we should consider it. +- Why is this proposal the best? Maybe it's not the best, I don't know! Let's discuss. +- What is the impact of not doing this? Risk of the lifecycle becoming outdated & insecure. + +# Prior Art +[prior-art]: #prior-art + +Discuss prior art, both the good and bad. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- What parts of the design do you expect to be resolved before this gets merged? +- What parts of the design do you expect to be resolved through implementation of the feature? +- What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC? + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes +Does this RFC entail any proposed changes to the core specifications or extensions? If so, please document changes here. +Examples of a spec. change might be new lifecycle flags, new `buildpack.toml` fields, new fields in the buildpackage label, etc. +This section is not intended to be binding, but as discussion of an RFC unfolds, if spec changes are necessary, they should be documented here. + +# History +[history]: #history + + \ No newline at end of file From 13d2047eb7da6726ef80e05bf0be0b9972b4eb86 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Thu, 10 Apr 2025 17:07:23 -0400 Subject: [PATCH 2/5] Add code references Signed-off-by: Natalie Arellano --- text/0000-extract-extender.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/text/0000-extract-extender.md b/text/0000-extract-extender.md index d38b3b3e8..ec0f9c99b 100644 --- a/text/0000-extract-extender.md +++ b/text/0000-extract-extender.md @@ -185,7 +185,15 @@ drwxr-xr-x 0:0 32 MB ├── lifecycle [how-it-works]: #how-it-works Some code that will probably need to change: -* TODO +* https://github.com/buildpacks/lifecycle/blob/main/.github/workflows/build.yml - we could templatize this file to build either a vanilla lifecycle or an extender +* https://github.com/buildpacks/lifecycle/blob/main/.github/workflows/post-release.yml - where we sign and promote the lifecycle image +* https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/create_builder.go#L270 would also need to fetch an extender +* https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/internal/builder/builder.go#L1029 would also need to embed an extender +* https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/build.go#L662 would need to fetch the "lifecycle layer" from the extender image, not the lifecycle image + * Currently, we support `pack build --lifecycle-image` to use the non-default one, perhaps we'd need to add `--extender-image` as well + * https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/internal/config/config.go#L152 - where we fall back to buildpacksio/lifecycle if the image is not explicitly provided +* https://github.com/buildpacks/lifecycle/blob/c895ed40025a70f3de5a197f111c7d27687e1d80/cmd/lifecycle/extender.go#L59 - if we're able to find a drop-in replacement for kaniko it would go here +* https://github.com/buildpacks/lifecycle/blob/c895ed40025a70f3de5a197f111c7d27687e1d80/phase/extender.go#L264 - the meat of extension logic is in this file; an alternative extender implementation would import or duplicate this logic - see for example https://github.com/buildpacks/pack/pull/1791/files#diff-3f80ad23f1b13e75bf22e1a8aaad7d86d57e41fb823cf6142e3588c67661d556R738 # Migration [migration]: #migration From 023b3255e3d873f196b5f60b6a50fba20b2f74b9 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Fri, 11 Apr 2025 13:14:38 -0400 Subject: [PATCH 3/5] Not so opinionated about separate delivery Signed-off-by: Natalie Arellano --- text/0000-extract-extender.md | 138 ++++++++++++++++++---------------- 1 file changed, 72 insertions(+), 66 deletions(-) diff --git a/text/0000-extract-extender.md b/text/0000-extract-extender.md index ec0f9c99b..100c4f2b9 100644 --- a/text/0000-extract-extender.md +++ b/text/0000-extract-extender.md @@ -15,58 +15,14 @@ The lifecycle currently uses [kaniko](https://github.com/GoogleContainerTools/kaniko) as a library to perform build- and run-image extension, but kaniko has fallen into [unmaintained](https://github.com/GoogleContainerTools/kaniko/issues/3348) status, making it impractical and undesirable to rely on this dependency. We should remove the kaniko dependency from the lifecycle. We should do this by: - -Phase 1 -* Separating the extender binary from the lifecycle -* Delivering the extender separately from the lifecycle - -Phase 2 -* Providing an alternative implementation of the extender that uses a different "dockerfile applier" - e.g., buildah or buildkit -* Removing kaniko-specific references from the platform spec +1. Separating the extender binary from the lifecycle +2. Delivering the extender separately from the lifecycle (maybe?) +3. Providing an alternative implementation of the extender that uses a different "dockerfile applier" - e.g., buildah or buildkit +4. Removing kaniko-specific references from the platform spec, along with any other spec changes that are needed # Definitions [definitions]: #definitions -## Extender binary - -The extender, according to the [platform spec](https://github.com/buildpacks/spec/blob/main/platform.md#outputs-3): - -- For each extension in `` in order, if a Dockerfile exists in `//.Dockerfile`, the lifecycle: - - SHALL apply the Dockerfile to the environment according to the process outlined in the [Image Extension Specification](image-extension.md). - - SHALL set the build context to the folder according to the process outlined in the [Image Extension Specification](image-extension.md). -- The extended image MUST be an extension of: - - The `build-image` in `` when `` is `build`, or - - The `run-image` in `` when `` is `run` -- When extending the build image, after all `build.Dockerfile`s are applied, the lifecycle: - - SHALL proceed with the `build` phase using the provided `` and `` -- When extending the run image, after all `run.Dockerfile`s are applied, the lifecycle: - - **If** any `run.Dockerfile` set the label `io.buildpacks.rebasable` to `false` or left the label unset: - - SHALL set the label `io.buildpacks.rebasable` to `false` on the extended run image - - **If** after the final `run.Dockerfile` the run image user is `root`, - - SHALL fail - - SHALL copy the manifest and config for the extended run image, along with any new layers, to ``/run - -**Kaniko currently does this piece**: `SHALL apply the Dockerfile to the environment according to the process outlined in the Image Extension Specification` - -Where the [image extension spec](https://github.com/buildpacks/spec/blob/main/image_extension.md#phase-extension) says: - -- The lifecycle MUST provide each Dockerfile with: -- A `base_image` build arg - - For the first Dockerfile, the value MUST be the original base image. - - When there are multiple Dockerfiles, the value MUST be the intermediate image generated from the application of the previous Dockerfile. -- A `build_id` build arg - - The value MUST be a UUID -- `user_id` and `group_id` build args - - For the first Dockerfile, the values MUST be the original `uid` and `gid` from the `User` field of the config for the original base image. - - When there are multiple Dockerfiles, the values MUST be the `uid` and `gid` from the `User` field of the config for the intermediate image generated from the application of the previous Dockerfile. - -Additionally: - -When extending the build image: - -- In addition to the outputs enumerated below, outputs produced by `extender` include those produced by `builder` - as the lifecycle will run the `build` phase after extending the build image. -- Platforms MUST skip the `builder` and proceed to the `exporter`. - ## Lifecycle delivery The lifecycle is delivered as a tarball on the GitHub releases page, and as an image at `buildpacksio/lifecycle:`. In both cases, the directory structure looks like the following: @@ -120,6 +76,46 @@ E.g., after the restorer has run, `/cache/base//cache` is an output (the changes here are specific to kaniko) * The ability to extend the build image in-container and drop down to the CNB user to run the build phase could potentially be a kaniko-specific behavior that we might not be able to replicate with an alternative implementation. +## Responsibilities of the extender + +The extender, according to the [platform spec](https://github.com/buildpacks/spec/blob/main/platform.md#outputs-3): + +- For each extension in `` in order, if a Dockerfile exists in `//.Dockerfile`, the lifecycle: + - SHALL apply the Dockerfile to the environment according to the process outlined in the [Image Extension Specification](image-extension.md). + - SHALL set the build context to the folder according to the process outlined in the [Image Extension Specification](image-extension.md). +- The extended image MUST be an extension of: + - The `build-image` in `` when `` is `build`, or + - The `run-image` in `` when `` is `run` +- When extending the build image, after all `build.Dockerfile`s are applied, the lifecycle: + - SHALL proceed with the `build` phase using the provided `` and `` +- When extending the run image, after all `run.Dockerfile`s are applied, the lifecycle: + - **If** any `run.Dockerfile` set the label `io.buildpacks.rebasable` to `false` or left the label unset: + - SHALL set the label `io.buildpacks.rebasable` to `false` on the extended run image + - **If** after the final `run.Dockerfile` the run image user is `root`, + - SHALL fail + - SHALL copy the manifest and config for the extended run image, along with any new layers, to ``/run + +**Kaniko currently does this piece**: `SHALL apply the Dockerfile to the environment according to the process outlined in the Image Extension Specification` + +Where the [image extension spec](https://github.com/buildpacks/spec/blob/main/image_extension.md#phase-extension) says: + +- The lifecycle MUST provide each Dockerfile with: +- A `base_image` build arg + - For the first Dockerfile, the value MUST be the original base image. + - When there are multiple Dockerfiles, the value MUST be the intermediate image generated from the application of the previous Dockerfile. +- A `build_id` build arg + - The value MUST be a UUID +- `user_id` and `group_id` build args + - For the first Dockerfile, the values MUST be the original `uid` and `gid` from the `User` field of the config for the original base image. + - When there are multiple Dockerfiles, the values MUST be the `uid` and `gid` from the `User` field of the config for the intermediate image generated from the application of the previous Dockerfile. + +Additionally: + +When extending the build image: + +- In addition to the outputs enumerated below, outputs produced by `extender` include those produced by `builder` - as the lifecycle will run the `build` phase after extending the build image. +- Platforms MUST skip the `builder` and proceed to the `exporter`. + # Motivation [motivation]: #motivation @@ -136,9 +132,15 @@ E.g., after the restorer has run, `/cache/base/`. In both cases, the directory structure would look like the following: ``` @@ -152,12 +154,15 @@ drwxr-xr-x 0:0 32 MB ├── lifecycle -rw-r--r-- 0:0 560 B └── lifecycle.toml ``` -* When creating builders, pack will inspect the lifecycle version provided and pull the extender with the same version from the GitHub releases page. - * This mimics the [behavior](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/create_builder.go#L437) when NO lifecycle version is specified - pack will pull the latest lifecycle that it knows about from the GitHub releases page. - * Alternatively, we could add a `[lifecycle.extender]` table in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-optional), but this is more cumbersome to implement. - * Alternatively, we could look for a sibling tarball or image for the lifecycle provided (e.g., if provided `buildpacksio/lifecycle:` we could look for `buildpacksio/extender:`), but this seems risky as it is non-declarative. -* pack should put `/cnb/lifecycle/lifecycle` and `/cnb/lifecycle/extender` within the SAME "lifecycle layer" in the builder image. - * From a platform's perspective, nothing about the builder would have really changed, except that the extender is no longer a symlink. +* This should be accompanied by a minor version bump, due to the breaking change in the delivery mechanism +* When creating builders, pack could look for a `[lifecycle.extender]` table in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-optional) to find the extender binary + * If `[lifecycle.extender]` is NOT present, pack could inspect the lifecycle version provided and pull the extender with the same version from the GitHub releases page + * This mimics the [behavior](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/create_builder.go#L437) when NO lifecycle version is specified (pack will pull the latest lifecycle that it knows about from the GitHub releases page) + * Alternatively, we could look for a sibling tarball or image for the lifecycle provided (e.g., if provided `buildpacksio/lifecycle:` we could look for `buildpacksio/extender:`), but this seems risky as it is non-declarative + * Alternatively, we could put the lifecycle and the extender in separate layers within the same image +* pack should put `/cnb/lifecycle/lifecycle` and `/cnb/lifecycle/extender` within the SAME "lifecycle layer" in the builder image + * We'd need to be careful not to write the parent directory (`/cnb/lifecycle`) twice, as this can produce an invalid image + * From a platform's perspective, nothing about the builder would have really changed, except that the extender is no longer a symlink * When running builds, platforms would launch extend-build and extend-run containers * Using the builder image (as before) when doing build image extension @@ -165,13 +170,15 @@ drwxr-xr-x 0:0 32 MB ├── lifecycle * See [here](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/build.go#L662) for how this is working today in pack * Other platforms would need to make a similar change in order to upgrade to a newer lifecycle version - +## 3. Providing an alternative implementation of the extender * We will need to explore alternatives to kaniko for Dockerfile application. * A spike using the Docker Daemon was already completed, see here: https://github.com/buildpacks/pack/pull/1791. We were able to get it working and saw improved performance when extending the build image, but in the end we felt that the additional complexity was too much to support. * Note that instead of extending the build image in-container and dropping down to the CNB user to run the build phase, we simply extended the build image and saved it back to the Docker Daemon with a new reference. We then provided the new reference as the builder to use going forward. We could have done something similar when extending the run image (i.e., updated the reference in analyzed.toml). * Other alternatives considered are buildah and buildkit. +## 4. Removing kaniko-specific references from the platform spec + * Once we have an alternative extender working, we can decide which spec changes (if any) are needed to make it work. * We might need to remove the stipulation that we extend the build image in-container and drop down to the CNB user to run the build phase if that is not technically feasible. This was always only an optimization, so we should be okay here. @@ -179,8 +186,6 @@ drwxr-xr-x 0:0 32 MB ├── lifecycle * `` -> `` * `` -> `` - - # How it Works [how-it-works]: #how-it-works @@ -198,10 +203,10 @@ Some code that will probably need to change: # Migration [migration]: #migration -Phase 1 +If we move the extender to a new image: * Platforms that are performing run image extension must take care to use the run image + extender binary from the _extender_ image (instead of the lifecycle image), as noted above. -Phase 2 +If we update the spec: * There may be a Platform API version bump for spec changes. # Drawbacks @@ -209,20 +214,21 @@ Phase 2 Why should we *not* do this? * It's more work! -* Platforms that are performing run image extension will have to make changes in order to upgrade the lifecycle to the version that uses the new delivery mechanism. This could be surprising. We should take care to socialize this change in advance of its deployment. +* If we move the extender to a new image, platforms that are performing run image extension will have to make changes in order to upgrade the lifecycle to the version that uses the new delivery mechanism. This could be surprising. We should take care to socialize this change in advance of its deployment. # Alternatives [alternatives]: #alternatives - What other designs have been considered? - - We thought about just shoving the new extender binary into the existing tarball & image, but this would - - Probably increase the size of the image (currently about 30 MB) + - We thought about NOT moving the extender binary, but this would + - Probably increase the size of the lifecycle image (currently about 30 MB) - Not solve the problem of the lifecycle image getting flagged by vulnerability scanners (at least for the period of time when we are relying on the old kaniko implementation) - Though, if we're still packing the extender into builders, builders are still going to get flagged - Delivering the new extender binary as a separate tarball & image would make it easier for platforms to prohibit the use of extensions by excluding the extender binary from builders, but that is beyond the scope of this RFC - - This would actually be the quickest and easiest to deliver, and would require no changes from platforms, so we should consider it. -- Why is this proposal the best? Maybe it's not the best, I don't know! Let's discuss. -- What is the impact of not doing this? Risk of the lifecycle becoming outdated & insecure. + - This would actually be the quickest and easiest to deliver, and would require no changes from platforms, so we should consider it + - We could fork kaniko and continue to use it as the dockerfile applier. Maybe it wouldn't be too hard to maintain, especially if we rip out the parts that we're not using. +- Why is this proposal the best? Maybe it's not the best, I don't know! Let's discuss +- What is the impact of not doing this? Risk of the lifecycle becoming outdated & insecure # Prior Art [prior-art]: #prior-art From 522c54059b9e8cc2506b6f582d25c57f6dba2486 Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Fri, 11 Apr 2025 13:21:58 -0400 Subject: [PATCH 4/5] Link and wording Signed-off-by: Natalie Arellano --- text/0000-extract-extender.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/text/0000-extract-extender.md b/text/0000-extract-extender.md index 100c4f2b9..e77991982 100644 --- a/text/0000-extract-extender.md +++ b/text/0000-extract-extender.md @@ -121,7 +121,7 @@ When extending the build image: - Why should we do this? - It is insecure to rely on an unsupported dependency - - We're currently blocked on upgrading the docker dependency in the lifecycle because of kaniko + - We're currently [blocked](https://github.com/buildpacks/imgutil/pull/287#issuecomment-2769797254) on upgrading the docker dependency in the lifecycle because of kaniko - What use cases does it support? - More flexibility for platforms to "bring your own" extender - What is the expected outcome? @@ -141,7 +141,7 @@ When extending the build image: ## 2. Delivering the extender separately from the lifecycle (maybe?) -* We should deliver the extender via a separate tarball on the GitHub releases page, and as a separate image at `buildpacksio/extender:`. In both cases, the directory structure would look like the following: +* We could deliver the extender via a separate tarball on the GitHub releases page, and as a separate image at `buildpacksio/extender:`. In both cases, the directory structure would look like the following: ``` Permission UID:GID Size Filetree @@ -154,7 +154,7 @@ drwxr-xr-x 0:0 32 MB ├── lifecycle -rw-r--r-- 0:0 560 B └── lifecycle.toml ``` -* This should be accompanied by a minor version bump, due to the breaking change in the delivery mechanism +* This should be accompanied in the lifecycle by a minor version bump, due to the breaking change in the delivery mechanism * When creating builders, pack could look for a `[lifecycle.extender]` table in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-optional) to find the extender binary * If `[lifecycle.extender]` is NOT present, pack could inspect the lifecycle version provided and pull the extender with the same version from the GitHub releases page * This mimics the [behavior](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/create_builder.go#L437) when NO lifecycle version is specified (pack will pull the latest lifecycle that it knows about from the GitHub releases page) @@ -168,7 +168,7 @@ drwxr-xr-x 0:0 32 MB ├── lifecycle * Using the builder image (as before) when doing build image extension * Using the run image + extender binary from the _extender_ image (this is different) when doing run image extension * See [here](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/build.go#L662) for how this is working today in pack - * Other platforms would need to make a similar change in order to upgrade to a newer lifecycle version + * Other platforms would need to make a similar change in order to upgrade to a newer lifecycle version IF they are looking in the lifecycle image for the extender binary (vs the builder) ## 3. Providing an alternative implementation of the extender @@ -220,13 +220,13 @@ Why should we *not* do this? [alternatives]: #alternatives - What other designs have been considered? - - We thought about NOT moving the extender binary, but this would + - We thought about NOT moving the extender binary to a new image, but this would - Probably increase the size of the lifecycle image (currently about 30 MB) - Not solve the problem of the lifecycle image getting flagged by vulnerability scanners (at least for the period of time when we are relying on the old kaniko implementation) - Though, if we're still packing the extender into builders, builders are still going to get flagged - - Delivering the new extender binary as a separate tarball & image would make it easier for platforms to prohibit the use of extensions by excluding the extender binary from builders, but that is beyond the scope of this RFC - - This would actually be the quickest and easiest to deliver, and would require no changes from platforms, so we should consider it - - We could fork kaniko and continue to use it as the dockerfile applier. Maybe it wouldn't be too hard to maintain, especially if we rip out the parts that we're not using. + - Delivering the new extender binary as a separate tarball & image would make it easier for platforms that don't support extensions to exclude the extender binary from builders, but that is beyond the scope of this RFC + - Keeping the extender in the lifecycle image (as a separate binary, not a symlink) would be quick and easy to deliver, and would require no changes from platforms, so we should consider it + - We could fork kaniko and continue to use it as the dockerfile applier. Maybe it wouldn't be too hard to maintain, especially if we rip out the parts that we're not using - Why is this proposal the best? Maybe it's not the best, I don't know! Let's discuss - What is the impact of not doing this? Risk of the lifecycle becoming outdated & insecure From cdee8c7e2e7349aa2ba5c8c6a9af75288131cb3f Mon Sep 17 00:00:00 2001 From: Natalie Arellano Date: Fri, 6 Jun 2025 13:28:03 -0400 Subject: [PATCH 5/5] Update approach * Removed the separate delivery mechanism for the extender * Added the optional "slim" version concept for future consideration * Simplified the migration path Signed-off-by: Natalie Arellano --- text/0000-extract-extender.md | 57 ++++++----------------------------- 1 file changed, 10 insertions(+), 47 deletions(-) diff --git a/text/0000-extract-extender.md b/text/0000-extract-extender.md index e77991982..6f23f1df0 100644 --- a/text/0000-extract-extender.md +++ b/text/0000-extract-extender.md @@ -16,9 +16,8 @@ The lifecycle currently uses [kaniko](https://github.com/GoogleContainerTools/ka We should do this by: 1. Separating the extender binary from the lifecycle -2. Delivering the extender separately from the lifecycle (maybe?) -3. Providing an alternative implementation of the extender that uses a different "dockerfile applier" - e.g., buildah or buildkit -4. Removing kaniko-specific references from the platform spec, along with any other spec changes that are needed +2. Providing an alternative implementation of the extender that uses a different "dockerfile applier" - e.g., buildah or buildkit +3. Removing kaniko-specific references from the platform spec, along with any other spec changes that are needed # Definitions [definitions]: #definitions @@ -138,46 +137,17 @@ When extending the build image: * We could create [cmd/extender](https://github.com/buildpacks/lifecycle/tree/main/cmd) with its own go.mod * This will allow us to upgrade the docker version used by the lifecycle * We'll have to update our `make` targets to build both binaries + * The extender binary will still be included in the lifecycle tarball and image, but as a separate binary rather than a symlink + * Optionally and in the future, we could publish a "slim" version of the lifecycle that excludes the extender binary, for platforms that don't need image extension support -## 2. Delivering the extender separately from the lifecycle (maybe?) - -* We could deliver the extender via a separate tarball on the GitHub releases page, and as a separate image at `buildpacksio/extender:`. In both cases, the directory structure would look like the following: - -``` -Permission UID:GID Size Filetree -drwxr-xr-x 0:0 32 MB └── cnb -drwxr-xr-x 0:0 32 MB ├── lifecycle --rwxrwxrwx 0:0 0 B │ ├── extender --rw-r--r-- 0:0 10 kB │ ├── extender.sbom.cdx.json --rw-r--r-- 0:0 16 kB │ ├── extender.sbom.spdx.json --rw-r--r-- 0:0 15 kB │ ├── extender.sbom.syft.json --rw-r--r-- 0:0 560 B └── lifecycle.toml -``` - -* This should be accompanied in the lifecycle by a minor version bump, due to the breaking change in the delivery mechanism -* When creating builders, pack could look for a `[lifecycle.extender]` table in [builder.toml](https://buildpacks.io/docs/reference/config/builder-config/#lifecycle-optional) to find the extender binary - * If `[lifecycle.extender]` is NOT present, pack could inspect the lifecycle version provided and pull the extender with the same version from the GitHub releases page - * This mimics the [behavior](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/create_builder.go#L437) when NO lifecycle version is specified (pack will pull the latest lifecycle that it knows about from the GitHub releases page) - * Alternatively, we could look for a sibling tarball or image for the lifecycle provided (e.g., if provided `buildpacksio/lifecycle:` we could look for `buildpacksio/extender:`), but this seems risky as it is non-declarative - * Alternatively, we could put the lifecycle and the extender in separate layers within the same image -* pack should put `/cnb/lifecycle/lifecycle` and `/cnb/lifecycle/extender` within the SAME "lifecycle layer" in the builder image - * We'd need to be careful not to write the parent directory (`/cnb/lifecycle`) twice, as this can produce an invalid image - * From a platform's perspective, nothing about the builder would have really changed, except that the extender is no longer a symlink - -* When running builds, platforms would launch extend-build and extend-run containers - * Using the builder image (as before) when doing build image extension - * Using the run image + extender binary from the _extender_ image (this is different) when doing run image extension - * See [here](https://github.com/buildpacks/pack/blob/a1edf2e9e3837c2e9eeca019b97c28896b40f778/pkg/client/build.go#L662) for how this is working today in pack - * Other platforms would need to make a similar change in order to upgrade to a newer lifecycle version IF they are looking in the lifecycle image for the extender binary (vs the builder) - -## 3. Providing an alternative implementation of the extender +## 2. Providing an alternative implementation of the extender * We will need to explore alternatives to kaniko for Dockerfile application. * A spike using the Docker Daemon was already completed, see here: https://github.com/buildpacks/pack/pull/1791. We were able to get it working and saw improved performance when extending the build image, but in the end we felt that the additional complexity was too much to support. * Note that instead of extending the build image in-container and dropping down to the CNB user to run the build phase, we simply extended the build image and saved it back to the Docker Daemon with a new reference. We then provided the new reference as the builder to use going forward. We could have done something similar when extending the run image (i.e., updated the reference in analyzed.toml). * Other alternatives considered are buildah and buildkit. -## 4. Removing kaniko-specific references from the platform spec +## 3. Removing kaniko-specific references from the platform spec * Once we have an alternative extender working, we can decide which spec changes (if any) are needed to make it work. * We might need to remove the stipulation that we extend the build image in-container and drop down to the CNB user to run the build phase if that is not technically feasible. This was always only an optimization, so we should be okay here. @@ -203,10 +173,6 @@ Some code that will probably need to change: # Migration [migration]: #migration -If we move the extender to a new image: -* Platforms that are performing run image extension must take care to use the run image + extender binary from the _extender_ image (instead of the lifecycle image), as noted above. - -If we update the spec: * There may be a Platform API version bump for spec changes. # Drawbacks @@ -214,18 +180,15 @@ If we update the spec: Why should we *not* do this? * It's more work! -* If we move the extender to a new image, platforms that are performing run image extension will have to make changes in order to upgrade the lifecycle to the version that uses the new delivery mechanism. This could be surprising. We should take care to socialize this change in advance of its deployment. +* The lifecycle image will still include the extender binary, which means it will still get flagged by vulnerability scanners until we implement an alternative to kaniko. # Alternatives [alternatives]: #alternatives - What other designs have been considered? - - We thought about NOT moving the extender binary to a new image, but this would - - Probably increase the size of the lifecycle image (currently about 30 MB) - - Not solve the problem of the lifecycle image getting flagged by vulnerability scanners (at least for the period of time when we are relying on the old kaniko implementation) - - Though, if we're still packing the extender into builders, builders are still going to get flagged - - Delivering the new extender binary as a separate tarball & image would make it easier for platforms that don't support extensions to exclude the extender binary from builders, but that is beyond the scope of this RFC - - Keeping the extender in the lifecycle image (as a separate binary, not a symlink) would be quick and easy to deliver, and would require no changes from platforms, so we should consider it + - We thought about moving the extender binary to a new image, but this would + - Require significant changes from platforms that are performing run image extension + - Though the lifecycle image might no longer get flagged by vulnerability scanners, if we're still packing the extender into builders, builders are still going to get flagged - We could fork kaniko and continue to use it as the dockerfile applier. Maybe it wouldn't be too hard to maintain, especially if we rip out the parts that we're not using - Why is this proposal the best? Maybe it's not the best, I don't know! Let's discuss - What is the impact of not doing this? Risk of the lifecycle becoming outdated & insecure