From 9b0deb73a80e7f43781f6abe4224999ebc1f7a4f Mon Sep 17 00:00:00 2001 From: Anthony Vollmer Date: Fri, 2 May 2025 15:26:00 -0400 Subject: [PATCH 1/6] PKO-275 Add documentation for Global Image Prefix Override feature --- .../global-prefix-override.md | 90 +++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 content/en/docs/advanced_features/global-prefix-override.md diff --git a/content/en/docs/advanced_features/global-prefix-override.md b/content/en/docs/advanced_features/global-prefix-override.md new file mode 100644 index 0000000..0d8d0f3 --- /dev/null +++ b/content/en/docs/advanced_features/global-prefix-override.md @@ -0,0 +1,90 @@ +--- +title: Global Image Prefix Override +weight: 2001 +--- + +The Global Image Prefix Override feature enables deployment of upstream package-operator artifacts from private registries without rebuilding them in downstream pipelines. This solves the previously impossible scenario where organizations needed to mirror upstream images while maintaining original deployment manifests. + +## Implementation + +Prefix overrides can be configured through either: +- Environment variable: `PKO_IMAGE_PREFIX_OVERRIDES` +- Command-line flag: `image-prefix-overrides` + +Both accept mappings in `from=to` format (e.g., `quay.io/my-org/=mirror.example.com/mirror-org/`). When Package Operator encounters an image reference starting with the `from` prefix, it automatically substitutes it with the `to` prefix while preserving the remainder of the image path. + +## Usage Example + +The following `mirror.sh` script demonstrates a complete workflow for deploying Package Operator using mirrored images: + +```bash +#!/bin/bash +set -euxo pipefail + +# Mirror upstream images to private registry +skopeo copy --all \ + docker://quay.io/package-operator/package-operator-package:v1.18.2 \ + docker://quay.io/erdii-test/pko-mirror/package-operator-package:v1.18.2 + +skopeo copy --all \ + docker://quay.io/package-operator/package-operator-manager:v1.18.2 \ + docker://quay.io/erdii-test/pko-mirror/package-operator-manager:v1.18.2 + +skopeo copy --all \ + docker://quay.io/package-operator/remote-phase-manager:v1.18.2 \ + docker://quay.io/erdii-test/pko-mirror/remote-phase-manager:v1.18.2 + +# Download original bootstrap manifest +curl -LO https://github.com/package-operator/package-operator/releases/download/v1.18.2/self-bootstrap-job.yaml + +# Apply global prefix override +yq -i 'with( + select(.kind == "Job").spec.template.spec.containers[] + | select(.name == "package-operator").env[] + | select(.name == "PKO_IMAGE_PREFIX_OVERRIDES") + ; .value = "quay.io/package-operator/=quay.io/erdii-test/pko-mirror/" + )' self-bootstrap-job.yaml + +# Update direct image references +yq -i 'with( + select(.kind == "Job").spec.template.spec.containers[] + | select(.name == "package-operator") + ; ( + .image |= sub("quay.io/package-operator", "quay.io/erdii-test/pko-mirror"), + .args[0] |= sub("quay.io/package-operator", "quay.io/erdii-test/pko-mirror") + ) + )' self-bootstrap-job.yaml +``` + +## Key Components Explained + +### Image Mirroring: +The `skopeo copy` commands create identical copies of all upstream images in your private registry, maintaining the original tags and manifests. + +### Prefix Override Injection: +The first `yq` command modifies the PKO_IMAGE_PREFIX_OVERRIDES environment variable to redirect all quay.io/package-operator/ pulls to quay.io/erdii-test/pko-mirror/ + +The pattern `original-prefix/=new-prefix/` ensures complete path substitution while preserving image names and tags + +### Direct Reference Updates: +The second `yq` command handles hardcoded image references that bypass the prefix override mechanism by: + +Modifying the container's `.image` field +Updating the first command-line argument in .args[0] + +# Deployment Process + +## 1. Apply the prepared manifest on the target cluster: +```kubectl apply -f self-bootstrap-job.yaml``` + +## 2. Monitor progress until the ClusterPackage CRD becomes available +## 3. Verification: +``` +# Verify package image source +kubectl get clusterpackage package-operator -o yaml | yq .spec.image + +# Confirm manager deployment uses mirrored images +kubectl get -n package-operator-system deployment/package-operator-manager -o yaml | yq '.spec.template.spec.containers[].image' +``` +The override system maintains all original functionality while transparently redirecting image pulls, enabling the enforcement of registry policies without modifying upstream deployment logic. + From 7971c25cd8a25154976a07a6ef50b90e54287039 Mon Sep 17 00:00:00 2001 From: Anthony Vollmer Date: Mon, 5 May 2025 08:11:15 -0400 Subject: [PATCH 2/6] PKO-275 revised comment on review of documentation for Global Image Prefix Override feature --- .../global-prefix-override.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/content/en/docs/advanced_features/global-prefix-override.md b/content/en/docs/advanced_features/global-prefix-override.md index 0d8d0f3..dd48434 100644 --- a/content/en/docs/advanced_features/global-prefix-override.md +++ b/content/en/docs/advanced_features/global-prefix-override.md @@ -3,19 +3,19 @@ title: Global Image Prefix Override weight: 2001 --- -The Global Image Prefix Override feature enables deployment of upstream package-operator artifacts from private registries without rebuilding them in downstream pipelines. This solves the previously impossible scenario where organizations needed to mirror upstream images while maintaining original deployment manifests. +The Global Image Prefix Override feature enables deployment of upstream package-operator artifacts from private registries without rebuilding them in downstream pipelines. This solves the previously impossible scenario where organizations needed to mirror upstream images while maintaining original deployment manifests, without referencing the workload images with PKO's internal image digest feature. ## Implementation -Prefix overrides can be configured through either: +Prefix overrides can be configured through either of these options supplied to the `package-operator-manager` binary: - Environment variable: `PKO_IMAGE_PREFIX_OVERRIDES` - Command-line flag: `image-prefix-overrides` -Both accept mappings in `from=to` format (e.g., `quay.io/my-org/=mirror.example.com/mirror-org/`). When Package Operator encounters an image reference starting with the `from` prefix, it automatically substitutes it with the `to` prefix while preserving the remainder of the image path. +Both accept mappings in `from=to` format (e.g., `quay.io/my-org/=mirror.example.com/mirror-org/`). A comma separataed list in the same format can also be provided for multiple overrides. When Package Operator encounters an image reference starting with the `from` prefix, it automatically substitutes it with the `to` prefix while preserving the remainder of the image path. ## Usage Example -The following `mirror.sh` script demonstrates a complete workflow for deploying Package Operator using mirrored images: +The following `mirror.sh` script demonstrates a the mirroring of the images and the modification of the bootstrap job manifests. (Images for version v1.82.2 are mirrored from the namespace `quary.io/package-operator` into the namespace `quay.io/erdii-test/pko-mirror`). ```bash #!/bin/bash @@ -59,18 +59,21 @@ yq -i 'with( ## Key Components Explained ### Image Mirroring: -The `skopeo copy` commands create identical copies of all upstream images in your private registry, maintaining the original tags and manifests. +The `skopeo copy` commands create identical copies of all upstream PKO package images in your private registry, maintaining the original tags and manifests. ### Prefix Override Injection: -The first `yq` command modifies the PKO_IMAGE_PREFIX_OVERRIDES environment variable to redirect all quay.io/package-operator/ pulls to quay.io/erdii-test/pko-mirror/ +The first `yq` command modifies the PKO_IMAGE_PREFIX_OVERRIDES environment variable to redirect all PKO package image pulls from quay.io/package-operator/ to quay.io/erdii-test/pko-mirror/ +while additionally rewriting the prefixes of the templated image address outputs of the [image digest feature](https://github.com/package-operator/website/blob/904d2c0a90d53e01ed59b602181d47b314925b8b/content/en/docs/guides/image-digests.md). Example template directive: {{.images.someNamedImage}} -The pattern `original-prefix/=new-prefix/` ensures complete path substitution while preserving image names and tags +The pattern `original-prefix/=new-prefix/` ensures complete prefix substitution while preserving image names and tags ### Direct Reference Updates: The second `yq` command handles hardcoded image references that bypass the prefix override mechanism by: Modifying the container's `.image` field Updating the first command-line argument in .args[0] + +This is required as some componenets might directly reference images without respecting the prefix override, so the need to be explicitly changed. This ensures all PKO package image pulls during bootstrap come from your mirror location. # Deployment Process @@ -86,5 +89,3 @@ kubectl get clusterpackage package-operator -o yaml | yq .spec.image # Confirm manager deployment uses mirrored images kubectl get -n package-operator-system deployment/package-operator-manager -o yaml | yq '.spec.template.spec.containers[].image' ``` -The override system maintains all original functionality while transparently redirecting image pulls, enabling the enforcement of registry policies without modifying upstream deployment logic. - From b69126aaf7d0395e8cbde712d219545a586d4917 Mon Sep 17 00:00:00 2001 From: Anthony Vollmer Date: Fri, 22 Aug 2025 14:16:20 -0400 Subject: [PATCH 3/6] docs(global-prefix-override): clarify feature description and fixed random typos --- .../docs/advanced_features/global-prefix-override.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/content/en/docs/advanced_features/global-prefix-override.md b/content/en/docs/advanced_features/global-prefix-override.md index dd48434..61aad9a 100644 --- a/content/en/docs/advanced_features/global-prefix-override.md +++ b/content/en/docs/advanced_features/global-prefix-override.md @@ -3,7 +3,11 @@ title: Global Image Prefix Override weight: 2001 --- -The Global Image Prefix Override feature enables deployment of upstream package-operator artifacts from private registries without rebuilding them in downstream pipelines. This solves the previously impossible scenario where organizations needed to mirror upstream images while maintaining original deployment manifests, without referencing the workload images with PKO's internal image digest feature. +The Global Image Prefix Override feature allows using mirrored package and workload images from a private registry without needing to rebuild the images with new image references. + +This feature solves the problem of mirroring upstream images while preserving their original deployment manifests, all without relying on the Package Operator's internal image digest feature. + +For example, this could enable deployment of upstream package-operator artifacts from private registries without rebuilding them in downstream pipelines. This solves the previously impossible scenario where organizations needed to mirror upstream images while maintaining original deployment manifests, without referencing the workload images with PKO's internal image digest feature. ## Implementation @@ -15,7 +19,7 @@ Both accept mappings in `from=to` format (e.g., `quay.io/my-org/=mirror.example. ## Usage Example -The following `mirror.sh` script demonstrates a the mirroring of the images and the modification of the bootstrap job manifests. (Images for version v1.82.2 are mirrored from the namespace `quary.io/package-operator` into the namespace `quay.io/erdii-test/pko-mirror`). +The following `mirror.sh` script demonstrates a the mirroring of the images and the modification of the bootstrap job manifests. (Images for version v1.82.2 are mirrored from the namespace `quay.io/package-operator` into the namespace `quay.io/erdii-test/pko-mirror`). ```bash #!/bin/bash @@ -73,7 +77,7 @@ The second `yq` command handles hardcoded image references that bypass the prefi Modifying the container's `.image` field Updating the first command-line argument in .args[0] -This is required as some componenets might directly reference images without respecting the prefix override, so the need to be explicitly changed. This ensures all PKO package image pulls during bootstrap come from your mirror location. +This is required as some components might directly reference images without respecting the prefix override, so they need to be explicitly changed. This ensures all PKO package image pulls during bootstrap come from your mirror location. # Deployment Process From 594b9d70d0cd3e5287d10b0380b3947d0b169d34 Mon Sep 17 00:00:00 2001 From: Josh Gwosdz Date: Mon, 8 Sep 2025 14:12:40 +0200 Subject: [PATCH 4/6] Update content/en/docs/advanced_features/global-prefix-override.md --- content/en/docs/advanced_features/global-prefix-override.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/advanced_features/global-prefix-override.md b/content/en/docs/advanced_features/global-prefix-override.md index 61aad9a..1a8bd43 100644 --- a/content/en/docs/advanced_features/global-prefix-override.md +++ b/content/en/docs/advanced_features/global-prefix-override.md @@ -7,7 +7,7 @@ The Global Image Prefix Override feature allows using mirrored package and workl This feature solves the problem of mirroring upstream images while preserving their original deployment manifests, all without relying on the Package Operator's internal image digest feature. -For example, this could enable deployment of upstream package-operator artifacts from private registries without rebuilding them in downstream pipelines. This solves the previously impossible scenario where organizations needed to mirror upstream images while maintaining original deployment manifests, without referencing the workload images with PKO's internal image digest feature. +For example, this enables deployment of upstream package-operator artifacts from private registries without rebuilding them in downstream pipelines. This solves the previously impossible scenario where organizations needed to mirror upstream images, because PKO's internal image digest feature insist on referencing images by their original address. ## Implementation From 66d55e58b104a7783401e8349fa9cd7e6e53c77e Mon Sep 17 00:00:00 2001 From: Josh Gwosdz Date: Mon, 8 Sep 2025 14:12:51 +0200 Subject: [PATCH 5/6] Update content/en/docs/advanced_features/global-prefix-override.md --- content/en/docs/advanced_features/global-prefix-override.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/advanced_features/global-prefix-override.md b/content/en/docs/advanced_features/global-prefix-override.md index 1a8bd43..709ac4e 100644 --- a/content/en/docs/advanced_features/global-prefix-override.md +++ b/content/en/docs/advanced_features/global-prefix-override.md @@ -19,7 +19,7 @@ Both accept mappings in `from=to` format (e.g., `quay.io/my-org/=mirror.example. ## Usage Example -The following `mirror.sh` script demonstrates a the mirroring of the images and the modification of the bootstrap job manifests. (Images for version v1.82.2 are mirrored from the namespace `quay.io/package-operator` into the namespace `quay.io/erdii-test/pko-mirror`). +The following `mirror.sh` script demonstrates the mirroring of the images and the modification of the bootstrap job manifests. (Images for version v1.82.2 are mirrored from the registry namespace `quay.io/package-operator` into the registry namespace `quay.io/erdii-test/pko-mirror`). ```bash #!/bin/bash From 43aaa243b2fdd9f1ada4a63a6158216948065fc3 Mon Sep 17 00:00:00 2001 From: Josh Gwosdz Date: Mon, 8 Sep 2025 14:12:59 +0200 Subject: [PATCH 6/6] Update content/en/docs/advanced_features/global-prefix-override.md --- content/en/docs/advanced_features/global-prefix-override.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/content/en/docs/advanced_features/global-prefix-override.md b/content/en/docs/advanced_features/global-prefix-override.md index 709ac4e..1bf193b 100644 --- a/content/en/docs/advanced_features/global-prefix-override.md +++ b/content/en/docs/advanced_features/global-prefix-override.md @@ -77,7 +77,9 @@ The second `yq` command handles hardcoded image references that bypass the prefi Modifying the container's `.image` field Updating the first command-line argument in .args[0] -This is required as some components might directly reference images without respecting the prefix override, so they need to be explicitly changed. This ensures all PKO package image pulls during bootstrap come from your mirror location. +To ensure that all artifacts required for PKO's bootstrap procedure are pulled correctly , it is required to reference the bootstrap job workload image directly from the mirrored location. + +The package image location also gets replaced for readability reasons. # Deployment Process