diff --git a/skills/README.md b/skills/README.md index 4d17d89e..a0d80193 100644 --- a/skills/README.md +++ b/skills/README.md @@ -64,13 +64,40 @@ Why: Paraphrased instructions may diverge from the SKILL.md, creating conflictin ## Available Skills +### Kubernetes Version Management + +Three-phase workflow for adding new Kubernetes versions: + +1. **analyze-kubernetes-release** - Deep analysis of KEPs and CHANGELOG to identify breaking changes, deprecations, and Bottlerocket impacts. Creates go/no-go recommendation. + +2. **add-kubernetes-prerelease** - Package beta/rc Kubernetes sources from EKS-D pre-release artifacts. Creates new kubernetes-{version} and ecr-credential-provider-{version} packages. + +3. **promote-kubernetes-ga** - Promote pre-release packages to GA by updating source URLs and hashes to official EKS-D release artifacts. + +4. **add-kubernetes-variant** - Add a new Kubernetes variant to the bottlerocket repository + +### Research & Documentation + - **fact-find** - Quick lookup of specific facts with citations. Use for concrete questions with definitive answers (e.g., "What partition scheme does Bottlerocket use?") -- **deep-research** - Create educational documents that build understanding progressively. Use for in-depth explanations of systems or features (e.g., "Explain how Bottlerocket's update system works") +- **research-document** - Create educational documents that build understanding progressively. Use for in-depth explanations of systems or features (e.g., "Explain how Bottlerocket's update system works") + +### Package Maintenance + +- **update-package** - Update a kit package to the latest upstream version with checksum verification and build testing + +### Build & Test + - **local-registry** - Start and manage a local OCI registry for development - **build-kit-locally** - Build a kit and publish it to a locally hosted registry for development testing - **build-variant-from-local-kits** - Build a variant using locally published kits for development validation - **test-local-twoliter** - Build and test local changes to twoliter before releasing - **update-twoliter** - Update all repositories to a new Twoliter version +- **launch-bottlerocket-ec2** - Launch Bottlerocket EC2 instances in standalone, ECS, or EKS modes +- **k8s-node-executor** - Execute commands on Bottlerocket K8s nodes via a privileged pod with host namespace access +- **ssm-executor** - Execute commands on any Bottlerocket EC2 instance via SSM (works with ECS, K8s, standalone) + +### Feature Development + - **idea-honing** - Clarify feature ideas through iterative Q&A, recording insights to guide concept development - **propose-feature-concept** - Create a new feature concept document to pitch the idea and explain the problem/solution - **propose-feature-requirements** - Create or update feature requirements specification using EARS notation with examples and appendices @@ -78,6 +105,13 @@ Why: Paraphrased instructions may diverge from the SKILL.md, creating conflictin - **propose-feature-test-plan** - Create a test plan mapping requirements and constraints to unit/integration tests - **propose-implementation-plan** - Create an implementation plan with atomic commits that build toward a complete feature +### Settings Development + +- **create-settings-model** - Define a new Bottlerocket settings model with SettingsModel trait implementation +- **add-settings-to-variant** - Wire an existing settings model into a Bottlerocket variant via settings-plugins +- **test-settings-locally** - Build and test settings SDK changes using local registry and kit builds +- **add-custom-settings** - Full workflow for adding custom settings: create model, wire to variant, test locally + ## Skill Format Each skill is a directory containing a `SKILL.md` file with: diff --git a/skills/add-kubernetes-prerelease/SKILL.md b/skills/add-kubernetes-prerelease/SKILL.md new file mode 100644 index 00000000..24a4c0c3 --- /dev/null +++ b/skills/add-kubernetes-prerelease/SKILL.md @@ -0,0 +1,283 @@ +--- +name: add-kubernetes-prerelease +description: Package Kubernetes pre-release (beta/rc) sources from EKS-D before GA +--- + +# Add Kubernetes Pre-Release + +Package Kubernetes beta/rc versions from EKS-D pre-release artifacts. + +## Purpose + +Creates ecr-credential-provider and kubernetes packages for pre-release versions (beta, rc) using EKS-D pre-release artifacts. +Differs from production packaging by using pre-release URLs and version strings with suffixes. + +## When to Use + +- Testing new Kubernetes versions before GA release +- EKS-D pre-release artifacts are available +- Need to validate breaking changes early + +## Prerequisites + +- Run `analyze-kubernetes-release` skill first +- EKS-D pre-release artifacts published at https://distro.eks.amazonaws.com/kubernetes-{major}-{minor}/releases/ +- Working in bottlerocket-core-kit repository (or forest worktree) +- Previous Kubernetes version packages exist to copy from + +## Tracking + +Create a tracking issue following this pattern: +https://github.com/bottlerocket-os/bottlerocket/issues/4620 + +Checklist items: +- [ ] Use pre-release sources in packages (this skill) +- [ ] Use official GA sources in packages (promote-kubernetes-ga) +- [ ] Add variants for k8s version (bottlerocket repo) + +## Procedure + +### 1. Set Version Variables + +```bash +# Set versions (example: adding 1.NN based on 1.MM) +PREV_VERSION="1.MM" # e.g., "1.34" +NEW_VERSION="1.NN" # e.g., "1.35" +PREV_VER_NODOT="1MM" # e.g., "134" (used in package names) +NEW_VER_NODOT="1NN" # e.g., "135" +PRERELEASE_TAG="beta.0" # or "rc.1", etc. +FULL_VERSION="${NEW_VERSION}.0-${PRERELEASE_TAG}" + +# Verify analysis exists +ANALYSIS="$FOREST_ROOT/planning/k8s-${NEW_VERSION}-analysis.md" +if [ ! -f "$ANALYSIS" ]; then + echo "ERROR: Run analyze-kubernetes-release first" + exit 1 +fi + +cd kits/bottlerocket-core-kit/packages +``` + +### 2. Verify Pre-release Sources Exist + +**Critical:** Verify EKS-D pre-release artifacts are published before making any changes. + +```bash +# Construct pre-release URLs +RELEASE="1" # EKS-D release number, typically "1" for first pre-release +K8S_URL="https://distro.eks.amazonaws.com/kubernetes-${NEW_VERSION}/releases/${RELEASE}/artifacts/kubernetes/v${FULL_VERSION}/kubernetes-src.tar.gz" +ECR_URL="https://distro.eks.amazonaws.com/kubernetes-${NEW_VERSION}/releases/${RELEASE}/artifacts/plugins/v${FULL_VERSION}/eks-distro-ecr-credential-provider-linux-amd64-v${FULL_VERSION}.tar.gz" + +# Verify kubernetes source exists +if ! curl -sfI "$K8S_URL" > /dev/null; then + echo "ERROR: Kubernetes source not found at $K8S_URL" + echo "Check https://distro.eks.amazonaws.com/kubernetes-${NEW_VERSION}/releases/ for available releases" + exit 1 +fi + +# Verify ecr-credential-provider source exists +if ! curl -sfI "$ECR_URL" > /dev/null; then + echo "ERROR: ECR credential provider not found at $ECR_URL" + echo "Check https://distro.eks.amazonaws.com/kubernetes-${NEW_VERSION}/releases/ for available releases" + exit 1 +fi + +echo "✓ Pre-release sources verified" +``` + +### 2. Copy ecr-credential-provider Package + +```bash +cp -r ecr-credential-provider-${PREV_VER_NODOT} ecr-credential-provider-${NEW_VER_NODOT} +cd ecr-credential-provider-${NEW_VER_NODOT} + +# Update Cargo.toml +sed -i "s/ecr-credential-provider-${PREV_VER_NODOT}/ecr-credential-provider-${NEW_VER_NODOT}/g" Cargo.toml +sed -i "s/${PREV_VERSION}/${NEW_VERSION}/g" Cargo.toml + +# Update spec file +mv ecr-credential-provider-${PREV_VER_NODOT}.spec ecr-credential-provider-${NEW_VER_NODOT}.spec +sed -i "s/${PREV_VERSION}/${NEW_VERSION}/g" ecr-credential-provider-${NEW_VER_NODOT}.spec + +cd .. +``` + +### 4. Copy kubernetes Package + +```bash +cp -r kubernetes-${PREV_VER_NODOT} kubernetes-${NEW_VER_NODOT} +cd kubernetes-${NEW_VER_NODOT} + +# Update Cargo.toml +sed -i "s/kubernetes-${PREV_VER_NODOT}/kubernetes-${NEW_VER_NODOT}/g" Cargo.toml +sed -i "s/${PREV_VERSION}/${NEW_VERSION}/g" Cargo.toml + +# Update spec file +mv kubernetes-${PREV_VER_NODOT}.spec kubernetes-${NEW_VER_NODOT}.spec +sed -i "s/${PREV_VERSION}/${NEW_VERSION}/g" kubernetes-${NEW_VER_NODOT}.spec +sed -i "s/${PREV_VER_NODOT}/${NEW_VER_NODOT}/g" kubernetes-${NEW_VER_NODOT}.spec + +# Update template and systemd files +find . -type f \( -name "*.service" -o -name "*.conf" -o -name "*.toml" -o -name "*.env" \) \ + -exec sed -i "s/${PREV_VERSION}/${NEW_VERSION}/g" {} \; + +cd ../.. +``` + +### 5. Apply Breaking Changes from Analysis + +**Critical:** Review the KEP analysis document and apply necessary changes. + +Common changes between Kubernetes versions: +- Removed kubelet flags (check kubelet-exec-start-conf) +- New kubelet config options (check kubelet-config) +- Deprecated features requiring opt-in/opt-out + +Refer to `$FOREST_ROOT/planning/k8s-${NEW_VERSION}-analysis.md` for version-specific changes. + +### 6. Update Source URLs and Release Field + +**Critical:** Pre-release URLs differ from production URLs. + +Format: `https://distro.eks.amazonaws.com/kubernetes-{major}-{minor}/releases/{release}/artifacts/kubernetes/v{version}/...` + +**Version Variables for Pre-release:** + +The pre-release status is indicated in the Release field, NOT in Version. + +**Example spec file structure:** + +```spec +# Version variables - use clean version, pre-release in Release field only +%global gover 1.35.0 +%global rpmver %{gover} + +Name: %{_cross_os}kubernetes +Version: %{rpmver} +Release: 0.beta0%{?dist} + +# Hardcode Source0 with actual pre-release version (will update for GA) +Source0: https://distro.eks.amazonaws.com/kubernetes-1-35/releases/1/artifacts/kubernetes/v1.35.0-beta.0/kubernetes-src.tar.gz +``` + +⚠️ **Why this matters:** +- Version field stays clean (1.35.0) for RPM compatibility +- Release field (0.beta0) sorts before GA release (1) +- Source0 URL contains the actual pre-release tag from upstream + +```bash +# Update kubernetes spec +vim packages/kubernetes-${NEW_VER_NODOT}/kubernetes-${NEW_VER_NODOT}.spec +# Set: %global gover ${NEW_VERSION}.0 +# Set: %global rpmver %{gover} +# Set: Release: 0.beta0%{?dist} +# Hardcode Source0 with full pre-release URL + +# Update ecr-credential-provider Cargo.toml URL +vim packages/ecr-credential-provider-${NEW_VER_NODOT}/Cargo.toml +# Update the url field to point to the new version +``` + +**Critical: Set Pre-release Release Field** + +For BOTH spec files, update the Release field to indicate pre-release: +```spec +Release: 0.beta0%{?dist} +``` + +Or for release candidates: +```spec +Release: 0.rc0%{?dist} +``` + +⚠️ **Why this matters:** RPM version ordering uses Release when Version is equal. +- `0.beta0` sorts before `1` (GA release) +- This ensures clean upgrades: pre-release → GA works correctly +- Using `Release: 1%{?dist}` for pre-release breaks upgrade ordering + +The pattern is: `0.%{?dist}` where prerelease-tag matches the source (beta0, rc0, rc1, etc.) + +**Check for Obsolete Patches** + +Review patches from the previous version: +```bash +# Check if patches still apply or were merged upstream +ls packages/kubernetes-${PREV_VER_NODOT}/*.patch 2>/dev/null +``` + +Remove patches that were merged upstream in the new release. + +### 7. Update Workspace Configuration + +```bash +cd ../.. # Back to core-kit root + +# Edit Cargo.toml to add new packages to workspace members +vim Cargo.toml + +# Add to [workspace] members array: +# "packages/ecr-credential-provider-${NEW_VER_NODOT}", +# "packages/kubernetes-${NEW_VER_NODOT}", +``` + +### 8. Build and Validate + +**Required:** Run the `build-kit-locally` skill to verify packages compile correctly. + +This builds the kit and publishes to the local registry, confirming the new packages work. + +### 9. Commit Changes + +Create separate commits for each package (following existing PR patterns): + +```bash +# Commit 1: ecr-credential-provider +git add packages/ecr-credential-provider-${NEW_VER_NODOT}/ +# Also add Cargo.toml with just the ecr-credential-provider line +git commit -m "packages: add ecr-credential-provider-${NEW_VER_NODOT}" + +# Commit 2: kubernetes +git add packages/kubernetes-${NEW_VER_NODOT}/ Cargo.toml Cargo.lock +git commit -m "packages: add kubernetes-${NEW_VER_NODOT}" +``` + +## Validation + +Success indicators: +- `build-kit-locally` skill completes successfully +- Kit visible in local registry via `brdev registry list` + +## Common Issues + +**404 on pre-release URLs:** +- EKS-D pre-release not published yet +- Check https://distro.eks.amazonaws.com/kubernetes-{major}-{minor}/releases/ for available releases +- Verify release number and pre-release tag in URL + +**Version mismatch errors:** +- Ensure all URLs use full pre-release version (e.g., v1.NN.0-beta.0, not v1.NN) +- Check spec file Version field matches + +**Build failures from removed flags:** +- Check KEP analysis for removed kubelet flags +- Update kubelet-exec-start-conf to remove deprecated flags + +**Wrong Release field for pre-release:** +- Symptom: RPM version conflicts when promoting to GA +- Cause: Using `Release: 1%{?dist}` instead of `Release: 0.beta0%{?dist}` +- Fix: Pre-release packages must use `Release: 0.%{?dist}` to sort before GA + +## Next Steps + +After successful build and commit: +1. **Optional:** Build variant to test end-to-end (`build-variant-from-local-kits` skill) +2. Test thoroughly - pre-release versions may have bugs +3. When GA releases, promote to production (`promote-kubernetes-ga` skill) + +## Related Skills + +This skill is part of the Kubernetes version workflow: +1. `analyze-kubernetes-release` - Deep analysis of KEPs and changes (run first) +2. `add-kubernetes-prerelease` - Package beta/rc for early testing (this skill) +3. `build-kit-locally` - Build and publish kit (included in step 7) +4. `promote-kubernetes-ga` - Promote to GA (run after upstream GA) diff --git a/skills/add-kubernetes-variant/SKILL.md b/skills/add-kubernetes-variant/SKILL.md new file mode 100644 index 00000000..c12f6d2a --- /dev/null +++ b/skills/add-kubernetes-variant/SKILL.md @@ -0,0 +1,182 @@ +--- +name: add-kubernetes-variant +description: Add a new Kubernetes variant to the bottlerocket repository +--- + +# Add Kubernetes Variant + +Add a new Kubernetes variant to the bottlerocket repository that consumes packages from core-kit. + +## Purpose + +After packaging Kubernetes in core-kit (via `add-kubernetes-prerelease`), this skill creates the variant definition in the bottlerocket repo that uses those packages. + +## When to Use + +- After `add-kubernetes-prerelease` has created kubernetes-{version} packages in core-kit +- When adding support for a new Kubernetes minor version +- When creating platform variants (aws, vmware) for a new k8s version + +## Prerequisites + +1. Core-kit with kubernetes-{version} packages published to registry +2. Worktree created for the k8s version: `forester worktree create k8s-{version}` +3. Reference variant exists (e.g., aws-k8s-1.34 when creating aws-k8s-1.35) + +## Variables + +Set these before starting: + +```bash +PREV_VERSION="1.34" # Previous k8s version to copy from +NEW_VERSION="1.35" # New k8s version +PLATFORM="aws" # Platform: aws, vmware, metal +WORKTREE="k8s-1.35" # Worktree name +``` + +## Procedure + +### Step 1: Create Settings-Defaults Directory + +```bash +cd worktrees/${WORKTREE}/bottlerocket + +# Copy from previous version +cp -r sources/settings-defaults/${PLATFORM}-k8s-${PREV_VERSION} sources/settings-defaults/${PLATFORM}-k8s-${NEW_VERSION} + +# Update Cargo.toml package name (replace dots with underscores in version) +PREV_UNDERSCORE=$(echo $PREV_VERSION | tr '.' '_') +NEW_UNDERSCORE=$(echo $NEW_VERSION | tr '.' '_') + +sed -i "s/${PLATFORM}-k8s-${PREV_UNDERSCORE}/${PLATFORM}-k8s-${NEW_UNDERSCORE}/g" sources/settings-defaults/${PLATFORM}-k8s-${NEW_VERSION}/Cargo.toml +``` + +The defaults.d/ symlinks are inherited and don't need modification. + +### Step 2: Create Variant Directory + +```bash +# Copy variant directory +cp -r variants/${PLATFORM}-k8s-${PREV_VERSION} variants/${PLATFORM}-k8s-${NEW_VERSION} + +# Update Cargo.toml +sed -i "s/${PLATFORM}-k8s-${PREV_UNDERSCORE}/${PLATFORM}-k8s-${NEW_UNDERSCORE}/g" variants/${PLATFORM}-k8s-${NEW_VERSION}/Cargo.toml + +# Update kubelet version reference +sed -i "s/kubelet-${PREV_VERSION}/kubelet-${NEW_VERSION}/g" variants/${PLATFORM}-k8s-${NEW_VERSION}/Cargo.toml +``` + +### Step 3: Update sources/Cargo.toml + +Add the new settings-defaults to workspace members: + +```bash +# Find the line with previous version and add new one after it +sed -i "/settings-defaults\/${PLATFORM}-k8s-${PREV_VERSION}"/a\ \"settings-defaults/${PLATFORM}-k8s-${NEW_VERSION}\"," sources/Cargo.toml +``` + +### Step 4: Update Root Cargo.toml + +Add the new variant to workspace members: + +```bash +# Add after previous version entry +sed -i "/variants\/${PLATFORM}-k8s-${PREV_VERSION}",/a\ \"variants/${PLATFORM}-k8s-${NEW_VERSION}\"," Cargo.toml +``` + +### Step 5: Update settings-defaults.spec + +Edit `packages/settings-defaults/settings-defaults.spec`: + +**Add package definition block** (after the ${PREV_VERSION} block): + +```spec +%package ${PLATFORM}-k8s-${NEW_VERSION} +Summary: Settings defaults for the ${PLATFORM}-k8s ${NEW_VERSION} variants +Requires: (%{shrink: + %{_cross_os}variant(${PLATFORM}-k8s-${NEW_VERSION}) or + %{_cross_os}variant(${PLATFORM}-k8s-${NEW_VERSION}-fips) + %{nil}}) +Provides: %{_cross_os}settings-defaults(any) +Provides: %{_cross_os}settings-defaults(${PLATFORM}-k8s-${NEW_VERSION}) +Provides: %{_cross_os}settings-defaults(${PLATFORM}-k8s-${NEW_VERSION}-fips) +Conflicts: %{_cross_os}settings-defaults(any) + +%description ${PLATFORM}-k8s-${NEW_VERSION} +%{summary}. +``` + +**Add %files section** (after the ${PREV_VERSION} %files block): + +```spec +%files ${PLATFORM}-k8s-${NEW_VERSION} +%{_cross_defaultsdir}/${PLATFORM}-k8s-${NEW_VERSION}.toml +%{_cross_tmpfilesdir}/storewolf-defaults-${PLATFORM}-k8s-${NEW_VERSION}.conf +``` + +### Step 6: Update settings-plugins.spec + +Edit `packages/settings-plugins/settings-plugins.spec`: + +**Add Provides lines** to the appropriate package section (find the ${PLATFORM}-k8s package): + +```spec +Provides: %{_cross_os}settings-plugin(${PLATFORM}-k8s-${NEW_VERSION}) +Provides: %{_cross_os}settings-plugin(${PLATFORM}-k8s-${NEW_VERSION}-fips) +``` + +### Step 7: Update README.md + +Add the new variant to the variants list in README.md. + +### Step 8: Regenerate Cargo.lock + +```bash +cargo generate-lockfile +cd sources && cargo generate-lockfile && cd .. +``` + +### Step 9: Commit Changes + +```bash +git add -A +git commit -m "variants: add ${PLATFORM}-k8s-${NEW_VERSION}" +``` + +## Validation + +1. Verify directories exist: + ```bash + ls variants/${PLATFORM}-k8s-${NEW_VERSION}/ + ls sources/settings-defaults/${PLATFORM}-k8s-${NEW_VERSION}/ + ``` + +2. Verify Cargo.toml entries: + ```bash + grep "${PLATFORM}-k8s-${NEW_VERSION}" Cargo.toml + grep "${PLATFORM}-k8s-${NEW_VERSION}" sources/Cargo.toml + ``` + +3. Build the variant (requires local registry with core-kit): + ```bash + cargo make -e BUILDSYS_VARIANT=${PLATFORM}-k8s-${NEW_VERSION} build-variant + ``` + +## Common Issues + +### "package not found" during build +- Ensure core-kit with kubernetes-${NEW_VERSION} is published to local registry +- Check Twoliter.toml points to correct registry + +### Symlink errors +- The defaults.d/ symlinks are relative paths - don't modify them +- They should all point to ../../../shared-defaults/ + +### Cargo.lock conflicts +- Run `cargo generate-lockfile` in both root and sources/ directories + +## Notes + +- FIPS variants share settings-defaults with non-FIPS (no separate directory needed) +- nvidia variants need separate settings-defaults (different containerd config) +- For nvidia variants, also copy from ${PLATFORM}-k8s-${PREV_VERSION}-nvidia diff --git a/skills/analyze-kubernetes-release/SKILL.md b/skills/analyze-kubernetes-release/SKILL.md new file mode 100644 index 00000000..53497634 --- /dev/null +++ b/skills/analyze-kubernetes-release/SKILL.md @@ -0,0 +1,165 @@ +--- +name: analyze-kubernetes-release +description: Analyze a Kubernetes release for Bottlerocket compatibility and required changes +--- + +# Analyze Kubernetes Release + +Analyze a new Kubernetes release to determine compatibility with Bottlerocket and identify required changes. + +## When to Use + +- New Kubernetes minor version released (e.g., 1.32) +- Planning to add support for a new Kubernetes version +- Evaluating breaking changes before implementation + +## Prerequisites + +- Internet access for Kubernetes release notes +- Understanding of Bottlerocket's Kubernetes components (kubelet, kube-proxy, credential provider) + +## Procedure + +### 1. Initialize Analysis Document + +Set the version being analyzed: +```bash +K8S_VERSION="1.32" # e.g., "1.32" for Kubernetes 1.32 +``` + +Create `$FOREST_ROOT/planning/k8s-${K8S_VERSION}-analysis.md` with this template: + +```markdown +# Kubernetes ${K8S_VERSION} Analysis for Bottlerocket + +**Date:** YYYY-MM-DD +**Analyst:** +**Status:** In Progress + +## Summary + +Brief overview of findings. + +## Breaking Changes + +### Kubelet (SIG Node) + +| Change | PR/KEP | Citation | +|--------|--------|----------| +| None identified | | | + +### kube-proxy (SIG Network) + +| Change | PR/KEP | Citation | +|--------|--------|----------| +| None identified | | | + +## Deprecations + +| Feature | Timeline | Citation | Bottlerocket Impact | +|---------|----------|----------|---------------------| +| None identified | | | | + +## KEP Analysis + +### KEP-XXXX: Title + +**Stage:** alpha/beta/GA +**Impact:** High/Medium/Low +**Summary:** {cited from KEP} +**Citation:** {KEP URL or file path} +**Action:** + +## Dependency Updates + +| Component | Version | Citation | Notes | +|-----------|---------|----------|-------| +| Go | | | | +| containerd API | | | | + +## Go/No-Go + +**Recommendation:** GO / NO-GO / CONDITIONAL +**Rationale:** {based on cited facts above} +**Blockers:** {cite specific changes} +``` + +### 2. Download CHANGELOG + +```bash +curl -sL "https://raw.githubusercontent.com/kubernetes/kubernetes/master/CHANGELOG/CHANGELOG-${K8S_VERSION}.md" > /tmp/k8s-${K8S_VERSION}-changelog.md +``` + +### 3. Analyze Changes Using fact-find + +**For each category, use fact-find to get cited answers:** + +``` +USING SKILL "fact-find" + +Question: "What are the Urgent Upgrade Notes in Kubernetes {VERSION} CHANGELOG?" +Question: "What SIG Node changes affect kubelet in Kubernetes {VERSION}?" +Question: "What SIG Network changes affect kube-proxy in Kubernetes {VERSION}?" +Question: "What credential provider changes are in Kubernetes {VERSION}?" +``` + +**Document each finding with its citation** in the analysis document. + +### 4. KEP Deep Dive (If Breaking Changes Found) + +For breaking changes identified in step 3, use fact-find to analyze the KEP: + +``` +USING SKILL "fact-find" + +Question: "What does KEP-XXXX change about kubelet behavior?" +Question: "What configuration changes does KEP-XXXX require?" +``` + +### 5. Check EKS-D Status + +```bash +curl -s "https://api.github.com/repos/aws/eks-distro/releases" | \ + grep -o '"tag_name": "v1-'${K8S_VERSION//./}-'[^"]*"' | head -5 +``` + +### 6. Make Go/No-Go Decision + +**Base decision on cited facts from steps 3-4.** + +**GO:** No blockers identified in cited changes +- No breaking kubelet/kube-proxy changes found, OR +- Breaking changes have simple fixes (cite specific changes) + +**NO-GO:** Blockers exist in cited changes +- Kubelet flag removal requiring template changes (cite KEP/PR) +- Config format changes (cite KEP/PR) +- Dependency issues (cite version requirements) + +**CONDITIONAL:** Minor issues in cited changes +- Deprecation warnings (cite specific deprecations) +- Optional features can be disabled (cite feature gates) + +**⚠️ All recommendations must reference specific citations from the analysis.** + +## Validation + +- [ ] Analysis document created +- [ ] All findings have citations (KEP URLs, PR numbers, CHANGELOG sections) +- [ ] fact-find used for each analysis question +- [ ] Go/No-Go decision references specific cited facts +- [ ] No unsupported claims or synthesis + +## Common Issues + +**Too many results:** Focus on "Breaking" and "Deprecation" in change descriptions. + +**Unclear impact:** Use fact-find to ask "Does {change} affect kubelet configuration?" + +**Missing citations:** Every claim in the analysis must trace to a source. + +## Reference + +- [Kubernetes CHANGELOG](https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG) +- [KEPs](https://github.com/kubernetes/enhancements/tree/master/keps) +- [SIG Node](https://github.com/kubernetes/community/tree/master/sig-node) diff --git a/skills/k8s-node-executor/SKILL.md b/skills/k8s-node-executor/SKILL.md new file mode 100644 index 00000000..ab25fa3a --- /dev/null +++ b/skills/k8s-node-executor/SKILL.md @@ -0,0 +1,240 @@ +--- +name: k8s-node-executor +description: Execute commands on Bottlerocket K8s nodes via a privileged pod with host namespace access +--- + +# Skill: K8s Node Executor + +## Purpose + +Deploy a privileged pod to execute commands directly on Bottlerocket nodes for debugging, testing, and exploration. Returns command output to stdout for scriptable use. + +## When to Use + +- Debugging node-level issues on Bottlerocket K8s nodes +- Inspecting host filesystem, processes, or network +- Running apiclient commands to view/modify Bottlerocket settings +- Container runtime inspection + +## Prerequisites + +- kubectl access to the K8s cluster with Bottlerocket nodes +- Permissions to create privileged pods +- Target node name (optional - defaults to any Linux node) + +## Procedure + +### 1. Deploy the Executor Pod + +**Basic (any node):** +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: br-executor +spec: + hostNetwork: true + hostPID: true + hostIPC: true + containers: + - name: exec + image: amazonlinux:2023 + securityContext: + privileged: true + volumeMounts: + - name: host + mountPath: /host + command: ["sleep", "infinity"] + volumes: + - name: host + hostPath: + path: / + restartPolicy: Never + nodeSelector: + kubernetes.io/os: linux +EOF +``` + +**Target specific node:** +```bash +kubectl apply -f - <<'EOF' +apiVersion: v1 +kind: Pod +metadata: + name: br-executor +spec: + nodeName: NODE_NAME_HERE + hostNetwork: true + hostPID: true + hostIPC: true + containers: + - name: exec + image: amazonlinux:2023 + securityContext: + privileged: true + volumeMounts: + - name: host + mountPath: /host + command: ["sleep", "infinity"] + volumes: + - name: host + hostPath: + path: / + restartPolicy: Never +EOF +``` + +### 2. Wait for Ready + +```bash +kubectl wait --for=condition=Ready pod/br-executor --timeout=60s +``` + +### 3. Execute Commands + +```bash +# Single command +kubectl exec br-executor -- + +# Interactive shell +kubectl exec -it br-executor -- /bin/bash +``` + +### 4. Cleanup + +```bash +kubectl delete pod br-executor +``` + +## Common Commands + +### Bottlerocket Settings (apiclient) + +```bash +# View all settings +kubectl exec br-executor -- /host/usr/bin/apiclient get settings + +# View specific setting +kubectl exec br-executor -- /host/usr/bin/apiclient get settings.kubernetes + +# View OS info +kubectl exec br-executor -- /host/usr/bin/apiclient get os + +# Modify setting +kubectl exec br-executor -- /host/usr/bin/apiclient set motd="Debug session" +``` + +### Host Filesystem + +```bash +# OS release +kubectl exec br-executor -- cat /host/etc/os-release + +# Bottlerocket settings JSON +kubectl exec br-executor -- cat /host/etc/bottlerocket/settings.json + +# List host binaries +kubectl exec br-executor -- ls /host/usr/bin/ +``` + +### System Info + +```bash +# Kernel version +kubectl exec br-executor -- uname -a + +# Memory +kubectl exec br-executor -- free -h + +# Disk +kubectl exec br-executor -- df -h + +# Processes +kubectl exec br-executor -- ps aux + +# Loaded modules +kubectl exec br-executor -- lsmod +``` + +### Networking + +```bash +# Interfaces +kubectl exec br-executor -- ip addr + +# Routes +kubectl exec br-executor -- ip route + +# Listening ports +kubectl exec br-executor -- ss -tlnp + +# iptables +kubectl exec br-executor -- iptables -L -n -v +``` + +### Container Runtime + +```bash +# List containers (k8s namespace) +kubectl exec br-executor -- ctr -n k8s.io containers list + +# List images +kubectl exec br-executor -- ctr -n k8s.io images list + +# Inspect container +kubectl exec br-executor -- ctr -n k8s.io containers info +``` + +### Systemd Services + +```bash +# List services +kubectl exec br-executor -- chroot /host systemctl list-units --type=service + +# Service status +kubectl exec br-executor -- chroot /host systemctl status kubelet +``` + +## Security Warning + +**This pod has full node access.** It can: +- Read/modify any host file +- Access all processes and containers +- Change system configuration +- Affect node stability + +**Best practices:** +- Use only in dev/test environments +- Delete immediately after use +- Never leave running in production + +## Troubleshooting + +### Pod won't start + +```bash +kubectl describe pod br-executor +``` + +Common causes: +- PodSecurityPolicy/PodSecurityStandard blocking privileged pods +- Node selector doesn't match any nodes +- Image pull failure + +### Command not found + +Host binaries need full path: +```bash +# Wrong +kubectl exec br-executor -- apiclient get os + +# Right +kubectl exec br-executor -- /host/usr/bin/apiclient get os +``` + +### Install additional tools + +```bash +kubectl exec br-executor -- yum install -y tcpdump strace bind-utils +``` diff --git a/skills/launch-bottlerocket-ec2/.gitignore b/skills/launch-bottlerocket-ec2/.gitignore new file mode 100644 index 00000000..6c96c16e --- /dev/null +++ b/skills/launch-bottlerocket-ec2/.gitignore @@ -0,0 +1 @@ +launch-config.toml diff --git a/skills/launch-bottlerocket-ec2/SKILL.md b/skills/launch-bottlerocket-ec2/SKILL.md new file mode 100644 index 00000000..51e7074c --- /dev/null +++ b/skills/launch-bottlerocket-ec2/SKILL.md @@ -0,0 +1,126 @@ +--- +name: launch-bottlerocket-ec2 +description: Launch a Bottlerocket EC2 instance for testing with proper user data configuration +--- + +# Launch Bottlerocket EC2 + +Launch a Bottlerocket EC2 instance for testing, with proper user data configuration for EKS or standalone use. + +## When to Use + +- Testing a custom-built Bottlerocket AMI +- Validating settings or configuration changes +- Debugging Bottlerocket behavior on EC2 + +## Prerequisites + +- AWS credentials configured +- VPC with subnet and security group +- IAM instance profile with required permissions +- For EKS: Existing EKS cluster + +## Procedure + +### 1. Gather Required Information + +From user or environment: +- AMI ID (custom build or public AMI) +- Target cluster name (for EKS variants) +- Instance type preference +- Region + +### 2. Verify AWS Access + +```bash +aws sts get-caller-identity +``` + +### 3. Get Network Configuration + +```bash +# List VPCs +aws ec2 describe-vpcs --query 'Vpcs[*].[VpcId,Tags[?Key==`Name`].Value|[0]]' --output table + +# List subnets in VPC +aws ec2 describe-subnets --filters "Name=vpc-id,Values=VPC_ID" \ + --query 'Subnets[*].[SubnetId,AvailabilityZone,Tags[?Key==`Name`].Value|[0]]' --output table + +# List security groups +aws ec2 describe-security-groups --filters "Name=vpc-id,Values=VPC_ID" \ + --query 'SecurityGroups[*].[GroupId,GroupName]' --output table +``` + +### 4. Prepare User Data + +Create `/tmp/userdata.toml` based on variant: + +**For EKS variants (aws-k8s-*):** +```toml +[settings.kubernetes] +api-server = "https://CLUSTER_ENDPOINT" +cluster-name = "CLUSTER_NAME" +cluster-certificate = "BASE64_ENCODED_CA_CERT" +``` + +To get EKS cluster details: +```bash +./scripts/get-eks-details.sh CLUSTER_NAME +``` + +### 5. Launch Instance + +```bash +./scripts/launch-instance.sh AMI_ID USERDATA_FILE +``` + +Returns the instance ID. + +### 6. Wait for Instance + +```bash +./scripts/wait-for-instance.sh INSTANCE_ID +``` + +Waits for instance to be running and displays state and IP. + +## Validation + +### Check Instance Status + +```bash +aws ec2 describe-instance-status --instance-ids INSTANCE_ID --region REGION +``` + +### For EKS: Verify Node Joined + +```bash +kubectl get nodes -l "kubernetes.io/hostname=PRIVATE_DNS" +``` + +### Check SSM Connectivity + +```bash +aws ssm describe-instance-information \ + --filters Key=InstanceIds,Values=INSTANCE_ID \ + --query 'InstanceInformationList[*].[InstanceId,PingStatus]' --output table +``` + +## Cleanup + +```bash +aws ec2 terminate-instances --instance-ids INSTANCE_ID --region REGION +``` + +## Common Issues + +**Instance fails to join cluster:** Check security group allows cluster communication. + +**SSM not connecting:** Verify IAM role has SSM permissions and instance has internet access. + +**User data not applied:** Ensure TOML syntax is valid. Check `/var/log/cloud-init-output.log` via SSM. + +## Reference + +- [Bottlerocket User Data](https://github.com/bottlerocket-os/bottlerocket#using-user-data) +- [EKS Node IAM Role](https://docs.aws.amazon.com/eks/latest/userguide/create-node-role.html) diff --git a/skills/launch-bottlerocket-ec2/scripts/get-eks-details.sh b/skills/launch-bottlerocket-ec2/scripts/get-eks-details.sh new file mode 100755 index 00000000..62b5e412 --- /dev/null +++ b/skills/launch-bottlerocket-ec2/scripts/get-eks-details.sh @@ -0,0 +1,6 @@ +#!/bin/bash +set -euo pipefail +CLUSTER_NAME="${1:?Usage: get-eks-details.sh CLUSTER_NAME}" +aws eks describe-cluster --name "$CLUSTER_NAME" \ + --query 'cluster.{endpoint:endpoint,ca:certificateAuthority.data,name:name}' \ + --output json diff --git a/skills/launch-bottlerocket-ec2/scripts/launch-instance.sh b/skills/launch-bottlerocket-ec2/scripts/launch-instance.sh new file mode 100755 index 00000000..6cdd1f74 --- /dev/null +++ b/skills/launch-bottlerocket-ec2/scripts/launch-instance.sh @@ -0,0 +1,23 @@ +#!/bin/bash +set -euo pipefail +AMI_ID="${1:?Usage: launch-instance.sh AMI_ID USERDATA_FILE}" +USERDATA_FILE="${2:?Usage: launch-instance.sh AMI_ID USERDATA_FILE}" + +: "${REGION:=us-west-2}" +: "${SUBNET_ID:?Set SUBNET_ID environment variable}" +: "${SG_ID:?Set SG_ID environment variable}" +: "${INSTANCE_PROFILE:?Set INSTANCE_PROFILE environment variable}" +: "${INSTANCE_TYPE:=m5.large}" +: "${INSTANCE_NAME:=bottlerocket-test}" + +aws ec2 run-instances \ + --region "$REGION" \ + --image-id "$AMI_ID" \ + --instance-type "$INSTANCE_TYPE" \ + --subnet-id "$SUBNET_ID" \ + --security-group-ids "$SG_ID" \ + --iam-instance-profile "Name=$INSTANCE_PROFILE" \ + --user-data "file://$USERDATA_FILE" \ + --tag-specifications "ResourceType=instance,Tags=[{Key=Name,Value=$INSTANCE_NAME}]" \ + --query 'Instances[0].InstanceId' \ + --output text diff --git a/skills/launch-bottlerocket-ec2/scripts/wait-for-instance.sh b/skills/launch-bottlerocket-ec2/scripts/wait-for-instance.sh new file mode 100755 index 00000000..fc9f1350 --- /dev/null +++ b/skills/launch-bottlerocket-ec2/scripts/wait-for-instance.sh @@ -0,0 +1,11 @@ +#!/bin/bash +set -euo pipefail +INSTANCE_ID="${1:?Usage: wait-for-instance.sh INSTANCE_ID}" +: "${REGION:=us-west-2}" + +echo "Waiting for instance $INSTANCE_ID to be running..." +aws ec2 wait instance-running --instance-ids "$INSTANCE_ID" --region "$REGION" + +aws ec2 describe-instances --instance-ids "$INSTANCE_ID" --region "$REGION" \ + --query 'Reservations[0].Instances[0].{State:State.Name,PrivateIP:PrivateIpAddress,PublicIP:PublicIpAddress}' \ + --output table diff --git a/skills/promote-kubernetes-ga/SKILL.md b/skills/promote-kubernetes-ga/SKILL.md new file mode 100644 index 00000000..f404b74e --- /dev/null +++ b/skills/promote-kubernetes-ga/SKILL.md @@ -0,0 +1,98 @@ +--- +name: promote-kubernetes-ga +description: Promote a pre-release Kubernetes version to GA by updating source URLs and removing pre-release markers +--- + +# Promote Kubernetes to GA + +Promote a pre-release Kubernetes version to GA release in bottlerocket-core-kit. + +## When to Use + +- Kubernetes version moves from pre-release (alpha/beta/rc) to GA +- EKS-D releases the GA version +- Pre-release packages already exist in core-kit + +## Prerequisites + +- Pre-release packages exist (kubernetes-X.Y, ecr-credential-provider-X.Y) +- GA release available from upstream +- EKS-D has released the GA version + +## Procedure + +### 1. Identify Version + +```bash +K8S_VERSION="1.32" +VER_NODOT="${K8S_VERSION//.}" # "132" +``` + +### 2. Navigate to Packages + +```bash +cd kits/bottlerocket-core-kit/packages +``` + +### 3. Update kubernetes Package + +```bash +cd kubernetes-${VER_NODOT} +``` + +**Update Cargo.toml:** +- Change source URLs from pre-release to GA +- Update checksums for new sources +- Remove any pre-release version suffixes + +**Update spec file:** +- Remove `%global prerelease` line if present +- Update `Release:` field: change from `0.pre.N%{?dist}` to `1%{?dist}` for GA +- Verify `Version:` matches GA version + +### 4. Update ecr-credential-provider Package + +```bash +cd ../ecr-credential-provider-${VER_NODOT} +``` + +**Update Cargo.toml:** +- Change source URLs to GA release +- Update checksums + +**Update clarify.toml:** +- Update license hashes if sources changed + +```bash +vim clarify.toml +``` + +### 5. Build and Validate + +Use the `build-kit-locally` skill to build and publish the updated core-kit: + +```bash +# See skills/build-kit-locally/SKILL.md +# This will build both packages and publish to local registry +``` + +## Validation + +Success indicators: +- Both packages build without errors (verified by build-kit-locally skill) +- License checks pass (clarify.toml hashes correct) +- Source downloads succeed (GA URLs valid) +- Kit successfully published to local registry + +## Common Issues + +**Source URL 404:** GA not yet released. Wait for upstream release. + +**Checksum mismatch:** Re-download and recalculate checksums. + +**License hash mismatch:** Source tarball contents changed. Update clarify.toml hashes. + +## Reference + +- [Kubernetes Releases](https://github.com/kubernetes/kubernetes/releases) +- [EKS-D Releases](https://github.com/aws/eks-distro/releases) diff --git a/skills/ssm-executor/SKILL.md b/skills/ssm-executor/SKILL.md new file mode 100644 index 00000000..962122cb --- /dev/null +++ b/skills/ssm-executor/SKILL.md @@ -0,0 +1,126 @@ +--- +name: ssm-executor +description: Execute commands on Bottlerocket EC2 instances via AWS Systems Manager +--- + +# SSM Executor + +Execute commands on Bottlerocket EC2 instances using AWS Systems Manager (SSM), with access to both the control container and host system. + +## When to Use + +- Debugging Bottlerocket instances (ECS, K8s, or standalone) +- Checking system state, logs, or configuration +- Running diagnostic commands +- When kubectl exec is not available or insufficient + +## Prerequisites + +- AWS credentials with SSM permissions +- Instance has SSM agent running (enabled by default in Bottlerocket) +- Instance has IAM role with `AmazonSSMManagedInstanceCore` policy +- Network path to SSM endpoints (internet or VPC endpoints) + +## Procedure + +### 1. Verify SSM Connectivity + +```bash +./scripts/verify-connectivity.sh INSTANCE_ID REGION +``` + +Expected: `Online` status and `Bottlerocket` platform. + +### 2. Execute Commands + +**Simple command (control container context):** +```bash +./scripts/control-container-command.sh INSTANCE_ID REGION "uname -a" +``` + +**Access host rootfs via sheltie (full host access):** +```bash +./scripts/sheltie-command.sh INSTANCE_ID REGION "containerd --version" +``` + +### 3. Understanding the Execution Context + +SSM commands run through a chain of contexts: + +``` +SSM → Control Container → (optional) Admin Container → Sheltie → Host +``` + +- **Control container**: Limited environment, has `apiclient` +- **Admin container**: Interactive shell, accessed via `apiclient exec admin bash` +- **Sheltie**: Direct host access via `apiclient exec admin sheltie -- ` + +## Common Commands + +### Bottlerocket Settings (control container) + +```bash +./scripts/control-container-command.sh INSTANCE_ID REGION "apiclient get settings.kubernetes" +./scripts/control-container-command.sh INSTANCE_ID REGION "apiclient set motd='Debug session'" +./scripts/control-container-command.sh INSTANCE_ID REGION "apiclient get os" +``` + +### Host Binaries (via sheltie) + +```bash +./scripts/sheltie-command.sh INSTANCE_ID REGION "containerd --version" +./scripts/sheltie-command.sh INSTANCE_ID REGION "kubelet --version" +./scripts/sheltie-command.sh INSTANCE_ID REGION "systemctl list-units --type=service" +./scripts/sheltie-command.sh INSTANCE_ID REGION "systemctl status containerd" +``` + +### Filesystem Inspection + +```bash +./scripts/sheltie-command.sh INSTANCE_ID REGION "cat /etc/os-release" +./scripts/sheltie-command.sh INSTANCE_ID REGION "df -h" +./scripts/sheltie-command.sh INSTANCE_ID REGION "free -h" +``` + +### Networking + +```bash +./scripts/sheltie-command.sh INSTANCE_ID REGION "ip addr" +./scripts/sheltie-command.sh INSTANCE_ID REGION "ip route" +./scripts/sheltie-command.sh INSTANCE_ID REGION "ss -tlnp" +``` + +## Comparison with k8s-node-executor + +| Feature | ssm-executor | k8s-node-executor | +|---------|--------------|-------------------| +| Works with | Any EC2 instance | K8s nodes only | +| Requires | SSM connectivity | kubectl access | +| Access level | Full host via sheltie | Host namespaces via pod | +| Best for | ECS, standalone, early boot | K8s-specific debugging | + +## Validation + +- [ ] Instance shows `Online` in SSM +- [ ] Control container commands execute +- [ ] Sheltie commands access host + +## Common Issues + +**Instance not showing in SSM:** +- Check IAM role has SSM permissions +- Verify network path to SSM endpoints +- Instance may need reboot after IAM role attachment + +**Command timeout:** +- Increase timeout in send-command +- Check instance is not overloaded + +**Permission denied:** +- Some commands require sheltie for host access +- Check if admin container is enabled + +## Reference + +- [Bottlerocket Admin Container](https://github.com/bottlerocket-os/bottlerocket#admin-container) +- [AWS SSM Run Command](https://docs.aws.amazon.com/systems-manager/latest/userguide/execute-remote-commands.html) diff --git a/skills/ssm-executor/scripts/control-container-command.sh b/skills/ssm-executor/scripts/control-container-command.sh new file mode 100755 index 00000000..a5b5b563 --- /dev/null +++ b/skills/ssm-executor/scripts/control-container-command.sh @@ -0,0 +1,20 @@ +#!/bin/bash +set -euo pipefail +INSTANCE_ID="${1:?Usage: control-container-command.sh INSTANCE_ID REGION COMMAND}" +REGION="${2:?Usage: control-container-command.sh INSTANCE_ID REGION COMMAND}" +COMMAND="${3:?Usage: control-container-command.sh INSTANCE_ID REGION COMMAND}" + +CMD_ID=$(aws ssm send-command \ + --instance-ids "$INSTANCE_ID" \ + --document-name "AWS-RunShellScript" \ + --parameters "{\"commands\":[\"$COMMAND\"]}" \ + --region "$REGION" \ + --query 'Command.CommandId' \ + --output text) +sleep 3 +aws ssm get-command-invocation \ + --command-id "$CMD_ID" \ + --instance-id "$INSTANCE_ID" \ + --region "$REGION" \ + --query 'StandardOutputContent' \ + --output text diff --git a/skills/ssm-executor/scripts/sheltie-command.sh b/skills/ssm-executor/scripts/sheltie-command.sh new file mode 100755 index 00000000..490f9748 --- /dev/null +++ b/skills/ssm-executor/scripts/sheltie-command.sh @@ -0,0 +1,20 @@ +#!/bin/bash +set -euo pipefail +INSTANCE_ID="${1:?Usage: sheltie-command.sh INSTANCE_ID REGION COMMAND}" +REGION="${2:?Usage: sheltie-command.sh INSTANCE_ID REGION COMMAND}" +COMMAND="${3:?Usage: sheltie-command.sh INSTANCE_ID REGION COMMAND}" + +CMD_ID=$(aws ssm send-command \ + --instance-ids "$INSTANCE_ID" \ + --document-name "AWS-RunShellScript" \ + --parameters "{\"commands\":[\"apiclient exec admin sheltie -- $COMMAND\"]}" \ + --region "$REGION" \ + --query 'Command.CommandId' \ + --output text) +sleep 4 +aws ssm get-command-invocation \ + --command-id "$CMD_ID" \ + --instance-id "$INSTANCE_ID" \ + --region "$REGION" \ + --query 'StandardOutputContent' \ + --output text diff --git a/skills/ssm-executor/scripts/verify-connectivity.sh b/skills/ssm-executor/scripts/verify-connectivity.sh new file mode 100755 index 00000000..e32eb6c4 --- /dev/null +++ b/skills/ssm-executor/scripts/verify-connectivity.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -euo pipefail +INSTANCE_ID="${1:?Usage: verify-connectivity.sh INSTANCE_ID REGION}" +REGION="${2:?Usage: verify-connectivity.sh INSTANCE_ID REGION}" + +aws ssm describe-instance-information \ + --filters "Key=InstanceIds,Values=$INSTANCE_ID" \ + --query 'InstanceInformationList[*].[InstanceId,PingStatus,PlatformName]' \ + --output table --region "$REGION" diff --git a/skills/update-package/SKILL.md b/skills/update-package/SKILL.md new file mode 100644 index 00000000..4777a8ac --- /dev/null +++ b/skills/update-package/SKILL.md @@ -0,0 +1,198 @@ +--- +name: update-package +description: Update an existing package to a new upstream version +--- + +# Update Package + +Update an existing Bottlerocket package to a new upstream version. + +## When to Use + +- Upstream releases a new version of a packaged component +- Security fix requires version bump +- New features needed from upstream + +## Prerequisites + +- Package already exists in a kit +- New upstream version available +- Understanding of package structure (Cargo.toml, spec file) + +## Procedure + +### 1. Identify Package Location + +```bash +# Find the package +find kits/ -type d -name "PACKAGE_NAME" + +# Common locations: +# kits/bottlerocket-core-kit/packages/ +# kits/bottlerocket-kernel-kit/packages/ +``` + +### 2. Check Current Version + +```bash +cd kits//packages/ + +# View current version in Cargo.toml +grep -A5 "external-files" Cargo.toml + +# View current version in spec +grep "^Version:" *.spec +``` + +### 3. Download New Source + +```bash +# Download from upstream +curl -LO + +# If signature exists +curl -LO .sig +``` + +### 4. Calculate Checksums + +```bash +sha512sum +sha512sum .sig # if applicable +``` + +### 5. Verify GPG Signature (if applicable) + +**For GPG-signed packages only** (check if package directory has `gpgkey-*.asc` file): + +```bash +# Import the package's GPG key +gpg --import /path/to/worktree/kits//packages//gpgkey-*.asc + +# Verify the signature +gpg --verify .sig +# or for .asc files: +gpg --verify .asc +``` + +**Expected output:** +``` +gpg: Signature made ... +gpg: Good signature from "Upstream Maintainer " +``` + +**If verification fails:** +- Check if the signing key has changed (see upstream release notes) +- Download the new public key from upstream's trusted source +- Replace `gpgkey-*.asc` in the package directory +- Re-import and verify again + +**For unsigned packages:** +- Skip this step if no `gpgkey-*.asc` file exists in the package directory +- Some packages (like containerd, amazon-ecs-cni-plugins) don't use GPG signatures +- Proceed directly to updating Cargo.toml + +### 6. Update Cargo.toml + +Edit `[[package.metadata.build-package.external-files]]` sections: + +```toml +[[package.metadata.build-package.external-files]] +path = "package-NEW_VERSION.tar.gz" +url = "https://example.com/package-NEW_VERSION.tar.gz" +sha512 = "" + +# If signature file exists: +[[package.metadata.build-package.external-files]] +path = "package-NEW_VERSION.tar.gz.sig" +url = "https://example.com/package-NEW_VERSION.tar.gz.sig" +sha512 = "" +``` + +### 7. Update .spec File + +The format varies by package type: + +**Standard package:** +```spec +Name: %{_cross_os}package-name +Version: NEW_VERSION +Release: 1%{?dist} +``` + +**Kubernetes-versioned package:** +```spec +%global kubever 1.32 +%global k8s_minor 32 +Version: NEW_VERSION +``` + +**Go package:** +```spec +%global goproject github.com/org +%global gorepo repo +%global goimport %{goproject}/%{gorepo} +Version: NEW_VERSION +Source0: https://example.com/package-%{version}.tar.gz +``` + +### 8. Test Build + +Build and test the single package: + +```bash +# From the kit directory (e.g., kits/bottlerocket-core-kit/) +PACKAGE= make twoliter build-package -e BUILDSYS_UPSTREAM_SOURCE_FALLBACK=true +``` + +This builds only the specified package, fetching sources directly from upstream URLs since they're not in the lookaside cache yet. + +**For batch updates:** +- When updating multiple related packages in one PR, validate each with single-package builds (can run in parallel) +- After all packages pass individual builds, do one full kit build using the `build-kit-locally` skill +- Single-package builds are faster for iteration; full kit build provides final validation +### 9. Handle Build Failures + +**Patch application failures:** +- Check if patches are still needed (upstream may have incorporated the fix) +- Update patch files for new source structure +- Remove obsolete patches from spec file + +**Dependency version conflicts:** +- Check if bundled dependencies changed versions +- Update `Provides: bundled()` in spec file + +**New build dependencies:** +- Update `[build-dependencies]` in Cargo.toml +- Update `BuildRequires` in .spec file + +### 10. Create Commit + +After successful build, create a signed commit: + +```bash +git add -A +git commit -s -m "package-name: update to NEW_VERSION" +``` + +## Validation + +- [ ] New source downloads successfully +- [ ] Checksums match downloaded files +- [ ] GPG signature verifies (if applicable) +- [ ] Package builds without errors +- [ ] Version appears correctly in build output + +## Common Issues + +**404 on source URL:** Check upstream for correct download location. + +**Patch conflicts:** Review patches against new source, update or remove as needed. + +**Missing dependencies:** Check upstream release notes for new requirements. + +## Reference + +- Package Cargo.toml for current configuration +- Upstream release notes for changes +- Kit's README for build instructions