Skip to content

feat(build): migrate content ingestion to hugo modules #647

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
TineoC wants to merge 1 commit into
base: master
Choose a base branch
from
Open

feat(build): migrate content ingestion to hugo modules #647

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ Session.vim
/public/
/resources/

# Hugo modules vendor directory (do not commit external content)
_vendor/

# Temp directory for processing content
_tmp/

Expand Down
6 changes: 6 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ FROM alpine:latest
# "make container-image"
ARG HUGO_VERSION
ARG TARGETARCH
ARG GO_VERSION=1.22.5

RUN apk add --no-cache \
bash \
Expand All @@ -17,6 +18,11 @@ RUN apk add --no-cache \
sed \
npm

RUN curl -sSfL "https://go.dev/dl/go${GO_VERSION}.linux-${TARGETARCH}.tar.gz" -o /tmp/go.tgz \
&& tar -xz -C /usr/local -f /tmp/go.tgz \
&& rm /tmp/go.tgz

ENV PATH="/usr/local/go/bin:${PATH}"
WORKDIR /src

COPY package*.json ./
Expand Down
95 changes: 58 additions & 37 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,28 +17,53 @@ STAGING_IMAGE_REGISTRY := us-central1-docker.pkg.dev/k8s-staging-images
IMAGE_REGISTRY ?= ${STAGING_IMAGE_REGISTRY}/contributor-site
IMAGE_NAME := k8s-contrib-site-hugo
IMAGE_REPO := $(IMAGE_REGISTRY)/$(IMAGE_NAME)
IMAGE_VERSION := $(shell scripts/hash-files.sh Dockerfile Makefile netlify.toml .dockerignore cloudbuild.yaml package.json package-lock.json | cut -c 1-12)
IMAGE_VERSION := $(shell scripts/hash-files.sh Dockerfile Makefile netlify.toml .dockerignore cloudbuild.yaml package.json package-lock.json hugo.yaml go.mod go.sum 2>/dev/null | cut -c 1-12)
COMMIT := $(shell git rev-parse --short HEAD)
CONTAINER_RUN := $(CONTAINER_ENGINE) run --rm -it -v "$(CURDIR):/src"
CONTAINER_RUN_TTY := $(CONTAINER_ENGINE) run --rm -it
CONTAINER_BASE_OPTS := --rm -it --security-opt=no-new-privileges --cap-drop=ALL
ifeq ($(CONTAINER_ENGINE),podman)
# Rootless Podman requires userns mapping and SELinux relabeling
CONTAINER_BASE_OPTS += --userns=keep-id
MOUNT_OPTS := ,relabel=shared
else
# Docker hardening: run as non-root user (assuming UID 1000 matches common dev setups)
# Note: In rootless Podman, keep-id maps the host user to the same UID inside,
# so we don't need (and shouldn't use) --user 1000:1000 there.
CONTAINER_BASE_OPTS += --user 1000:1000
endif

CONTAINER_RUN := $(CONTAINER_ENGINE) run $(CONTAINER_BASE_OPTS) -v "$(CURDIR):/src$(if $(filter podman,$(CONTAINER_ENGINE)),:z)"
CONTAINER_RUN_TTY := $(CONTAINER_ENGINE) run $(CONTAINER_BASE_OPTS)

HUGO_VERSION := $(shell grep ^HUGO_VERSION netlify.toml | tail -n 1 | cut -d '=' -f 2 | tr -d " \"\n")
GIT_TAG ?= v$(HUGO_VERSION)-$(IMAGE_VERSION)
CONTAINER_IMAGE := $(IMAGE_REPO):$(GIT_TAG)

# Docker buildx related settings for multi-arch images
DOCKER_BUILDX ?= docker buildx

# Reuse host Go module cache in container so modules aren't re-downloaded each run (default: $HOME/go/pkg/mod)
Copy link
Copy Markdown
Member

@lmktfy lmktfy Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this have any security implications?

Copy link
Copy Markdown
Contributor Author

@TineoC TineoC Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cache has to be read-write for development. I mitigated this risk by reducing the privileges in the development container in the following commits.

GOMODCACHE_HOST ?= $(HOME)/go/pkg/mod
CONTAINER_HUGO_ENV := -e GOMODCACHE=/tmp/gomod
CONTAINER_HUGO_MOUNTS = \
--read-only \
--mount type=bind,source=$(CURDIR)/.git,target=/src/.git,readonly \
--mount type=bind,source=$(CURDIR)/assets,target=/src/assets,readonly \
--mount type=bind,source=$(CURDIR)/content,target=/src/content,readonly \
--mount type=bind,source=$(CURDIR)/external-sources,target=/src/external-sources,readonly \
--mount type=bind,source=$(CURDIR)/hack,target=/src/hack,readonly \
--mount type=bind,source=$(CURDIR)/layouts,target=/src/layouts,readonly \
--mount type=bind,source=$(CURDIR)/static,target=/src/static,readonly \
--mount type=bind,source=$(CURDIR)/.git,target=/src/.git,readonly$(MOUNT_OPTS) \
--mount type=bind,source=$(CURDIR)/go.mod,target=/src/go.mod$(MOUNT_OPTS) \
--mount type=bind,source=$(CURDIR)/go.sum,target=/src/go.sum$(MOUNT_OPTS) \
--mount type=bind,source=$(GOMODCACHE_HOST),target=/tmp/gomod$(MOUNT_OPTS) \
--mount type=bind,source=$(CURDIR)/assets,target=/src/assets,readonly$(MOUNT_OPTS) \
--mount type=bind,source=$(CURDIR)/content,target=/src/content,readonly$(MOUNT_OPTS) \
--mount type=bind,source=$(CURDIR)/layouts,target=/src/layouts,readonly$(MOUNT_OPTS) \
--mount type=bind,source=$(CURDIR)/static,target=/src/static,readonly$(MOUNT_OPTS) \
--mount type=tmpfs,destination=/tmp,tmpfs-mode=01777 \
--mount type=bind,source=$(CURDIR)/hugo.yaml,target=/src/hugo.yaml,readonly
--mount type=bind,source=$(CURDIR)/hugo.yaml,target=/src/hugo.yaml,readonly$(MOUNT_OPTS)
# Writable mount for container-render output (Hugo writes to /out -> host public/)
CONTAINER_RENDER_MOUNT := --mount type=bind,source=$(CURDIR)/public,target=/out$(MOUNT_OPTS)

# Command to ensure the container image is available locally (pull or build)
IMAGE_ENSURE := $(CONTAINER_ENGINE) pull $(CONTAINER_IMAGE) || $(MAKE) container-image

# Base command for running Hugo in the container with all shared mounts and env vars
CONTAINER_HUGO_RUN := $(CONTAINER_RUN_TTY) $(CONTAINER_HUGO_ENV) $(CONTAINER_HUGO_MOUNTS)

# Fast NONBLOCKING IO to stdout caused by the hack/gen-content.sh script can
# cause Netlify builds to terminate unexpectedly. This forces stdout to block.
Expand All @@ -49,22 +74,27 @@ BLOCK_STDOUT_CMD := python -c "import os,sys,fcntl; \
.DEFAULT_GOAL := help

.PHONY: targets container-targets
targets: help gen-content render server clean clean-all production-build preview-build
container-targets: container-image container-push container-gen-content container-render container-server
targets: help modules-get modules-tidy render server clean clean-all production-build preview-build
container-targets: container-image container-push container-render container-server

help: ## Show this help text.
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'

dependencies:
npm ci

gen-content: ## Generates content from external sources.
hack/gen-content.sh
modules-get: ## Download and update Hugo modules.
hugo mod get -u

modules-tidy: ## Clean up Hugo module dependencies.
hugo mod tidy

render: dependencies ## Build the site using Hugo on the host.

render: dependencies ## Build the site using Hugo on the host. Run 'make modules-get' once if modules are missing.
hugo --logLevel info --ignoreCache --minify

server: dependencies ## Run Hugo locally (if Hugo "extended" is installed locally)

server: dependencies ## Run Hugo locally. Run 'make modules-get' once if modules are missing.
hugo server \
--logLevel info \
--buildDrafts \
Expand Down Expand Up @@ -99,33 +129,24 @@ docker-push: ## Build a multi-architecture image and push that into the registry
$(DOCKER_BUILDX) stop image-builder
rm Dockerfile.cross

docker-gen-content:
@echo -e "**** The use of docker-gen-content is deprecated. Use container-gen-content instead. ****" 1>&2
$(MAKE) container-gen-content

container-gen-content: ## Generates content from external sources within a container (equiv to gen-content).
$(CONTAINER_RUN) $(CONTAINER_IMAGE) hack/gen-content.sh

docker-render:
@echo -e "**** The use of docker-render is deprecated. Use container-render instead. ****" 1>&2
$(MAKE) container-render

container-render: ## Build the site using Hugo within a container (equiv to render).
$(CONTAINER_RUN_TTY) $(CONTAINER_HUGO_MOUNTS) $(CONTAINER_IMAGE) hugo --logLevel info --ignoreCache --minify
@$(IMAGE_ENSURE)
$(CONTAINER_HUGO_RUN) $(CONTAINER_RENDER_MOUNT) $(CONTAINER_IMAGE) bash -c 'cd /src && hugo mod get && hugo --noBuildLock --destination /out --logLevel info --ignoreCache --minify'

docker-server:
@echo -e "**** The use of docker-server is deprecated. Use container-server instead. ****" 1>&2
$(MAKE) container-server

container-server: ## Run Hugo locally within a container, available at http://localhost:1313/
# no build lock to allow for read-only mounts
$(CONTAINER_RUN_TTY) -p 1313:1313 \
$(CONTAINER_HUGO_MOUNTS) \
--cap-drop=ALL \
--cap-drop=AUDIT_WRITE \
$(CONTAINER_IMAGE) \
bash -c 'cd /src && hack/gen-content.sh --in-container && \
cd /tmp/src && \
@$(IMAGE_ENSURE)
$(CONTAINER_HUGO_RUN) -p 1313:1313 $(CONTAINER_IMAGE) \
bash -c 'cd /src && hugo mod get && \
hugo server \
--environment preview \
--logLevel info \
Expand All @@ -139,10 +160,10 @@ container-server: ## Run Hugo locally within a container, available at http://lo
--cleanDestinationDir'

clean: ## Cleans build artifacts.
rm -rf public/ resources/ _tmp/
rm -rf public/ resources/ _tmp/ _vendor/

clean-all: ## Cleans both build artifacts and files synced to content directory
rm -rf public/ resources/ _tmp/
rm -rf public/ resources/ _tmp/ _vendor/
rm -f content/en/events/community-meeting.md
rm -f content/en/events/meet-our-contributors.md
rm -f content/en/events/office-hours.md
Expand Down Expand Up @@ -172,18 +193,18 @@ clean-all: ## Cleans both build artifacts and files synced to content directory
-not -name "code-of-conduct.md" \
-exec rm -rf {} \;

production-build: ## Builds the production site (this command used only by Netlify).
production-build: dependencies ## Builds the production site (this command used only by Netlify).
$(BLOCK_STDOUT_CMD)
hack/gen-content.sh
hugo mod get
hugo \
--environment production \
--logLevel info \
--ignoreCache \
--minify

preview-build: ## Builds a deploy preview of the site (this command used only by Netlify).
preview-build: dependencies ## Builds a deploy preview of the site (this command used only by Netlify).
$(BLOCK_STDOUT_CMD)
hack/gen-content.sh
hugo mod get
hugo \
--environment preview \
--logLevel info \
Expand Down
43 changes: 15 additions & 28 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,15 @@ found within the [`content`][ct] directory. To update the site's content,
make changes to the Markdown sources and [submit a pull request][pr] to this
repository.

Some content is externally sourced and changes to that must be made in the
original location. A list of sources and their locations within the
[`content`][ct] is available below:
Some content is externally sourced via Hugo Modules and changes to that must be made in the
original location. The site uses Hugo Modules to automatically pull content from:

### External sources
- **github.com/kubernetes/community** - Contributor guides, community resources, and communication guidelines
- **github.com/kubernetes/sig-release** - Release information
- **github.com/cncf/foundation** - CNCF Code of Conduct

- **Source:** https://git.k8s.io/community/contributors/guide <br>
**Destination:** `/guide`
- **Source:** https://github.com/cncf/foundation/blob/master/code-of-conduct.md <br>
**Destination:** `/code-of-conduct.md`
- **Source:** https://git.k8s.io/sig-release/releases/release-1.18/README.md <br>
**Destination:** `/release.md`
Content is mapped from these repositories to the site's content directory using Hugo Module mounts
configured in `hugo.yaml`. See the `module.mounts` section for the complete mapping configuration.

## Running the site locally

Expand All @@ -42,9 +39,8 @@ The easiest and most cross-system-compatible way to run the Contributor
Site is to use [Docker][docker]. To begin, create the docker image to be used
with generating the site by executing `make container-image`.

To ensure you can view the site with externally sourced content, run
`make container-gen-content` before previewing the site by with
`make container-server`.
Hugo will automatically download modules when you run `make container-server`.
If you need to update modules manually, run `make modules-get` or `make modules-download`.

**NOTE to Apple Silicon Mac Users**

Expand Down Expand Up @@ -72,23 +68,14 @@ To fetch docsy and its requirements, run the command:
git submodule update --init --recursive --depth 1
```

To ensure you can view the site with externally sourced content, run
`make gen-content` before previewing the site by with `make server`.
Hugo will automatically download modules when you run `make server`.
If you need to update modules manually, run `make modules-get` or `make modules-download`.

**NOTE to MacOS Users**
**Hugo Module Commands**

The `hack/gen-content.sh` script requires the gnu version
of base packages such as `find`, `grep`, and `sed`.

```
brew install coreutils findutils grep gnu-sed gnu-tar make readlink
```

You will then need to update your path to include these:

```
export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
```
- `make modules-get` - Download and update Hugo modules to latest versions
- `make modules-tidy` - Clean up Hugo module dependencies
- `make modules-download` - Download Hugo modules (also runs automatically during build)

## Community, discussion, contribution, and support

Expand Down
2 changes: 1 addition & 1 deletion content/en/blog/2020/contributing-to-the-development-guide/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ My fears were unfounded. Immediately, I felt welcome. I like to think this isn't
working on a much needed task, but rather because the Kubernetes community is filled
with friendly, welcoming people. During the weekly SIG ContribEx meetings, our reports on progress
with the Development Guide were included immediately. In addition, the leader of the meeting would
always stress that the [Kubernetes Code of Conduct]({{< ref "/community/code-of-conduct" >}}) was in
always stress that the [Kubernetes Code of Conduct](/community/code-of-conduct/) was in
effect, and that we should, like Bill and Ted, be excellent to each other.

## This Doesn't Mean It's All Easy
Expand Down
14 changes: 14 additions & 0 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
module github.com/kubernetes-sigs/contributor-site

go 1.22.4

toolchain go1.22.5

// Dependencies are pinned to specific versions (commit hashes) to ensure build
// reproducibility and stability. They are NOT updated automatically on every
// upstream commit. To update content, run 'make modules-get'.
require (
github.com/cncf/foundation v0.0.0-20260213115547-899b0de7f20d // indirect
github.com/kubernetes/sig-release v0.0.0-20260213154600-4664fda9dd9b // indirect
k8s.io/community v0.0.0-20260214141759-4bea83be3fe0 // indirect
)
12 changes: 12 additions & 0 deletions go.sum
View file
Open in desktop
Copy link
Copy Markdown
Member

@lmktfy lmktfy Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Under what circumstances will these values change?

Copy link
Copy Markdown
Contributor Author

@TineoC TineoC Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With these changes... when someone manually runs hugo mod get. This isn't ideal. But right now, we run the latest changes always from every project, so we also have some security/stability risks because we also run the risk of breaking the site every time there's new content in every other repo and page.

I would like to know what you guys like most:

  1. Ensuring scheduled workflows to update content (weekly, daily) to open a PR so we could have traceability of which new content broke the contributor website, and if, by any reason, any of our external sources broke, we could have better guidance for remediation.
  2. Only update external sources modules on PR merged (the downside is that new content is dependent on how often we merge to master)

Copy link
Copy Markdown
Contributor Author

@TineoC TineoC Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example something like .github/workflows/update-modules.yml

    1 name: Update Hugo Modules
    2 on:
    3   schedule:
    4     - cron: '0 0 * * *' # Daily at midnight
    5   workflow_dispatch: # Allow manual trigger
    6
    7 jobs:
    8   update:
    9     runs-on: ubuntu-latest
   10     steps:
   11       - uses: actions/checkout@v4
   12       - name: Setup Hugo
   13         uses: peaceiris/actions-hugo@v3
   14         with:
   15           hugo-version: 'latest'
   16           extended: true
   17       - name: Update Modules
   18         run: |
   19           hugo mod get -u
   20           hugo mod tidy
   21       - name: Create Pull Request
   22         uses: peter-evans/create-pull-request@v6
   23         with:
   24           commit-message: "chore: update hugo modules"
   25           title: "Update external content modules"
   26           body: "This PR updates the Hugo modules to pull the latest content from upstream repositories."
   27           branch: "automation/update-modules"

Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
github.com/cncf/foundation v0.0.0-20260202205210-94f5c56dccf4 h1:BWKfIbicTFE8C2QgQS4uTtvcfc97qWVeGZUE9mOC+t8=
github.com/cncf/foundation v0.0.0-20260202205210-94f5c56dccf4/go.mod h1:Bc+TWRzbCv6T+tkQqNuwGA3cGk017o2TE5CAmu9EY/Q=
github.com/cncf/foundation v0.0.0-20260213115547-899b0de7f20d h1:b9WxNhDNhAvGHn677OkjIlSTkl+b0N2O1x+5CCjxhc4=
github.com/cncf/foundation v0.0.0-20260213115547-899b0de7f20d/go.mod h1:Bc+TWRzbCv6T+tkQqNuwGA3cGk017o2TE5CAmu9EY/Q=
github.com/kubernetes/sig-release v0.0.0-20260202214826-624822f8822c h1:l7+EGkMqg/SBK2bY28ZgnuHW0PfLHr2L3TIAE3O8D0o=
github.com/kubernetes/sig-release v0.0.0-20260202214826-624822f8822c/go.mod h1:sNxEc7U86IjC93kZywZHM8psA/Moy8DMXXwP2N+EAe0=
github.com/kubernetes/sig-release v0.0.0-20260213154600-4664fda9dd9b h1:Kd2CNI54Ny0QuWiiEPws3FAysot+QRIbdVt9zrDW2tA=
github.com/kubernetes/sig-release v0.0.0-20260213154600-4664fda9dd9b/go.mod h1:sNxEc7U86IjC93kZywZHM8psA/Moy8DMXXwP2N+EAe0=
k8s.io/community v0.0.0-20260203101630-a24805997f22 h1:f+gIgCWlPdh6Gvxsm9muQHR12ZSaGThLVzlgf5aShqA=
k8s.io/community v0.0.0-20260203101630-a24805997f22/go.mod h1:xtSUbAS2+b+VSyXBArm0L8NyzxCRQnrZj1PrFq2v1Zw=
k8s.io/community v0.0.0-20260214141759-4bea83be3fe0 h1:HlCv+8PcjEC9K/ytuYfTp/ZJagIFnwl1xEbuLUWU3FI=
k8s.io/community v0.0.0-20260214141759-4bea83be3fe0/go.mod h1:xtSUbAS2+b+VSyXBArm0L8NyzxCRQnrZj1PrFq2v1Zw=
Loading