Skip to content

Commit b31494d

Browse files
pieternpietern
committed
Add release-build and release-publish workflows
Split the release pipeline into two independent workflows: - release-build: produces all artifacts (binaries, archives, checksums, signed Windows binaries) without publishing - release-publish: takes a build run ID and tag, downloads artifacts, creates GitHub release, builds Docker images, notifies downstream The build workflow also handles snapshot builds on branch pushes, consolidating the snapshot and release build paths. Co-authored-by: Isaac
1 parent f20d01e commit b31494d

File tree

3 files changed

+609
-0
lines changed

3 files changed

+609
-0
lines changed

‎.github/workflows/RELEASE-DESIGN.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Release Pipeline Design
2+
3+
## Goal
4+
5+
Split the release pipeline into two independent workflows:
6+
7+
1. **Build** — produce all release artifacts (binaries, archives, checksums, signed Windows binaries)
8+
2. **Publish** — take artifacts from a previous build and publish them (GitHub release, Docker images, downstream notifications)
9+
10+
A manual inspection/scanning step happens between build and publish.
11+
12+
## Workflows
13+
14+
### `release-build.yml`
15+
16+
Produces release artifacts without publishing anything.
17+
18+
**Triggers:**
19+
- `push` on tags matching `v*`
20+
- `push` on branches `main`, `split-release-workflows` (replaces `release-snapshot.yml`)
21+
- `workflow_dispatch` (for testing)
22+
23+
When triggered by a branch push (not a tag), goreleaser runs in `--snapshot` mode.
24+
This consolidates snapshot and release builds into a single workflow.
25+
The `release-snapshot.yml` workflow can be removed once this is in place.
26+
27+
**Single job: `build`**
28+
29+
Runs on: `ubuntu-latest-deco` (self-hosted Linux)
30+
31+
Steps:
32+
1. Checkout (with full history and tags for goreleaser versioning)
33+
2. Setup JFrog CLI with OIDC
34+
3. Setup Go
35+
4. Download Go modules via JFrog (`jf goc` + `jf go mod download`)
36+
5. Setup Java (for jsign)
37+
6. Download and verify jsign jar
38+
7. Acquire Azure Key Vault access token
39+
8. Run GoReleaser with `--skip=publish,docker`
40+
- Builds linux/darwin/windows for amd64/arm64
41+
- Signs Windows binaries via jsign post-hook
42+
9. Verify Windows binary signatures
43+
10. Upload `dist/` contents as GitHub Actions artifacts:
44+
- `release-archives`: `*.zip`, `*.tar.gz`, `*SHA256SUMS*`
45+
- `release-binaries-linux`: `dist/unix_linux_*/databricks`
46+
- `release-binaries-darwin`: `dist/unix_darwin_*/databricks`
47+
- `release-binaries-windows`: `dist/windows_windows_*/databricks.exe`
48+
49+
The linux binaries are uploaded separately because they're needed to build Docker images in the publish step. GoReleaser does not include raw binaries in archives, so we need them as a separate artifact.
50+
51+
### `release-publish.yml`
52+
53+
Publishes artifacts from a previous build run.
54+
55+
**Triggers:**
56+
- `workflow_dispatch` with inputs:
57+
- `build-run-id` (required): the run ID of a `release-build.yml` run
58+
- `tag` (required): the version tag to release (e.g. `v0.296.0`)
59+
60+
**Jobs:**
61+
62+
#### `create-github-release`
63+
64+
Download archives and checksums from the build run and create a GitHub release.
65+
66+
Steps:
67+
1. Download `release-archives` artifact from the specified build run
68+
2. Create GitHub release for the tag
69+
3. Upload archives and checksums to the release
70+
71+
#### `docker` (needs: `create-github-release`)
72+
73+
Build and push multi-arch Docker images to GHCR.
74+
75+
Steps:
76+
1. Checkout (for Dockerfile and docker/ files)
77+
2. Download `release-binaries-linux` artifact from the build run
78+
3. Login to GHCR
79+
4. Setup QEMU (for cross-platform builds)
80+
5. Setup Docker (pinned version for buildx compatibility)
81+
6. For each arch (amd64, arm64):
82+
- Copy the binary into a temp build context alongside Dockerfile and docker/ files
83+
- Build with `docker buildx build --push`
84+
7. Create and push multi-arch manifest
85+
86+
#### `notify-downstream` (needs: `create-github-release`)
87+
88+
Trigger downstream repository updates. Parallel jobs for:
89+
- `setup-cli` — workflow dispatch to `databricks/setup-cli`
90+
- `homebrew-tap` — workflow dispatch to `databricks/homebrew-tap` (with checksums)
91+
- `vscode-extension` — workflow dispatch to `databricks/databricks-vscode`
92+
93+
#### `pypi-publish` (needs: `create-github-release`)
94+
95+
Build and publish Python wheel to PyPI.
96+
97+
#### `publish-to-winget-pkgs` (needs: `create-github-release`)
98+
99+
Publish to Windows Package Manager.
100+
101+
## GoReleaser Configuration
102+
103+
A single `.goreleaser-release.yaml` replaces both `.goreleaser-unix.yaml` and `.goreleaser-windows.yaml`.
104+
105+
- Two build IDs: `unix` (linux + darwin) and `windows` (with jsign signing hook)
106+
- No `dockers:` or `docker_manifests:` sections (Docker is handled by the publish workflow)
107+
- No `release:` section (GitHub release is handled by the publish workflow)
108+
- No `before.hooks` (module download handled by the workflow)
109+
110+
## Testing approach
111+
112+
While iterating on this branch:
113+
- Both workflows use a dummy binary (`.github/release-test/main.go`) for fast builds
114+
- The build workflow has a `push` trigger for the branch
115+
- Windows signing is exercised on every build
116+
- Docker builds use the real Dockerfile
117+
- The publish workflow can be tested with `workflow_dispatch` pointing at a build run
118+
119+
To go to production:
120+
- Remove `main:` overrides from `.goreleaser-release.yaml` (builds the real CLI)
121+
- Remove the `split-release-workflows` branch from the push trigger
122+
- Delete `.goreleaser-unix.yaml` and `.goreleaser-windows.yaml`
123+
- Replace `release.yml` with `release-build.yml` + `release-publish.yml`
124+
- Replace `release-snapshot.yml` with `release-build.yml` (already handles snapshot mode)
125+
- Delete `release-test.yml`
126+
127+
## Artifact flow
128+
129+
```
130+
release-build.yml (run ID: 12345)
131+
└─ uploads artifacts ─┐
132+
│
133+
[manual inspection] │
134+
│
135+
release-publish.yml │
136+
(input: build-run-id=12345, tag=v0.296.0)
137+
├─ downloads artifacts ┘
138+
├─ creates GitHub release with archives
139+
├─ builds + pushes Docker images from linux binaries
140+
└─ notifies downstream repos
141+
```

‎.github/workflows/release-build.yml‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
name: release-build
2+
3+
on:
4+
push:
5+
tags:
6+
- "v*"
7+
branches:
8+
- "main"
9+
- "split-release-workflows"
10+
11+
workflow_dispatch:
12+
13+
jobs:
14+
build:
15+
environment: sign
16+
runs-on:
17+
group: databricks-deco-testing-runner-group
18+
labels: ubuntu-latest-deco
19+
20+
permissions:
21+
id-token: write
22+
contents: write
23+
24+
steps:
25+
- name: Checkout repository
26+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
27+
with:
28+
fetch-depth: 0
29+
fetch-tags: true
30+
31+
- name: Setup JFrog CLI with OIDC
32+
uses: jfrog/setup-jfrog-cli@279b1f629f43dd5bc658d8361ac4802a7ef8d2d5 # v4.9.1
33+
env:
34+
JF_URL: https://databricks.jfrog.io
35+
with:
36+
oidc-provider-name: github-actions
37+
38+
- name: Setup Go
39+
uses: actions/setup-go@4b73464bb391d4059bd26b0524d20df3927bd417 # v6.3.0
40+
with:
41+
go-version-file: go.mod
42+
cache-dependency-path: |
43+
go.sum
44+
.goreleaser-release.yaml
45+
46+
- name: Download Go modules via JFrog
47+
run: |
48+
jf goc --repo-resolve=db-golang
49+
jf go mod download
50+
51+
- name: Setup Java
52+
uses: actions/setup-java@c1e323688fd81a25caa38c78aa6df2d33d3e20d9 # v4.8.0
53+
with:
54+
distribution: temurin
55+
java-version: '21'
56+
57+
# jsign 7.4 from https://github.com/ebourg/jsign/releases/tag/7.4
58+
- name: Download and verify jsign
59+
run: |
60+
curl -sfL -o "$RUNNER_TEMP/jsign.jar" \
61+
https://github.com/ebourg/jsign/releases/download/7.4/jsign-7.4.jar
62+
echo "2abf2ade9ea322acc2d60c24794eadc465ff9380938fca4c932d09e0b25f1c28 $RUNNER_TEMP/jsign.jar" | sha256sum -c -
63+
echo "JSIGN_JAR=$RUNNER_TEMP/jsign.jar" >> $GITHUB_ENV
64+
65+
- name: Get Azure Key Vault access token
66+
run: |
67+
TOKEN=$(curl -sf -X POST \
68+
"https://login.microsoftonline.com/${{ secrets.DECO_SIGN_AZURE_TENANT_ID }}/oauth2/v2.0/token" \
69+
-d "client_id=${{ secrets.DECO_SIGN_AZURE_CLIENT_ID }}" \
70+
-d "client_secret=${{ secrets.DECO_SIGN_AZURE_CLIENT_SECRET }}" \
71+
-d "scope=https://vault.azure.net/.default" \
72+
-d "grant_type=client_credentials" | jq -r '.access_token')
73+
echo "::add-mask::$TOKEN"
74+
echo "AZURE_VAULT_TOKEN=$TOKEN" >> $GITHUB_ENV
75+
76+
- name: Hide snapshot tag to outsmart GoReleaser
77+
run: git tag -d snapshot || true
78+
79+
# Use --snapshot for branch builds (non-tag refs).
80+
- name: Run GoReleaser
81+
uses: goreleaser/goreleaser-action@ec59f474b9834571250b370d4735c50f8e2d1e29 # v7.0.0
82+
with:
83+
version: ~> v2
84+
args: release -f .goreleaser-release.yaml --skip=publish,docker ${{ !startsWith(github.ref, 'refs/tags/') && '--snapshot' || '' }}
85+
env:
86+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
87+
88+
- name: Verify Windows binary signatures
89+
run: |
90+
for exe in dist/windows_*/databricks.exe; do
91+
echo "=== $exe ==="
92+
java -jar "$JSIGN_JAR" extract --format PEM "$exe"
93+
openssl pkcs7 -in "${exe}.sig.pem" -inform PEM -print_certs -text -noout
94+
rm "${exe}.sig.pem"
95+
echo
96+
done
97+
98+
- name: Upload archives
99+
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
100+
with:
101+
name: release-archives
102+
path: |
103+
dist/*.zip
104+
dist/*.tar.gz
105+
dist/*SHA256SUMS*
106+
107+
# Upload raw linux binaries separately; needed to build Docker images in the publish workflow.
108+
- name: Upload linux binaries
109+
uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0
110+
with:
111+
name: release-binaries-linux
112+
path: dist/unix_linux_*/databricks
113+
114+
# For snapshot builds on main: update the snapshot tag and release.
115+
- name: Update snapshot tag
116+
if: github.ref == 'refs/heads/main'
117+
run: |
118+
git tag snapshot
119+
git push origin snapshot --force
120+
121+
- name: Update snapshot release
122+
if: github.ref == 'refs/heads/main'
123+
uses: softprops/action-gh-release@153bb8e04406b158c6c84fc1615b65b24149a1fe # v2.6.1
124+
with:
125+
name: Snapshot
126+
prerelease: true
127+
tag_name: snapshot
128+
token: ${{ secrets.GITHUB_TOKEN }}
129+
files: |-
130+
dist/databricks_cli_*.zip
131+
dist/databricks_cli_*.tar.gz

0 commit comments

Comments
 (0)