Skip to content

Configure JFrog Artifactory as PyPI proxy for hardened runners#1379

Open
mihaimitrea-db wants to merge 12 commits intomainfrom
mihaimitrea-db/jfrog-artifactory-migration
Open

Configure JFrog Artifactory as PyPI proxy for hardened runners#1379
mihaimitrea-db wants to merge 12 commits intomainfrom
mihaimitrea-db/jfrog-artifactory-migration

Conversation

@mihaimitrea-db
Copy link
Copy Markdown
Contributor

@mihaimitrea-db mihaimitrea-db commented Apr 1, 2026 •
edited
Loading

Summary

Hardened GitHub Actions runners block direct access to public registries (PyPI, files.pythonhosted.org). This adds OIDC-based JFrog Artifactory authentication so CI can download Python packages through the db-pypi proxy.

Uses the same jfrog/setup-jfrog-cli action as CLI PR #4875.

What changed

  • New composite action (.github/actions/setup-jfrog-pypi/action.yml): Uses jfrog/setup-jfrog-cli for OIDC authentication (same action + SHA as the CLI repo), then sets UV_INDEX_URL using the token from the action's oidc-token/oidc-user outputs.
  • test.yml: Added id-token: write permissions, JFrog setup step, and uv lock before make dev test.
  • push.yml: Added workflow-level id-token: write permissions (needed for workflow_call to test.yml). Added JFrog setup + uv lock to fmt and check-manifest jobs.
  • Makefile: Added fix-lockfile target. All existing targets are unchanged.

How it works

uv.lock stores full registry URLs per package. Setting UV_INDEX_URL to JFrog causes uv sync --locked to fail (stale lockfile). To work around this, CI runs uv lock after JFrog setup, which rewrites the lockfile with JFrog URLs while keeping the same versions. Then make dev test / make dev fmt work unchanged with --locked. The lockfile rewrite is ephemeral (never committed).

In the fmt job, git checkout -- '*.lock' restores the committed lockfile after formatting, then make fix-lockfile normalizes any accidentally committed proxy URLs. git diff --exit-code catches both unformatted code and dirty lockfiles.

make fix-lockfile

Replaces JFrog proxy URLs with their public PyPI equivalents in all *.lock files. Prevents proxy URLs from being accidentally committed to the repo.

When to use it: If you run uv lock while UV_INDEX_URL points to JFrog (e.g. after sourcing CI env vars), run make fix-lockfile before committing to normalize URLs back to public PyPI.

CI enforcement: The fmt job runs this before git diff --exit-code, so committed proxy URLs fail CI.

Out of scope

  • tagging.yml: Generated from the Universe codegen template (openapi/genkit/sync/workflows/tagging.yml). JFrog setup needs to be upstreamed there.
  • release.yml / release-test.yml: Need special publish runners per the migration guide. Separate follow-up with the security team.
  • Runner migration: Switching to hardened runner labels can be done as a follow-up.
  • Trigger Tests CI failure: Pre-existing IP allowlist issue (DECO-26816).

NO_CHANGELOG=true

@mihaimitrea-db
Configure JFrog Artifactory as PyPI proxy for hardened runners
1198a01 
Hardened GitHub Actions runners block direct access to public registries.
This adds OIDC-based authentication to JFrog Artifactory and routes all
uv package downloads through the db-pypi proxy. A `make fix-lockfile`
target prevents proxy URLs from leaking into the committed lockfile.
@mihaimitrea-db
Fix lockfile sed pattern and restore lockfiles before diff check
6a83a5b 
The JFrog proxy URL structure has /packages/packages/ (the proxy path
plus the original PyPI path), so the sed replacement must map
db-pypi/packages to files.pythonhosted.org (without /packages suffix).

Also, uv lock through JFrog drops size/upload-time metadata from the
lockfile. Restore committed lockfiles with git checkout before the diff
check so only formatting changes and accidentally committed proxy URLs
are caught.
@mihaimitrea-db
Pin uv version in tagging workflow to match other workflows
455f6d2 
The tagging workflow used setup-uv v4.2.0 without a version pin,
installing whatever uv release was latest. Align it with the rest
of the workflows: setup-uv v7.3.0 pinned to uv 0.6.5.
@mihaimitrea-db
Remove tagging.yml changes (generated file, not in P00 scope)
2ec2a40 
tagging.yml is generated from the Universe codegen template and is not
in the critical path for unblocking unit tests. The uv version pin and
JFrog setup need to be upstreamed to the genkit sync template instead.
@mihaimitrea-db
Remove release workflow changes (out of P00 scope)
915a823 
Release workflows need special publish runners and coordination with the
security team per the migration guide. Keep this PR focused on unblocking
CI (tests, fmt, check-manifest).
@mihaimitrea-db
Test transparent proxy routing: remove uv lock, use --frozen
2261e52 
Remove uv lock from CI and use uv sync --frozen instead of --locked.
If the hardened runners transparently route files.pythonhosted.org
through the JFrog proxy, downloads from lockfile URLs should work
without rewriting them.
@mihaimitrea-db
Pass --frozen to uv run to prevent auto-sync re-locking
0cdc22a 
uv run without --frozen triggers auto-sync, which re-locks the lockfile
with JFrog URLs and drops size/upload-time metadata. Pass --frozen to
all uv run calls since we already synced the environment explicitly.
@mihaimitrea-db
Use Makefile targets with flag overrides instead of inline uv commands
1fa0511 
Add UV_SYNC_FLAGS and UV_RUN_FLAGS variables to the Makefile so CI can
pass --frozen without duplicating commands. Local dev defaults remain
unchanged (--locked for sync, no flags for run).
@mihaimitrea-db
Simplify: use UV_FROZEN=1 instead of Makefile overrides
c12c6f6 
Set UV_FROZEN=1 in the composite action so uv natively skips lockfile
re-resolution on all commands. This lets workflows call make dev test
and make dev fmt unchanged -- no Makefile modifications needed beyond
the new fix-lockfile target.
@mihaimitrea-db
Simplify JFrog auth: use setup-jfrog-cli instead of manual curl
535d374 
Replace the manual OIDC token exchange (~20 lines of curl/jq) with the
jfrog/setup-jfrog-cli action, which handles OIDC internally and exposes
the token via step outputs. Same action + SHA as the CLI repo (PR #4875).
@mihaimitrea-db
Fix --locked/--frozen conflict: use UV_LOCK_FLAG in Makefile
01ff01e 
uv rejects --locked and --frozen together. Add UV_LOCK_FLAG variable
(defaults to --locked) that the composite action clears via env var,
letting UV_FROZEN=1 handle lockfile behavior in CI instead.
@mihaimitrea-db
Use uv lock instead of UV_FROZEN to avoid Makefile changes
5381c9d 
Run uv lock after JFrog setup to rewrite the lockfile with JFrog URLs
(same versions, different registry). This lets make dev test and make
dev fmt work unchanged with --locked. In fmt, restore committed
lockfiles before the diff check so only formatting issues and
accidentally committed proxy URLs are caught.
@mihaimitrea-db mihaimitrea-db deployed to test-trigger-is April 7, 2026 07:23 — with GitHub Actions Active
@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 7, 2026

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-py

Inputs:

  • PR number: 1379
  • Commit SHA: 5381c9da913bf07cc5379094f3bb9476090d0f63

Checks will be approved automatically on success.

@mihaimitrea-db mihaimitrea-db mentioned this pull request Apr 7, 2026
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@parthban-db parthban-db Awaiting requested review from parthban-db

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mihaimitrea-db