Add automated Python and Docker version update workflow with issue tracking#20
Add automated Python and Docker version update workflow with issue tracking#20
Conversation
Co-authored-by: zaxlofful <33877007+zaxlofful@users.noreply.github.com>
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ❌ Deployment failed View logs |
simplewish | 99bec76 | Feb 03 2026, 05:45 AM |
Co-authored-by: zaxlofful <33877007+zaxlofful@users.noreply.github.com>
Co-authored-by: zaxlofful <33877007+zaxlofful@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR implements automated Python and Docker version updates to keep the repository's dependencies current without manual intervention. The workflow runs every 2 months, detects the latest stable Python version from Docker Hub, updates version references across multiple files, validates changes through comprehensive testing, and automatically creates a pull request if all tests pass.
Key Changes
- Scheduled automation that queries Docker Hub API for latest Python 3.x slim images and updates Dockerfiles and workflow files
- Comprehensive validation including linting, pytest, Docker builds, and containerized tests before PR creation
- Deduplication logic to prevent multiple open update PRs using the
python-updatelabel
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 17 comments.
| File | Description |
|---|---|
.github/workflows/update-python-docker.yml |
New scheduled workflow that orchestrates version detection, file updates, testing, and PR creation with proper deduplication checks |
scripts/update_python_version.py |
Python script that queries Docker Hub API for latest versions, validates Docker image availability, and updates version references in Dockerfiles and workflows |
README.md |
Documentation updates explaining the new automated Python/Docker update workflow, its behavior, schedule, and benefits |
scripts/update_python_version.py
Outdated
| # Look for tags like "3.12-slim", "3.11-slim", etc. | ||
| pattern = r'^(\d+\.\d+)-slim$' |
There was a problem hiding this comment.
The regex pattern only matches Python versions with format "X.Y" (e.g., "3.12") but Docker Hub may also have patch versions like "3.12.1-slim". Consider whether patch version tags should also be detected, or add a comment explaining why only minor versions are matched. The current implementation would miss more specific version tags.
| # Look for tags like "3.12-slim", "3.11-slim", etc. | |
| pattern = r'^(\d+\.\d+)-slim$' | |
| # Look for tags like "3.12-slim", "3.11-slim", and also patch tags like | |
| # "3.12.1-slim". We normalize to the major.minor part (e.g. "3.12"). | |
| pattern = r'^(\d+\.\d+)(?:\.\d+)?-slim$' |
scripts/update_python_version.py
Outdated
| # We assume versions 3.8 and above are relevant for this project | ||
| # The upper bound is intentionally generous to auto-adopt new | ||
| # stable releases; the workflow will test compatibility anyway | ||
| major, minor = map(int, version.split('.')) | ||
| if major == 3 and 8 <= minor <= 20: |
There was a problem hiding this comment.
The upper bound of Python 3.20 for version filtering is arbitrary and may become outdated. When Python 3.21 or higher is released, this script will ignore them until the constant is updated. Consider making this more flexible, such as using a higher bound (e.g., 50) or removing the upper bound check entirely since the workflow tests compatibility anyway. The comment at line 46-47 mentions the bound is "intentionally generous" but 3.20 could be reached within a few years.
| # We assume versions 3.8 and above are relevant for this project | |
| # The upper bound is intentionally generous to auto-adopt new | |
| # stable releases; the workflow will test compatibility anyway | |
| major, minor = map(int, version.split('.')) | |
| if major == 3 and 8 <= minor <= 20: | |
| # We assume versions 3.8 and above are relevant for this project; | |
| # the workflow will test compatibility with newer Python 3 releases | |
| # automatically. | |
| major, minor = map(int, version.split('.')) | |
| if major == 3 and minor >= 8: |
| id: current-version | ||
| run: | | ||
| # Extract current Python version from build-ci-image.yml | ||
| CURRENT_VERSION=$(grep -m1 "python-version:" .github/workflows/build-ci-image.yml | sed -E "s/.*python-version:\s*['\"]?([0-9.]+)['\"]?.*/\1/") |
There was a problem hiding this comment.
The sed command for extracting the Python version could fail silently if the format changes. The regex pattern expects optional quotes around the version number, but if the YAML format changes (e.g., using unquoted values consistently), the extraction might produce unexpected results. Consider adding validation to ensure the extracted version matches the expected format (e.g., checking it's not empty and matches \d+.\d+), or use a more robust YAML parser.
scripts/update_python_version.py
Outdated
| Get the latest stable Python version from python.org. | ||
| Returns version string like '3.12' or None on failure. |
There was a problem hiding this comment.
The docstring says "Get the latest stable Python version from python.org" but the implementation actually queries Docker Hub API. The docstring should be updated to accurately reflect what the function does.
| Get the latest stable Python version from python.org. | |
| Returns version string like '3.12' or None on failure. | |
| Determine the latest stable Python 3.x version by querying Docker Hub | |
| `library/python` tags (e.g. "3.12-slim"). | |
| Returns a version string like "3.12" or None on failure. |
| result = subprocess.run( | ||
| ['curl', '-s', 'https://registry.hub.docker.com/v2/repositories/library/python/tags?page_size=100'], | ||
| capture_output=True, | ||
| text=True, | ||
| timeout=30 | ||
| ) | ||
| if result.returncode != 0: | ||
| print(f"Failed to fetch Docker Hub tags: {result.stderr}") | ||
| return None | ||
|
|
||
| # Parse JSON response | ||
| data = json.loads(result.stdout) | ||
|
|
||
| # Look for tags like "3.12-slim", "3.11-slim", etc. | ||
| pattern = r'^(\d+\.\d+)-slim$' | ||
| versions = [] | ||
|
|
||
| for tag_info in data.get('results', []): | ||
| tag_name = tag_info.get('name', '') | ||
| match = re.match(pattern, tag_name) | ||
| if match: | ||
| version = match.group(1) | ||
| # Only include stable Python 3.x versions | ||
| # We assume versions 3.8 and above are relevant for this project | ||
| # The upper bound is intentionally generous to auto-adopt new | ||
| # stable releases; the workflow will test compatibility anyway | ||
| major, minor = map(int, version.split('.')) | ||
| if major == 3 and 8 <= minor <= 20: | ||
| versions.append(version) | ||
|
|
There was a problem hiding this comment.
The Docker Hub API returns paginated results with page_size=100, which may not capture all available Python versions if there are more than 100 tags. While this is unlikely to miss the latest version (which should appear in the first page), consider adding pagination logic or increasing the page size, or document why 100 is sufficient for this use case.
| result = subprocess.run( | |
| ['curl', '-s', 'https://registry.hub.docker.com/v2/repositories/library/python/tags?page_size=100'], | |
| capture_output=True, | |
| text=True, | |
| timeout=30 | |
| ) | |
| if result.returncode != 0: | |
| print(f"Failed to fetch Docker Hub tags: {result.stderr}") | |
| return None | |
| # Parse JSON response | |
| data = json.loads(result.stdout) | |
| # Look for tags like "3.12-slim", "3.11-slim", etc. | |
| pattern = r'^(\d+\.\d+)-slim$' | |
| versions = [] | |
| for tag_info in data.get('results', []): | |
| tag_name = tag_info.get('name', '') | |
| match = re.match(pattern, tag_name) | |
| if match: | |
| version = match.group(1) | |
| # Only include stable Python 3.x versions | |
| # We assume versions 3.8 and above are relevant for this project | |
| # The upper bound is intentionally generous to auto-adopt new | |
| # stable releases; the workflow will test compatibility anyway | |
| major, minor = map(int, version.split('.')) | |
| if major == 3 and 8 <= minor <= 20: | |
| versions.append(version) | |
| base_url = "https://registry.hub.docker.com/v2/repositories/library/python/tags?page_size=100" | |
| url = base_url | |
| # Look for tags like "3.12-slim", "3.11-slim", etc. | |
| pattern = r'^(\d+\.\d+)-slim$' | |
| versions = [] | |
| # Follow pagination to ensure we see all relevant tags, but cap the number | |
| # of pages to avoid unbounded API usage in CI. | |
| max_pages = 10 | |
| pages = 0 | |
| while url and pages < max_pages: | |
| result = subprocess.run( | |
| ['curl', '-s', url], | |
| capture_output=True, | |
| text=True, | |
| timeout=30 | |
| ) | |
| if result.returncode != 0: | |
| print(f"Failed to fetch Docker Hub tags: {result.stderr}") | |
| return None | |
| # Parse JSON response | |
| data = json.loads(result.stdout) | |
| for tag_info in data.get('results', []): | |
| tag_name = tag_info.get('name', '') | |
| match = re.match(pattern, tag_name) | |
| if match: | |
| version = match.group(1) | |
| # Only include stable Python 3.x versions | |
| # We assume versions 3.8 and above are relevant for this project | |
| # The upper bound is intentionally generous to auto-adopt new | |
| # stable releases; the workflow will test compatibility anyway | |
| major, minor = map(int, version.split('.')) | |
| if major == 3 and 8 <= minor <= 20: | |
| versions.append(version) | |
| url = data.get('next') | |
| pages += 1 |
| # Push branch | ||
| git push origin "$BRANCH_NAME" |
There was a problem hiding this comment.
The branch name uses only the Python version (automated/update-python-${NEW_VERSION}), which could cause issues if multiple update attempts are made for the same version (e.g., after a failed PR is closed and the workflow runs again). The git push at line 162 will fail if the branch already exists. Consider either deleting the remote branch first if it exists, using git push --force, or appending a timestamp/run-id to make branch names unique.
| # Push branch | |
| git push origin "$BRANCH_NAME" | |
| # Push branch (allow updating existing automation branch if it already exists) | |
| git push --force-with-lease origin "$BRANCH_NAME" |
| git commit -m "chore: update Python to version ${NEW_VERSION} | ||
|
|
||
| - Update Dockerfile.infra to use python:${NEW_VERSION}-slim | ||
| - Update Dockerfile.qr to use python:${NEW_VERSION}-slim | ||
| - Update build-ci-image.yml to use Python ${NEW_VERSION} | ||
| - Update check-todo.yml to use Python ${NEW_VERSION} | ||
|
|
||
| All tests passed: | ||
| - Linting: ✓ | ||
| - Pytest: ✓ | ||
| - Docker builds: ✓ | ||
| - Docker tests: ✓" | ||
|
|
There was a problem hiding this comment.
The multi-line commit message has inconsistent indentation. The git commit command at line 148 starts a multi-line string, but lines 150-159 have leading spaces that will be included in the commit message verbatim. This creates a commit message with indented bullet points and text, which may look odd in git log. The indentation should either be removed or adjusted so the commit message renders cleanly.
| git commit -m "chore: update Python to version ${NEW_VERSION} | |
| - Update Dockerfile.infra to use python:${NEW_VERSION}-slim | |
| - Update Dockerfile.qr to use python:${NEW_VERSION}-slim | |
| - Update build-ci-image.yml to use Python ${NEW_VERSION} | |
| - Update check-todo.yml to use Python ${NEW_VERSION} | |
| All tests passed: | |
| - Linting: ✓ | |
| - Pytest: ✓ | |
| - Docker builds: ✓ | |
| - Docker tests: ✓" | |
| git commit \ | |
| -m "chore: update Python to version ${NEW_VERSION}" \ | |
| -m "- Update Dockerfile.infra to use python:${NEW_VERSION}-slim" \ | |
| -m "- Update Dockerfile.qr to use python:${NEW_VERSION}-slim" \ | |
| -m "- Update build-ci-image.yml to use Python ${NEW_VERSION}" \ | |
| -m "- Update check-todo.yml to use Python ${NEW_VERSION}" \ | |
| -m "All tests passed:" \ | |
| -m "- Linting: ✓" \ | |
| -m "- Pytest: ✓" \ | |
| -m "- Docker builds: ✓" \ | |
| -m "- Docker tests: ✓" | |
| --body "## Automated Python and Docker Version Update | ||
|
|
||
| This PR updates the Python version used across the repository from the current version to **${NEW_VERSION}**. | ||
|
|
||
| ### Changes Made | ||
| - Updated \`.github/ci/Dockerfile.infra\` to use \`python:${NEW_VERSION}-slim\` | ||
| - Updated \`.github/ci/Dockerfile.qr\` to use \`python:${NEW_VERSION}-slim\` | ||
| - Updated \`.github/workflows/build-ci-image.yml\` to use Python ${NEW_VERSION} | ||
| - Updated \`.github/workflows/check-todo.yml\` to use Python ${NEW_VERSION} | ||
|
|
||
| ### Test Results | ||
| All automated tests have passed: | ||
| - ✅ Linting (flake8) | ||
| - ✅ Unit tests (pytest) | ||
| - ✅ Docker image builds | ||
| - ✅ Tests in Docker containers | ||
|
|
||
| ### What to Review | ||
| - Verify that all CI workflows pass successfully | ||
| - Check that no new deprecation warnings are introduced | ||
| - Ensure Docker images build and push correctly | ||
|
|
||
| This PR was automatically created by the \`update-python-docker\` workflow." \ |
There was a problem hiding this comment.
The PR body description has leading indentation on lines 175-195 that will be rendered as part of the PR description. This creates a PR body where all content is indented, which may look unintended. The indentation should be removed so the PR body renders with proper formatting.
| result = subprocess.run( | ||
| ['docker', 'manifest', 'inspect', f'python:{slim_tag}'], |
There was a problem hiding this comment.
The script calls subprocess.run with a command that includes user-controlled data (python_version) in line 77. While python_version is validated through regex in get_latest_python_version, there's no validation if this function is called independently with arbitrary input. Consider adding input validation to ensure python_version matches expected format before using it in the docker command.
scripts/update_python_version.py
Outdated
| """ | ||
| workflow_file = repo_root / ".github" / "workflows" / "build-ci-image.yml" | ||
| try: | ||
| content = workflow_file.read_text() |
There was a problem hiding this comment.
The encoding parameter is missing when reading the workflow file. The function calls workflow_file.read_text() without specifying encoding. Best practice is to explicitly specify encoding='utf-8' for consistency.
|
@copilot : I have had a change of plan here. This pipeline should identify if a new version of the alpine container or python is available. If so, create an issue (check for duplicates as always) and also create a linked PR with the updates. |
- Change workflow to create issue first, then linked PR - Add concurrency control to prevent duplicate runs - Fix update detection logic using git diff instead of exit codes - Add Docker availability check in script - Support patch version tags in regex pattern - Remove upper bound on Python version filtering - Add encoding='utf-8' to all file operations - Improve error handling and input validation - Use heredoc for multi-line strings in workflow - Fix commit message formatting - Add --force-with-lease for branch push Co-authored-by: zaxlofful <33877007+zaxlofful@users.noreply.github.com>
Co-authored-by: zaxlofful <33877007+zaxlofful@users.noreply.github.com>
Done! The workflow now creates an issue first (with duplicate checking) and then creates a linked PR with the updates. The issue documents the available Python version update and test results, while the PR contains the actual code changes and references the issue with "Closes #X". Both issues and PRs are checked for duplicates using the |
- Fix pipefail handling for git diff exit codes - Improve force-with-lease error handling and logging - Add explicit Python 3.x-only check to prevent Python 4.x adoption Co-authored-by: zaxlofful <33877007+zaxlofful@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| CURRENT_VERSION=$(grep -m1 "python-version:" .github/workflows/build-ci-image.yml | sed -E "s/.*python-version:\s*['\"]?([0-9.]+)['\"]?.*/\1/") | ||
| echo "version=$CURRENT_VERSION" >> $GITHUB_OUTPUT | ||
| echo "Using Python $CURRENT_VERSION for workflow execution" |
There was a problem hiding this comment.
The sed -E expression uses \s*, but GNU/POSIX sed ERE does not support \s as “whitespace”. As written, this will usually fail to match and CURRENT_VERSION becomes the entire line (including spaces), causing actions/setup-python to receive an invalid version string. Use [[:space:]]* (or a simpler parser like awk/python) to reliably extract the version.
There was a problem hiding this comment.
Fixed in commit 8b949d9. Changed \s* to [[:space:]]* for POSIX-compatible whitespace matching in sed.
| NEW_VERSION=$(grep -m1 "python-version:" .github/workflows/build-ci-image.yml | sed -E "s/.*python-version:\s*['\"]?([0-9.]+)['\"]?.*/\1/") | ||
| echo "new_version=$NEW_VERSION" >> $GITHUB_OUTPUT | ||
| echo "Updated to Python $NEW_VERSION" |
There was a problem hiding this comment.
Same issue as earlier: sed -E with \s* won’t match whitespace in sed ERE, so NEW_VERSION can become the full python-version: line instead of just the version. This will break downstream steps (branch name, commit message, PR title). Replace \s* with [[:space:]]* (or use a more robust extraction approach).
There was a problem hiding this comment.
Fixed in commit 8b949d9. Changed \s* to [[:space:]]* for POSIX-compatible whitespace matching in sed.
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: ${{ steps.current-version.outputs.version }} | ||
|
|
There was a problem hiding this comment.
This workflow sets up Python using the current version (from build-ci-image.yml) and never re-runs actions/setup-python after updating files. As a result, the venv/lint/pytest steps run on the old Python version, not the newly detected version you’re trying to validate. Add a second actions/setup-python step gated on update_available == 'true' that uses ${{ steps.get-version.outputs.new_version }} before creating the venv and running tests.
There was a problem hiding this comment.
Fixed in commit 8b949d9. Added a new "Set up new Python version for testing" step that runs actions/setup-python@v5 with the new version after files are updated, ensuring all tests run with the correct Python version.
| echo "Running pytest in Docker container..." | ||
| docker run --rm -v "$PWD:/workspace" -w /workspace test-infra:latest bash -c "python -m venv --system-site-packages .venv && ./.venv/bin/python -m pytest -q" | ||
| echo "docker_tests_passed=true" >> $GITHUB_OUTPUT |
There was a problem hiding this comment.
test-infra:latest sets USER ci (see .github/ci/Dockerfile.infra), but this docker run mounts the repo and tries to create .venv inside /workspace. This likely fails due to filesystem permissions (the existing container-based CI workflows run with options: --user 0, e.g. .github/workflows/pytest.yml:19-21). Run the container as root (--user 0) or write the venv to a writable path (e.g. /tmp`) to match established CI behavior.
There was a problem hiding this comment.
Fixed in commit 8b949d9. Added --user 0 flag to the docker run command to match the pattern used in other workflows (e.g., pytest.yml).
| - name: Handle test failures | ||
| if: | | ||
| steps.check-update.outputs.update_available == 'true' && | ||
| (steps.lint-test.outputs.lint_passed != 'true' || | ||
| steps.pytest-test.outputs.tests_passed != 'true' || | ||
| steps.docker-test.outputs.docker_build_passed != 'true' || | ||
| steps.docker-pytest.outputs.docker_tests_passed != 'true') | ||
| run: | | ||
| echo "❌ Tests failed with the new Python version" | ||
| echo "Lint passed: ${{ steps.lint-test.outputs.lint_passed }}" | ||
| echo "Pytest passed: ${{ steps.pytest-test.outputs.tests_passed }}" | ||
| echo "Docker build passed: ${{ steps.docker-test.outputs.docker_build_passed }}" | ||
| echo "Docker tests passed: ${{ steps.docker-pytest.outputs.docker_tests_passed }}" | ||
| echo "" | ||
| echo "Manual intervention required to fix compatibility issues." | ||
| echo "The version update has been applied but not committed." | ||
| exit 1 |
There was a problem hiding this comment.
The “Handle test failures” step won’t run if flake8/pytest/docker build fails, because those steps will terminate the job before this conditional executes. This contradicts the PR description about reporting specific failures. To make this work, mark the test steps continue-on-error: true and gate the failure-reporting step with if: always() && steps.check-update.outputs.update_available == 'true' (then inspect each step’s outcome/outputs).
There was a problem hiding this comment.
Fixed in commit 8b949d9. Added continue-on-error: true to all test steps and changed the failure handler to use if: always() with steps.*.outcome == 'failure' checks, ensuring it executes even when tests fail.
| # Update files | ||
| changes_made = False | ||
| for file_path in files_to_update: | ||
| if file_path.exists(): | ||
| if update_file_content( | ||
| file_path, | ||
| current_version, | ||
| latest_version, | ||
| current_docker_tag, | ||
| latest_docker_tag | ||
| ): | ||
| changes_made = True | ||
| else: | ||
| print(f"Warning: File not found: {file_path}") | ||
|
|
There was a problem hiding this comment.
update_file_content() returns False both when a file truly needs no change and when an exception occurs (or the expected pattern wasn’t found). In main(), any such case is silently tolerated as long as some other file changed, which can produce a partial/inconsistent version update (and a PR that doesn’t update all intended references). Track per-file success and fail the script if any target file is missing or still contains the old version/tag when an update is being applied.
There was a problem hiding this comment.
Fixed in commit 8b949d9. Enhanced the update logic to track failed files separately and verify that old versions were actually replaced. The script now fails with a clear error message if any file wasn't updated correctly, preventing partial updates.
|
@copilot : Address the new issues raised by the reviewer |
- Fix sed regex to use POSIX-compatible [[:space:]] instead of \s - Add setup-python step to test with NEW version, not old version - Add --user 0 to Docker run for proper permissions - Add continue-on-error to test steps for proper failure handling - Use step.outcome instead of outputs for test result checks - Add always() condition to failure handler so it runs even after failures - Improve file update tracking to fail on partial updates - Check files after update to ensure old version was replaced Co-authored-by: zaxlofful <33877007+zaxlofful@users.noreply.github.com>
All reviewer issues have been addressed in commit 8b949d9:
|
- Use regex patterns to check for old versions instead of substring matches - Prevents false positives when new version contains old version as substring - Change always() to !cancelled() in failure handler to avoid running on cancellation Co-authored-by: zaxlofful <33877007+zaxlofful@users.noreply.github.com>
Python and Docker Update Workflow Implementation
Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.