Skip to content

Conversation

@kaihowl
Copy link
Owner

@kaihowl kaihowl commented Jan 11, 2026

Summary

  • Removed hermetic git environment variables from mutation-testing workflow job
  • Added pull_request path filter to run the workflow only when this workflow file changes
  • Added PR-based quick smoke test for PR validation: on PRs, run a targeted mutation test; on schedule, run full mutation testing
  • Test infrastructure already sets hermetic git settings automatically
  • No other workflow changes besides env-var removal and PR-path gating

Problem

The weekly mutation testing workflow checkout could fail at the checkout step due to hermetic git environment variables being set at the job level, which prevented actions/checkout from configuring git's safe.directory.

Root Cause

These environment variables were redundant—the test infrastructure already configures hermetic git settings in tests:

  1. Rust unit tests: hermetic_git_env() in git_perf/src/test_helpers.rs:20-26
  2. Bash integration tests: export when spawning bash processes in git_perf/tests/bash_tests.rs:26-37
  3. Bash test scripts: exported in test/common.sh:41-47

Solution

  • Remove the hermetic git environment variables from the mutation-testing workflow:
    • GIT_AUTHOR_NAME / GIT_AUTHOR_EMAIL
    • GIT_COMMITTER_NAME / GIT_COMMITTER_EMAIL
    • GIT_CONFIG_NOSYSTEM
    • GIT_CONFIG_GLOBAL
  • This allows checkout and all workflow steps to use normal git configuration while tests continue to run in hermetic environment via test infrastructure.
  • Added pull_request path filter so the workflow only runs when this workflow file changes
  • Added a PR-triggered smoke-test: when the workflow is invoked by a PR, run a quick mutation test against a single file to validate workflow wiring; scheduled runs still perform full mutation testing

Test Plan

  • Verify workflow syntax is valid
  • Gate mutation-testing workflow to PR changes (triggered when this workflow file is modified)
  • PR quick smoke-test runs for PR validation
  • Full mutation testing runs on schedule
  • Checkout step completes without errors
  • Mutation testing runs with hermetic git config (from test infrastructure)

Fixes #611

🤖 Generated with Claude Code

📎 Task: https://www.terragonlabs.com/task/83b22e1e-62d7-46cd-aa3f-6e277c4bed37

@github-actions
Copy link
Contributor

github-actions bot commented Jan 11, 2026

Performance Report

Performance Results

Audit Results

✅ 'add-benchmark'
z-score (stddev): ↓ 0.17
Head: μ: 21.5ms σ: N/A MAD: 0ns n: 1
Tail: μ: 21.7ms σ: 892.3µs MAD: 294.4µs n: 25
 [-11.56% – +7.17%] ▇▅▆▅▁▅▅▄█▅▇▇▅▆▅▅▅▅▅▅▅▆▅▅▁▅
✅ 'release-binary-size'
z-score (stddev): ↑ 0.70
Head: μ: 8.1MB σ: N/A MAD: 0B n: 1
Tail: μ: 8MB σ: 9.3kB MAD: 2.3kB n: 25
 [-0.52% – +0.03%] ██████████████▆▆▆▆▆▆▆▆▆▆▁█
✅ 'report'
z-score (stddev): ↑ 0.63
Head: μ: 28ms σ: N/A MAD: 0ns n: 1
Tail: μ: 26.3ms σ: 2.7ms MAD: 1.6ms n: 38
 [-34.42% – +10.87%] ▇▇▇▇▃▇▇▆█▇█▇▇▇▆▆▆▆▆▆▆▇▇▆▅▄▄▄▄▄▄▄▁▄▄▄▅▄▇
✅ 'report-benchmark'
z-score (stddev): ↓ 0.19
Head: μ: 28.2ms σ: N/A MAD: 0ns n: 1
Tail: μ: 28.4ms σ: 1.2ms MAD: 599.1µs n: 25
 [-9.44% – +10.98%] ▆▅▅▅▄▄▅▄█▄▆▆▅▆▃▃▄▄▃▄▃▄▄▃▁▄
✅ 'report-size'
z-score (stddev): ↑ 1.29
Head: μ: 11.6kB σ: N/A MAD: 0B n: 1
Tail: μ: 5.5kB σ: 4.7kB MAD: 62B n: 38
 [-3.18% – +496.20%] ██████████████▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁█
✅ 'report-size-benchmark'
z-score (stddev): →
Head: μ: 11.6kB σ: N/A MAD: 0B n: 1
Tail: μ: 11.6kB σ: 0B MAD: 0B n: 14
 [+0.00% – +0.00%] ▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅

Measurement Storage Size

Live Measurement Size Report
============================

⚠️  Shallow clone detected - measurement counts may be incomplete (see FAQ)

Number of commits with measurements: 259
Total measurement data size (on-disk (compressed)): 298.4kB

Repository Statistics (for context):
-------------------------------------
  Loose objects: 0 (0B)
  Packed objects: 2115 (983kB)
  Total repository size: 983kB

Created by git-perf

The mutation testing workflow was failing during checkout because
GIT_CONFIG_GLOBAL=/dev/null was set at the job level, preventing
actions/checkout@v6 from configuring git safe.directory.

These environment variables are unnecessary in the workflow because
the test infrastructure already sets them:
- Rust unit tests: hermetic_git_env() in test_helpers.rs
- Bash integration tests: explicitly set in bash_tests.rs
- Bash test scripts: exported in test/common.sh

Removing them entirely fixes the checkout issue and eliminates
redundant configuration.

Fixes #611

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@kaihowl kaihowl force-pushed the terragon/fix-issue-611-analysis-z1ksj3 branch from 84ae5dc to 9ce9559 Compare January 11, 2026 06:27
@kaihowl kaihowl changed the title fix(ci): move hermetic git env vars to mutation testing step only fix(ci): remove hermetic git env vars from mutation-testing workflow Jan 11, 2026
Add pull_request trigger with path filter to run mutation testing
workflow when the workflow file itself is modified. This allows
validating workflow changes before merging.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@kaihowl kaihowl changed the title fix(ci): remove hermetic git env vars from mutation-testing workflow fix(ci): remove hermetic git env vars and gate workflow on PR changes Jan 11, 2026
When running on pull requests, use a limited smoke test that only
tests one file (config.rs) with reduced timeout (20s). This verifies
the workflow infrastructure works without running the full mutation
test suite.

On scheduled runs, continue to run full mutation testing.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@kaihowl kaihowl changed the title fix(ci): remove hermetic git env vars and gate workflow on PR changes fix(ci): gate PR changes and add PR smoke-test for mutation-testing Jan 11, 2026
@kaihowl kaihowl merged commit 3e2e826 into master Jan 11, 2026
24 checks passed
@kaihowl kaihowl deleted the terragon/fix-issue-611-analysis-z1ksj3 branch January 11, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🔴 Scheduled Mutation Testing Workflow Failed

2 participants