Fix Error Code #31
Merged
Fix Error Code #31
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Replace linear regression breakdown with 6x6 state transition matrix - States ordered: Pass, Skip, XFail, Fail, Error, Nonexistent - Improvements appear below diagonal, regressions above - Bold formatting for regressions, italic for improvements - Add Error as separate state (previously grouped with Fail) - Track error_tests separately in pytest, nunit, xunit workflows - Move discovery warnings section between matrix and detail lists - Keep collapsible detail sections for test names below matrix
Replace generic "Baseline" and "Current" labels with actual branch names from workflow inputs (baseline_label, current_label)
Resolves merge conflict in regression-test.yml by keeping the new matrix-aligned collapsible sections that iterate through all state transitions. Removes unused intermediate variables that were added in main.
The target branch inline extraction code was grouping 'error' outcomes with 'failed', while test-py-pytest.yml tracks them separately. This mismatch caused phantom state transitions in the regression matrix (e.g., thousands of "Fail → Error" shown because the baseline grouped errors with failures but the current branch tracked them separately). Changes: - Add error_tests list to track error outcomes separately - Split 'elif outcome in ["failed", "error"]' into separate conditions - Add error_tests to test_data.json output - Add error_count to job outputs and cache
Old cached results may have errors grouped with failures (pre-fix). Adding a version string (v2) to the cache key ensures all PRs get fresh test results with proper error tracking. Bump CACHE_VERSION when extraction logic changes in the future.
Adds tracking for pytest's 'xpassed' outcome - when a test marked with @pytest.mark.xfail unexpectedly passes. Changes: - test-py-pytest.yml: Track xpassed_tests separately - run-branch-test.yml: Track xpassed_tests in inline extraction code - regression-test.yml: Add XPass to STATES matrix (7x7 now) - XPass placed between Pass and Skip in state ordering - Pass <-> XPass transitions are neutral (marker change only) - XPass to worse states = regression - Worse states to XPass = improvement - Bump cache version to v3 to invalidate stale results
Previously, test results were only extracted if pytest exitcode <= 1. This caused entire test lists to be empty when pytest had internal errors, collection errors, or interrupts (exit codes 2-5), leading to massive phantom state transitions in the regression matrix. Now we extract whatever test results are available in results.json regardless of the exit code. This ensures accurate regression comparisons even when one run had errors. Changes: - test-py-pytest.yml: Remove exitcode <= 1 gate on result extraction - run-branch-test.yml: Remove exitcode <= 1 gate on result extraction - Bump cache version to v4 to invalidate stale results
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.