Skip to content

fix: fall back to simple assertions when test gen can't resolve value types#937

Merged
chubes4 merged 1 commit intomainfrom
fix/test-gen-todo-fallback
Mar 23, 2026
Merged

fix: fall back to simple assertions when test gen can't resolve value types#937
chubes4 merged 1 commit intomainfrom
fix/test-gen-todo-fallback

Conversation

@chubes4
Copy link
Member

@chubes4 chubes4 commented Mar 23, 2026

Summary

Fixes a gap in the test generation pipeline (#818) where unresolvable return types produced dead TODO stubs instead of real tests.

Problem

The test gen pipeline has two tiers of assertion templates:

result_ok       → assert!(result.is_ok(), ...)     ← real test
result_ok_value → let _ = inner; // TODO: assert   ← dead stub

When a function returns Ok(value) and the value's type is in the registry, enrich_assertion_with_fields replaces the TODO with real assert_eq! calls. But when the type ISN'T in the registry (common for complex or external types), the TODO stays.

These stubs:

  • Compile but test nothing
  • Get flagged as TodoMarker audit findings
  • Can't be auto-fixed (circular: autofix generates them → audit flags them)
  • Pollute the codebase with noise

Fix

After enrich_assertion_with_fields, if the assertion still contains // TODO:, fall back to the simpler non-value template:

result_ok_value  →  result_ok
option_some_value → option_some
result_err_value  → result_err

Before: let _ = inner; // TODO: assert specific value for "skipped"
After: assert!(result.is_ok(), "expected Ok for: ...");

A test that asserts the discriminant (is_ok, is_some, is_err, is_none) is a real behavioral test. The value-level assertion is a nice-to-have that should only appear when it can actually produce meaningful assertions.

Impact

This should reduce the missing_test_method and todo_marker issue counts in both homeboy and data-machine by eliminating the stub generation path.

… types (#818)

When the test generator detects a function returns Ok(value) or
Some(value) but can't resolve the type for field-level assertions,
it was leaving TODO placeholder stubs:

  let _ = inner; // TODO: assert specific value for "skipped"

These compile but test nothing, get flagged as TodoMarker findings,
and create a circular loop in the autofix pipeline.

Now: if enrich_assertion_with_fields fails to resolve the type, the
generator falls back to the simpler discriminant-only assertion
(e.g. result_ok instead of result_ok_value):

  assert!(result.is_ok(), "expected Ok for: ...");

A test that asserts is_ok() is better than a dead TODO stub.
@chubes4 chubes4 merged commit 103ba8a into main Mar 23, 2026
1 check passed
@chubes4 chubes4 deleted the fix/test-gen-todo-fallback branch March 23, 2026 01:57
@homeboy-ci
Copy link
Contributor

homeboy-ci bot commented Mar 23, 2026

Homeboy Results — homeboy

Audit

Failure Digest

Audit Failure Digest

  • Alignment score: 0.921
  • Severity counts: info: 2, unknown: 3, warning: 13
  • Outliers in current run: 3
  • Parsed outlier entries: 3
  • Drift increased: yes
  • New findings since baseline: 2
    1. dead_code — Public function 'generate_tests_for_file' is not referenced by any other file (dead_code::src/core/engine/contract_testgen.rs::UnreferencedExport)
    2. dead_code — Public function 'generate_tests_for_methods' is not referenced by any other file (dead_code::src/core/engine/contract_testgen.rs::UnreferencedExport)
  • Top actionable findings:
    1. src/core/engine/contract_testgen.rs — god_file — File has 2877 lines (threshold: 1000)
    2. src/core/engine/contract_testgen.rs — unreferenced_export — Public function 'generate_tests_for_file' is not referenced by any other file
    3. src/core/engine/contract_testgen.rs — unreferenced_export — Public function 'generate_tests_for_methods' is not referenced by any other file
    4. src/core/engine/contract_testgen.rs — missing_test_method — Method 'build_project_type_registry' has no corresponding test (expected 'test_build_project_type_registry')
    5. src/core/engine/contract_testgen.rs — missing_test_method — Method 'generate_test_plan' has no corresponding test (expected 'test_generate_test_plan')
All parsed audit findings (18)
1. **src/core/engine/contract_testgen.rs** — god_file — File has 2877 lines (threshold: 1000)
2. **src/core/engine/contract_testgen.rs** — unreferenced_export — Public function 'generate_tests_for_file' is not referenced by any other file
3. **src/core/engine/contract_testgen.rs** — unreferenced_export — Public function 'generate_tests_for_methods' is not referenced by any other file
4. **src/core/engine/contract_testgen.rs** — missing_test_method — Method 'build_project_type_registry' has no corresponding test (expected 'test_build_project_type_registry')
5. **src/core/engine/contract_testgen.rs** — missing_test_method — Method 'generate_test_plan' has no corresponding test (expected 'test_generate_test_plan')
6. **src/core/engine/contract_testgen.rs** — missing_test_method — Method 'generate_test_plan_with_types' has no corresponding test (expected 'test_generate_test_plan_with_types')
7. **src/core/engine/contract_testgen.rs** — missing_test_method — Method 'generate_tests_for_file' has no corresponding test (expected 'test_generate_tests_for_file')
8. **src/core/engine/contract_testgen.rs** — missing_test_method — Method 'generate_tests_for_file_with_types' has no corresponding test (expected 'test_generate_tests_for_file_with_types')
9. **src/core/engine/contract_testgen.rs** — missing_test_method — Method 'generate_tests_for_methods' has no corresponding test (expected 'test_generate_tests_for_methods')
10. **src/core/engine/contract_testgen.rs** — missing_test_method — Method 'generate_tests_for_methods_with_types' has no corresponding test (expected 'test_generate_tests_for_methods_with_types')
11. **src/core/engine/contract_testgen.rs** — missing_test_method — Method 'render_test_plan' has no corresponding test (expected 'test_render_test_plan')
12. **src/core/engine/contract_testgen.rs** — orphaned_test — Test method 'test_empty_grammar' references 'empty_grammar' which no longer exists in the source
13. **src/core/engine/contract_testgen.rs** — orphaned_test — Test method 'test_enrich_assertion_replaces_todo_with_field_hints' references 'enrich_assertion_replaces_todo_with_field_hints' which no longer exists in the source
14. **src/core/engine/contract_testgen.rs** — orphaned_test — Test method 'test_enrich_assertion_skips_when_no_type_in_registry' references 'enrich_assertion_skips_when_no_type_in_registry' which no longer exists in the source
15. **src/core/engine/contract_testgen.rs** — orphaned_test — Test method 'test_full_grammar' references 'full_grammar' which no longer exists in the source
16. **src/commands/docs.rs** — outlier — (outlier)
17. **tests/commands/deploy_test.rs** — outlier — (outlier)
18. **src/core/engine/undo/snapshot.rs** — outlier — (outlier)
- Full audit log: https://github.com/Extra-Chill/homeboy/actions/runs/23418161184

Autofixability classification

  • Overall: auto_fixable
  • Autofix enabled: yes
  • Autofix attempted this run: no
  • Auto-fixable failed commands:
    • audit
  • Failed commands with available automated fixes:
    • audit

Machine-readable artifacts

  • homeboy-lint-summary.json
  • homeboy-test-failures.json
  • homeboy-audit-summary.json
  • homeboy-autofixability.json

⚡ Scope: changed files only

audit (changed files only)

Auto-refactor

ℹ️ Autofix enabled, but no fixable file changes were produced

Failure Digest

Autofixability classification

  • Overall: human_needed
  • Autofix enabled: yes
  • Autofix attempted this run: no
  • Human-needed failed commands:
    • refactor --from all

Machine-readable artifacts

  • homeboy-lint-summary.json
  • homeboy-test-failures.json
  • homeboy-audit-summary.json
  • homeboy-autofixability.json

⚡ Scope: changed files only

refactor --from all

Tooling versions
  • Homeboy CLI: homeboy 0.85.3+a98da6a
  • Extension: rust from https://github.com/Extra-Chill/homeboy-extensions
  • Extension revision: unknown
  • Action: Extra-Chill/homeboy-action@v2

Homeboy Action v1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant