Skip to content

Plugin backend: mutations invisible to tests using from module import func #8

@sakost

Description

@sakost

Description

The plugin backend (persistent pytest worker pool) applies mutations by re-executing mutated source code into the target module's namespace via exec(). However, this approach fails when test files use from module import func style imports.

Root cause: When Python executes from mymath import add, it copies a direct reference to the function object into the test module's namespace. When the plugin replaces mymath.add with a new function object (via exec() into the module dict), the test's local add still points to the original function object. The mutation is invisible to the test. This is the same reason unittest.mock.patch requires patching where a name is used, not where it's defined.

Steps to reproduce

  1. Create a module mymath.py with a function add(a, b): return a + b
  2. Create a test file that uses from mymath import add and tests add(1, 2) == 3
  3. Run fest (defaults to plugin backend)
  4. Observe ~10% mutation score
  5. Run fest --backend subprocess
  6. Observe ~90% mutation score

Expected behavior

Plugin backend should detect mutants at a similar rate to the subprocess backend.

Actual behavior

Plugin backend reports ~10% mutation score vs ~90% for subprocess backend. Mutations applied via exec() into the module namespace create new function objects, but test modules that used from X import Y still hold references to the old (unmutated) function objects.

fest version

0.1.2

Operating system

All platforms

Python version

All versions

Additional context

  • Relevant code: src/plugin/_fest_plugin.py lines 160-163 (_handle_mutant)
  • Workaround: Use --backend subprocess
  • Possible fixes: walk sys.modules to replace stale references, re-execute test imports after mutation, or document as known limitation
  • Reported by https://www.reddit.com/user/DamnDoofus/

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions