fixtures: add dynamic sample discovery and reverse MD5 lookup

[kami922]

Summary

Implements dynamic file discovery for test fixtures, addressing issue #1743.

This is a cleaned-up resubmission of #2802, incorporating maintainer feedback.

Background

Issue #1743 requested adding get_sample_short_name_by_md5() as the inverse of the existing get_sample_md5_by_name() function. The previous PR #2802 was closed with feedback to follow the original issue's vision of using dynamic discovery instead of maintaining hardcoded lists.

Problem

The codebase maintains hardcoded lists of file names and MD5 hashes in get_sample_md5_by_name(), requiring manual updates whenever test samples are added. This creates maintenance burden and potential for errors.

Solution

This PR implements dynamic sample discovery as requested by @williballenthin in #1743:

• _build_sample_cache() - Walks tests/data/ directory once at module import time
• Auto-indexing - Samples indexed by MD5, SHA256, filename, and stem
• get_sample_short_name_by_md5() - Reverse MD5→name lookup using the dynamic cache
• Comprehensive tests - Parametrized tests for forward/reverse lookups and roundtrips

Benefits

• Zero maintenance: New test files are automatically discovered
• No manual hashing: Eliminates typo errors in 32-char MD5 strings
• Scalable: Code size stays constant regardless of sample count
• Fast: O(1) lookup performance via dictionary cache
• Foundation for future work: Enables generic lookup functions as discussed in fixtures: add function to resolve sample shortened name by MD5 #1743

Implementation Details

Follows the approach suggested by @williballenthin in #1743:

"we should tear a lot of that down and provide some generic routines that use os.walk to find the requested file"

Implementation is similar to collect_samples() in scripts/lint.py, as referenced in the issue discussion.

Changes

• tests/fixtures.py: Add cache building and reverse lookup function (+90 lines)
• tests/test_fixtures.py: Add parametrized tests for lookups and roundtrips (+126 lines)

Testing

pytest [test_fixtures.py](http://_vscodecontentref_/0) -v

[gemini-code-assist]

Summary of Changes

Hello @kami922, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant improvement to how test fixtures are managed by transitioning from a brittle, manually updated system to a dynamic, automated discovery and indexing mechanism. By building a comprehensive cache of test samples at module import, it enables efficient lookups by various identifiers, including a new reverse MD5-to-name lookup. This enhancement not only streamlines the process of adding new test data but also boosts the reliability and scalability of the testing infrastructure.

Highlights

• Dynamic Sample Discovery: Implemented _build_sample_cache() to automatically discover test fixtures by walking the tests/data/ directory at module import time, replacing manual updates.
• Automated Indexing: Samples are now indexed by MD5, SHA256, filename, and filename stem, providing flexible lookup capabilities.
• Reverse MD5 Lookup: A new function, get_sample_short_name_by_md5(), has been added to perform reverse lookups from an MD5 hash to the sample's short name.
• Comprehensive Testing: A new test file, tests/test_fixtures.py, introduces parametrized tests for both forward and reverse lookups, including roundtrip validation, ensuring the robustness of the new system.
• Reduced Maintenance: This change eliminates the need for hardcoded lists of file names and MD5 hashes, significantly reducing maintenance burden and potential for errors.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the master (unreleased) section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: [x] No CHANGELOG update needed

[gemini-code-assist]

Code Review

This pull request introduces dynamic sample discovery and a reverse MD5 lookup function, significantly improving the maintainability and scalability of test fixtures. The new _build_sample_cache function efficiently indexes samples by various attributes, and get_sample_short_name_by_md5 provides a convenient way to retrieve sample names from their MD5 hashes. Comprehensive tests have been added to validate the new functionality, including forward, reverse, and round-trip lookups. Overall, this is a well-implemented solution that addresses the maintenance burden of hardcoded lists.

CHANGELOG updated or no update needed, thanks! 😄

[kami922]

@mike-hunhoff @mr-tz Hello I am working on CI failed tests meanwhile can you please tell me is running CI tests locally Via "act" a good approach? so I can see failed tests and fix them before committing and pushing. currently I am not following this approach if yes could you also tell me few guidelines and best practices related to this project.

[williballenthin]

using "act" is fine. sometimes i do this to avoid waiting for the round trip of committing, pushing, and waiting for GH Actions to spawn and complete.

i'm not quite sure what hints would help. i'd suggest running the linters, formatters, and tests before submitting a PR. what else?

[kami922]

using "act" is fine. sometimes i do this to avoid waiting for the round trip of committing, pushing, and waiting for GH Actions to spawn and complete.

i'm not quite sure what hints would help. i'd suggest running the linters, formatters, and tests before submitting a PR. what else?

ok alot of tests are failing currently I will fix them. but one confusion I am having currently regarding running act locally is that I think act might take a lot of disk space of my laptop even tho docker image is pulled and current failing tests might according to my comprehension are related to OS. anyways I will experiment with it then fix it and commit again

[kami922]

Hello @mike-hunhoff @williballenthin

The CI is showing one failing test (test_fix262 in tests/test_main.py), but this failure is unrelated to the fixture discovery changes in this PR. Here's what's happening:

The failure is caused by upstream rules changes (capa-rules#1099) that were merged after I created this branch. The "send HTTP request" rule was made more strict to fix false positives, but this broke the test expectations.

• All 64 fixture discovery tests pass successfully
• The fixture discovery functionality works correctly
• The only failing test (test_fix262) is testing rule matching behavior, which is completely unrelated to fixture discovery

I've verified this locally - the test fails with the same error after rebasing onto latest master with updated submodules.

[kami922]

@mike-hunhoff @mr-tz Hello can I Please get an update on this one. Also Please tell me what can I do about the failing tests.

[mike-hunhoff]

@kami922 the tests are passing now. Please address any open feedback and we'll give this another review.

[kami922]

@kami922 the tests are passing now. Please address any open feedback and we'll give this another review.

hello I believe everything is done from my side, awaiting for your review

[mike-hunhoff]

@kami922 it looks like there are a couple of feedback comments listed above that need to be addressed. Please explicitly respond/resolve these comments if/when they are completed. Generally, this is a good habit for you to build for future code reviews. It makes the process smoother for you and the reviewers.

[kami922]

@mike-hunhoff Hello 1-2 changes were skipped and now done also I have responded to comments with appropriate message.

[kami922]

@mike-hunhoff Hello any update on this one? I had update this branch to merge with main one. can you please run the tests.

[mike-hunhoff]

@kami922 CI is failing:

=================================== FAILURES ===================================
_ TestGetSampleShortNameByMd5.test_reverse_lookup[5f66b82558ca92e54e77f216ef4c066c-mimikatz] _

self = <test_fixtures.TestGetSampleShortNameByMd5 object at 0x7f61767cd6c0>
md5 = '5f66b82558ca92e54e77f216ef4c066c', name = 'mimikatz'

    @pytest.mark.parametrize(
        "md5, name",
        [
            ("5f66b82558ca92e54e77f216ef4c066c", "mimikatz"),
            ("e80758cf485db142fca1ee03a34ead05", "kernel32"),
            ("a8565440629ac87f6fef7d588fe3ff0f", "kernel32-64"),
            ("db648cd247281954344f1d810c6fd590", "al-khaser_x86"),
            ("3cb21ae76ff3da4b7e02d77ff76e82be", "al-khaser_x64"),
        ],
    )
    def test_reverse_lookup(self, md5, name):
        """Test reverse MD5 lookup returns correct sample name."""
        result = fixtures.get_sample_short_name_by_md5(md5)
>       assert result == name
E       AssertionError: assert '5f66b82558ca...7f216ef4c066c' == 'mimikatz'
E         
E         - mimikatz
E         + 5f66b82558ca92e54e77f216ef4c066c

tests/test_fixtures.py:102: AssertionError
______ TestMd5NameLookupRoundtrip.test_roundtrip_simple_samples[mimikatz] ______

self = <test_fixtures.TestMd5NameLookupRoundtrip object at 0x7f61767cf0a0>
name = 'mimikatz'

    @pytest.mark.parametrize("name", ["mimikatz", "kernel32", "kernel32-64"])
    def test_roundtrip_simple_samples(self, name):
        """Test name->MD5->name roundtrip for samples whose filename stem matches the lookup name."""
        md5 = fixtures.get_sample_md5_by_name(name)
        result_name = fixtures.get_sample_short_name_by_md5(md5)
>       assert result_name == name
E       AssertionError: assert '5f66b82558ca...7f216ef4c066c' == 'mimikatz'
E         
E         - mimikatz
E         + 5f66b82558ca92e54e77f216ef4c066c

tests/test_fixtures.py:128: AssertionError
=========================== short test summary info ============================
FAILED tests/test_fixtures.py::TestGetSampleShortNameByMd5::test_reverse_lookup[5f66b82558ca92e54e77f216ef4c066c-mimikatz] - AssertionError: assert '5f66b82558ca...7f216ef4c066c' == 'mimikatz'
  
  - mimikatz
  + 5f66b82558ca92e54e77f216ef4c066c
FAILED tests/test_fixtures.py::TestMd5NameLookupRoundtrip::test_roundtrip_simple_samples[mimikatz] - AssertionError: assert '5f66b82558ca...7f216ef4c066c' == 'mimikatz'
  
  - mimikatz
  + 5f66b82558ca92e54e77f216ef4c066c
====== 2 failed, 791 passed, 53 skipped, 179 xfailed in 465.09s (0:07:45) ======

Please ensure all tests are passing locally before you request another review, thank you!

[kami922]

@mike-hunhoff Local tests: 992 passed, 179 xfailed, 566 skipped, 3 failed. The 2 failures in test_segfault_fix.py are from a local debug file (not part of the repo). test_fix262 fails due to rules PR #1099 (8caf489d) changing string: /HTTP/i to a stricter HTTP request regex — pre-existing on upstream/master, unrelated to this PR. check ss

the skipped tests are related to pyghidra ida pro and binary ninja as i dont have them installed locally.

[kami922]

@mike-hunhoff can you please review?