Seeking examples of pytest-testmon in GitHub Actions

[xianyuanliu]

Is your feature request related to a problem? Please describe.

I'm looking to use pytest-testmon to optimize test execution time in the GitHub Action. I use the GitHub Action Cache to manage .testmondata storage and restoration. Here's my design approach:

1. The main branch undergoes scheduled tests to generate a new .testmondata.
2. Branches within PRs simply restore the generated .testmondata for a faster test, without updating it.

However, an issue arises when I execute pytest --testmon-nocollect within the PR: a new .testmondata is unexpectedly generated in the GitHub Action Cache. I'm unsure about the root cause of this. Could you possibly share any examples or repositories that have encountered and addressed a similar scenario?

Describe the solution you'd like
When I run pytest --testmon-nocollect within the PR, no new .testmondata is generated in the GitHub Action Cache. However, executing pytest --testmon does result in the creation of a new .testmondata in the cache.

Describe alternatives you've considered
Not yet.

Additional context
Nope.

[shristi0210]

Hello @xianyuanliu , I've been trying to cache the .testmondata like you mentioned in your issue. Would that be possible if you could help me with the code of doing so?
Thanks.

[xianyuanliu]

I was unable to successfully resolve it yet due to differences between the cache systems of GitHub Action and the local machine. While I am glad to help, I might not be able to resolve the issue if it is the same issue.

[tarpas]

@xianyuanliu At first I misunderstood your suggested process.
I think it makes more sense to:

1. use www.testmon.net
2. use pytest --tmnet --testmon-noselect on the main branch
3. use pytest --tmnet on all other branches.

Collecting testmon data only nightly would lead to unnecessary re-execution of all tests related to all changes of recent changes and branches.
As for "new" .testmondata being created I'm not sure what the main problematic aspect. --testmon-nocollect is intended for situation when you can't collect testmondata because you're running some incompatible coverage switches or you can't afford the overhead. I would hope none of that is your case. It's not intended to keep the .testmondata intact. Testmon is not designed that way.

Are you up for using --testmon and --testmon-noselect? Or should we explore more your use case of --testmon-nocollect?

[marcodicro-dp]

@tarpas Is there a way to run all my tests and build the testmon DB faster by running tests in parallel workers (e.g using a GHA matrix)?

Does --tmnet already handle that distributed processing case?