Feature request: CI testing on opensource projects

[furtib]

We should have a CI job test the code on smaller open-source projects, to test on real-world examples.

The idea as a template is viewable in: #16

Scope:

1. Add CI job to run one, singular test
2. Expand the CI job to handle multiple tests
3. Add local running of tests
   • Add .gitignore for artifacts created
4. Document how to add tests to the test suite# DoD:

We have a working test set of at least 2 open source projects that can be run both locally and in github action

[furtib]

We spoke about how the file structure is not good.

Currently, it is as follows:

• The main workflow (test.yml) is not an ideal name, but I have no idea what we could call it
• The install dependencies step is in actions/env_setup/action.yml. This was done because it is used in multiple tests. It is a composite action; the file name came from the GitHub guide I was following. I haven't paid much attention to it. Maybe we could move it to workflows/composit_actions/env_setup.yml
• The scripts that set up the projects to be tested currently reside in workflows/patches/ I thought I would call them patches because they change the BUILD and WORKSPACE files, but we could also call the folder init-scripts (?). Currently, the script file names follow this naming convention: patch-project_name.sh. This could be changed to just project_name.sh

[Szelethus]

I think one of the things we want to define is what a contributor needs to do to add a new project to the test suite. I think its sensible that each project should have its own setup file. Considering that

• whether the repo should be cloned as-is or recursively,
• which version we should check out that supports the desired bazel version (6.5.0. currently),
• the placement of the BUILD/WORKSPACE file,
• what target we want analyzed,

it makes sense that at least these things must be defined in the project-specific setup file. One thing that could be omitted, is the the contents of the patch applied to the WORKSPACE file, that doesn't seem to change from one project setup to the next.

The general idea of the PR makes sense in that one shouldn't need to make any boilerplate changes to the workflow file, it should be sufficient to add a setup file to the correct directory and just be done with it.

The main workflow (test.yml) is not an ideal name, but I have no idea what we could call it

With regards to the directory structure, I think what is missing from the naming is that this is an open-source project benchmark or testsuite, but the directory name "patches" doesn't hint at that. How about this?

.github
├── actions
│   └── setup_codechecker_bazel.yml <--- shared setup steps 
└── workflows
    ├── open_source_project_testsuite
    │   ├── open_source_project_testsuite.yml <--- only testsuite actions
    │   └── project_setup_scripts
    │       ├── setup-yaml-cpp.sh
    │       └── setup-zlib.sh
    └── unit_tests.yml <--- only running tests found in the test folder

The install dependencies step is in actions/env_setup/action.yml. This was done because it is used in multiple tests. It is a composite action; the file name came from the GitHub guide I was following. [...]

I think the idea is nice, but we should create a separate NFC change that initially just moves the shared setup steps to a separate files, thats is.

The scripts that set up the projects to be tested currently reside in workflows/patches/ I thought I would call them patches because they change the BUILD and WORKSPACE files, but we could also call the folder init-scripts (?).

Yeah, something like that, considering that a traditional patch file is nothing but a patch. We do some other stuff (see the list above). Either init-.* or setup-.* would be good IMO.

[nettle]

How can we run tests on open source projects locally? What if GitHub CI is not available?

[Szelethus]

The suggestion implied in #22, where the patch scripts are not in .github sounds very agreeable to me. My only thought is that we should still run the analysis of each project in a github-workflow managed way, not through test_foss.py to leverage parallelism and better separation of sub jobs.