Lots of changes to test.py

[dwRchyngqxs]

DO NOT MERGE, THIS IS STILL WIP. To be done by next year.

I was so happy about how things turned out, I forgot to actually make small meaningful commits. This still needs some work including changes to the ProgressBar (see comments below).
Here is a list of changes:

Imports use from
Use of terminfo instead of raw vt codes for portability
Add print mutex to prevent jumbled messages
Logging to JUnit file is done in run_test (remove Result class)
Add JUnit mutex to prevent jumbled messages
Add JUnit title to be able to reuse the same class for other purposes (otherwise what's the point of the class...) ProgressBar -> AsyncProgressWithLog
ProgressBar uses background color
Progress bar creates a temporary screen
Removed overuse of color making the text hard to read
Add hook to print updating the progress bar(s)
Get terminal size from terminfo
Get terminal size each time it is needed (instead of once per run)
ProgressBar is now a context (with ProgressBar)
ProgressBar can display arbitrarily many kind of segments
ProgressBar stores the value of each segment
ProgressBar can be queried for value of a segment
Remove the ugly boolean list storing successes (only used for counting anyways)
ProgressBar doesn't display counts, it's going too fast anyways so what's the point
ProgressBar bar logic rework
Subprocesses only return a log (=failure) or None (=success), no point in remembering anything else
Added DEBUG to the environment of make clean
Redirect output of make and cmake
Make the compiler-to-run a python function so that we can do interesting stuff (like ac)
Remove directories for relative path, it gets too much in our way, add it back yourself if you care enough
Reference assembly is produced first to check if test is valid
Add description to run_subrocess to describe passes
Reprint all logs that matter after exiting the ProgressBar temporary screen
Add warnings and invalid tests to report
Remove all of the verbose stuff, output is clean anyways
Remove os.system call to fix terminal since nothing escapes the temporary screen
Prevent displaying key presses in new window

[dwRchyngqxs]

Ok, this is ready for initial review.
I need to make a simple AC PR, and split all the changes here in small commits. I am also not satisfied with some parts of this PR, for instance each AsyncProgress makes a new screens...
I will AC PR once this is in a good state with no significant changes to come.

[Fiwo735]

I've taken a quick look at the changes, look rather impressive! The anti-cheat stuff makes sense to me, but I have some questions and worries regarding printing/progress bar.

Regarding all the progress bar stuff, what is the actual goal? A progress bar that can update in place while logs are printed to terminal (above it?)? Currently we'd either print logs or have the progress bar depending on verbosity and it all works with multithreading. Doing both would be cool, but when would we actually need that?

Your current implementation is double in length and I worry it's quite cryptic (multiple code fragments which I have a hard time understanding) when it comes to what's actually happening due to all the low-level terminal manipulation and short variable names. You've probably looked into that, but with how complicated AsyncProgressWithLog has become now, is there enough benefit of using it instead of existing libraries like tqdm (simpler, we already use that for marking) or rich (more advanced, but I don't have experience with it). I think it could be a good time we transition to using them?

I have also for some time wanted to move our project from print(...) to logging to allow for an even simpler colour+verbosity selection, as logging would allow us to easily control both across files using existing colour codes and verbosity settings (hopefully even be able to get rid of stdout/stderr redirects). I was about to start researching that, but realised the your prints become quite convoluted now, e.g.:

print(f"{RED}Coverage is not supported with CMake. Switching to make.{RESET}")

became

print(
    ti_parm("setaf", COLOR_RED),
    "Coverage is not supported with CMake. Switching to make.",
    ti_get_str("sgr0"),
    sep=""
)

where I wanted to go in a direction of even more clarity, e.g.

logging.warning("Coverage is not supported with CMake. Switching to make.")

I worry that some of the proposed changes would make students' less likely to appreciate our coding design/style, as they would get scared away looking at the first dozens LoC 😆 Do you think there's some space to combine my simplification goal with your current async tech?

[dwRchyngqxs]

Because I'm gonna nuke this PR, here is a list of the stuff I think should be preserved/reimplemented:

• Add JUnit title as a constructor argument to be able to reuse the same class for other purposes (otherwise what's the point of the class...)
• Logging to JUnit file is done in run_test (remove Result class)
• Remove the ugly boolean list storing successes in run_tests (only used for counting anyways)
• Remove directories passed for relative path computation, use Path.cwd instead and relative_to only if the path is relative to cwd (otherwise walking up the path fragments can be an option but it's not in the PR)
• Subprocesses only return a log (=failure) or None (=success), this should be easy after removing Result by pushing the JUnit file as argument to run_test
• Add description to run_subrocess to describe passes (I think you already reimplemented it, but not sure so putting it here)
• (Re)Print all logs that matter after exiting the ProgressBar view
• Detect invalid added tests by failing at generating reference assembly
• Add warnings and invalid tests to report