Deploy coverage into invoke dev-system

[gpsaggese]

Follow up to

• Improve system coverage code #888

• Extend coverage in pytest to cover when we run through system #705

• Maddy: write a markdown ELI5 explaining the problem (with a system call and with dockerized executables), the solution, a examples of how to solve it by itself

  • ETA: Fri (Aug 22) or Mon

• Read the markdown, the code, and the documentation

  • @shaunak
  • @Heanh

• Propose a plan for integrating the machinery inside the invoke task

• Create an invoke that runs this end-to-end and shows that the coverage goes up

• Add to the paper, improve the internal documentation

The entry point:

> i run_coverage_subprocess -t dev_scripts_helpers/llms/test/

[Shaunak01]

@madhurlak0810 The .md LGTM. Just one question - did you validate this with Codecov? (We are using Codecov as our coverage visualization tool)

[madhurlak0810]

I checked the original codecov on master branch and I cross verified with the pytest results but I haven't used codecov on my branch.

[heanhsok]

Some QQ for now:

1. I'm seeing coverage from other dirs in the terminal output even with the --target-dir flag. Could you check this out? When looking at the HTML report, it shows only the specified target dir.

> invoke run_coverage_subprocess --target-dir=dev_scripts_helpers/llms --generate-html-report

14:13:05 - INFO  hdbg.py init_logger:1037                               > cmd='/Users/heanhsok/src/venv/client_venv.helpers/bin/invoke run_coverage_subprocess --target-dir=dev_scripts_helpers/llms --generate-html-report'
14:13:05 - INFO  lib_tasks_pytest.py run_coverage_subprocess:1010       Running comprehensive test coverage with subprocess injection...
14:13:15 - INFO  hcoverage.py coverage_combine:162                      Found coverage files in coverage_data/, copying to current directory
14:13:15 - INFO  hcoverage.py coverage_combine:175                      Found 7 coverage data files to combine
  ... Combined data file .coverage.6644a499fa34.1.XpxKNXNx
  ... Combined data file .coverage.96b5ce1511d5.1.XWYnwRWx
  ... Skipping duplicate data .coverage.9c32d2ce5590.1.XNKhrbBx
  ... Combined data file .coverage.Heanhs.local.92790.XdildZVx
  ... Combined data file .coverage.Heanhs.local.92815.XTYnqJux
  ... Combined data file .coverage.Heanhs.local.92884.XguChKmx
  ... Combined data file .coverage.Heanhs.local.92926.XLTcEdyx
  ... Name                                                    Stmts   Miss Branch BrPart  Cover
  ... -----------------------------------------------------------------------------------------
  ... conftest.py                                                79     36     18      7    52%
  ... dev_scripts_helpers/llms/dockerized_llm_transform.py       32      4      8      4    80%
  ... dev_scripts_helpers/llms/llm_prompts.py                   501     29     50     12    92%
  ... dev_scripts_helpers/llms/llm_transform.py                  97     25     22      7    68%
  ... dev_scripts_helpers/llms/llm_utils.py                      57     29     18      6    48%
  ... dev_scripts_helpers/llms/test/test_llm_apply_cfile.py       7      0      0      0   100%
  ... dev_scripts_helpers/llms/test/test_llm_prompts.py          50     19      0      0    62%
  ... dev_scripts_helpers/llms/test/test_llm_transform.py        67     13      6      1    78%
  ... helpers/hcache_simple.py                                  317    181    116     25    38%
  ... helpers/hcoverage.py                                       85     66     24      0    17%
  ... helpers/hdbg.py                                           399    257    136     21    32%
  ... helpers/hdocker.py                                        281    128     80     13    48%
  ... helpers/hdockerized_executables.py                        336    258     48      7    22%
  ... helpers/henv.py                                           218     93     48      4    49%
  ... helpers/hgit.py                                           569    456    130      3    17%
  ... helpers/hintrospection.py                                 129    106     50      0    13%
  ... helpers/hio.py                                            391    283    136     12    24%
  ... helpers/hlatex.py                                          37     29      4      0    20%
  ... helpers/hllm.py                                           200     95     50     11    46%
  ... helpers/hlogging.py                                       295    157     94     20    42%
  ... helpers/hmarkdown.py                                       11      0      0      0   100%
  ... helpers/hmarkdown_bullets.py                               78     67     34      0    10%
  ... helpers/hmarkdown_coloring.py                              85     73     30      0    10%
  ... helpers/hmarkdown_comments.py                              28     20     10      0    21%
  ... helpers/hmarkdown_fenced_blocks.py                         55     46     14      0    13%
  ... helpers/hmarkdown_filtering.py                             48     37      6      0    20%
  ... helpers/hmarkdown_formatting.py                           114     54     18      0    47%
  ... helpers/hmarkdown_headers.py                              274    240    106      0     9%
  ... helpers/hmarkdown_rules.py                                103     85     42      0    12%
  ... helpers/hmarkdown_slides.py                                69     58     22      0    12%
  ... helpers/hmarkdown_tables.py                                48     42     14      0    10%
  ... helpers/hmarkdown_toc.py                                    4      2      0      0    50%
  ... helpers/hparser.py                                        206    119     64      5    39%
  ... helpers/hprint.py                                         439    237    136     12    42%
  ... helpers/hserver.py                                        419    210    134     14    42%
  ... helpers/hsystem.py                                        457    327    140      8    24%
  ... helpers/htimer.py                                         116     60     14      3    45%
  ... helpers/hunit_test.py                                     676    480    176     19    26%
  ... helpers/hunit_test_purification.py                        126     93     20      0    24%
  ... helpers/hversion.py                                        84     23     14      3    65%
  ... helpers/hwall_clock_time.py                                40     19     10      1    44%
  ... helpers/hwarnings.py                                       25      4      4      1    83%
  ... helpers/repo_config_utils.py                              158     88     30      6    41%
  ... -----------------------------------------------------------------------------------------
  ... TOTAL                                                    7810   4648   2076    225    36%
  ... 
  ... 5 empty files skipped.
14:13:17 - INFO  hcoverage.py remove:75             Removed coverage hook from /Users/heanhsok/src/venv/client_venv.helpers/lib/python3.9/site-packages/coverage.pth via sudo rm
14:13:17 - INFO  hcoverage.py remove:86             Removed COVERAGE_PROCESS_START from environment

2. Are you able to run full coverage on the entire //helper repo? I'm seeing these errors. Probably it's dependency issues? Maybe it needs to be run in a container with all the deps installed? Could you try to run it and see how it would look like?

> invoke run_coverage_subprocess --generate-html-report
14:17:07 - INFO  hdbg.py init_logger:1037                               > cmd='/Users/heanhsok/src/venv/client_venv.helpers/bin/invoke run_coverage_subprocess --generate-html-report'
14:17:07 - INFO  lib_tasks_pytest.py run_coverage_subprocess:1010       Running comprehensive test coverage with subprocess injection...
14:17:10 - ERROR lib_tasks_pytest.py run_coverage_subprocess:1059       Coverage with subprocess failed: {e}
14:17:10 - INFO  hcoverage.py remove:75             Removed coverage hook from /Users/heanhsok/src/venv/client_venv.helpers/lib/python3.9/site-packages/coverage.pth via sudo rm
14:17:10 - INFO  hcoverage.py remove:86             Removed COVERAGE_PROCESS_START from environment
Traceback (most recent call last):
  File "/Users/heanhsok/src/venv/client_venv.helpers/bin/invoke", line 8, in <module>
    sys.exit(program.run())
  File "/Users/heanhsok/src/venv/client_venv.helpers/lib/python3.9/site-packages/invoke/program.py", line 398, in run
    self.execute()
  File "/Users/heanhsok/src/venv/client_venv.helpers/lib/python3.9/site-packages/invoke/program.py", line 583, in execute
    executor.execute(*self.tasks)
  File "/Users/heanhsok/src/venv/client_venv.helpers/lib/python3.9/site-packages/invoke/executor.py", line 140, in execute
    result = call.task(*args, **call.kwargs)
  File "/Users/heanhsok/src/venv/client_venv.helpers/lib/python3.9/site-packages/invoke/tasks.py", line 138, in __call__
    result = self.body(*args, **kwargs)
  File "/Users/heanhsok/src/helpers2/helpers/lib_tasks_pytest.py", line 1026, in run_coverage_subprocess
    hsystem.system(test_cmd, abort_on_error=True)
  File "/Users/heanhsok/src/helpers2/helpers/hsystem.py", line 329, in system
    rc, _ = _system(
  File "/Users/heanhsok/src/helpers2/helpers/hsystem.py", line 283, in _system
    raise RuntimeError(msg)
RuntimeError: 
################################################################################
################################################################################
_system() failed
################################################################################
################################################################################
# _system: cmd='(coverage \\\nrun \\\n--parallel-mode \\\n-m \\\npytest \\\n. ) 2>&1', abort_on_error=True, suppress_error=None, suppress_output=True, blocking=True, wrapper=None, output_file=None, num_error_lines=30, tee=False, dry_run=False, log_level=10
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
cmd='(coverage \
run \
--parallel-mode \
-m \
pytest \
. ) 2>&1'
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
- rc='2'
- output='
ERROR dev_scripts_helpers/github/test/test_invite_gh_contributors.py
ERROR dev_scripts_helpers/test/test_run_notebook.py
ERROR helpers/test/test_hasyncio.py
ERROR helpers/test/test_haws.py
ERROR helpers/test/test_hcache.py
ERROR helpers/test/test_hcache_simple.py
ERROR helpers/test/test_hcsv.py
ERROR helpers/test/test_hdataframe.py
ERROR helpers/test/test_hdatetime.py
ERROR helpers/test/test_hdict.py
ERROR helpers/test/test_hio.py
ERROR helpers/test/test_hllm.py
ERROR helpers/test/test_hlogging.py
ERROR helpers/test/test_hnumpy.py
ERROR helpers/test/test_hobject.py
ERROR helpers/test/test_hpandas.py
ERROR helpers/test/test_hparquet.py
ERROR helpers/test/test_hplayback.py
ERROR helpers/test/test_hunit_test.py
ERROR helpers/test/test_joblib_helpers.py
ERROR helpers/test/test_lib_tasks_docker_release.py
ERROR helpers/test/test_master_buildmeister_dashboard.py
ERROR import_check/test/test_detect_import_cycles.py
ERROR import_check/test/test_show_imports.py
ERROR linters/test/test_amp_class_method_order.py
ERROR linters/test/test_amp_dev_scripts.py
ERROR linters/test/test_amp_normalize_import.py
!!!!!!!!!!!!!!!!!!! Interrupted: 31 errors during collection !!!!!!!!!!!!!!!!!!!
======================== 4 skipped, 31 errors in 2.79s =========================
'
- Output saved in 'tmp.system_output.txt'
- Command saved in 'tmp.system_cmd.sh'

[madhurlak0810]

@heanhsok

1. The first issue comes from a filtering problem when I run through a target directory, all other subprocesses are triggered as well. I’ll work on fixing this, but technically those subprocesses are expected. Should I just filter down to only the target directory?
2. The main issue is setting up a proper test environment with all dependencies. The sibling-to-sibling container setup or running through ctx prevents coverage files from being pulled correctly since it executes outside the build_container_image limitation. Attempts to fix this keep leading to more complexities. Since a lot of the code spins up a bunch of containers with different dependencies and in different ways.

[heanhsok]

@madhurlak0810

Should I just filter down to only the target directory?

Yes, let's apply the filter to match the intended functionality of the flag. Feel free to file an issue

2. The main issue is setting up a proper test environment with all dependencies. The sibling-to-sibling container setup or running through ctx prevents coverage files from being pulled correctly since it executes outside the build_container_image limitation. Attempts to fix this keep leading to more complexities. Since a lot of the code spins up a bunch of containers with different dependencies and in different ways.

Could you elaborate on the issue? And the solution that you have tried, and why it doesn't work or too complex? Then we can review and brainstorm some solutions tgt.

[madhurlak0810]

1. I will file an issue for the filter.
2. So right now the system works such that any docker container built with build_container_image in hdocker works as intended i.e there is a coverage_data folder where coverage is collected which is injected but with the testing suite which spins a container with the dependencies installed it doesn't work as the container running the process stops before data is collected and stops. So I tried fixing this by rectifying that by making all containers stop until the data is mounted back but it caused some other errors and ends up crashing my system.