-
Notifications
You must be signed in to change notification settings - Fork 18
Description
One of the two jobs we perform on Travis-CI often errors for a while now due to exceeding time limits (The job exceeded the maximum time limit for jobs, and has been terminated. or No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.). After some investigations, I realised that many doctests run much longer when executing the complete nox doctest session than when executing it only in "Python mode" or "Cython mode". This difference is because code coverage with Coverage.py is measured only when executing the complete session. The problem is most severe when doctests involve the spawning of subprocesses. In one experiment, all doctests of the hyd module took 365 seconds with and 8 seconds without coverage measurement.
Possible options:
- Use a faster tool. SlipCover could be an option, but it might still not be able to combine the coverages of multiple processes (see this issue).
- Split the affected job as suggested here. However, then we would need to double build time (or can we share the built wheel somehow?) and would somehow have to combine the coverage reports of different jobs.
- Execute the session only in "Python mode" but still measure code coverage, which would likely require adding Cython-specific tests that are always performed to yield 100% coverage.
- Speed up the complete session by measuring the code coverage of only some spawned subprocesses. (This would be my preferred option, so I did a quick experiment but did not achieve a sufficient performance increase.)
I am afraid we need to measure code coverage manually on our local from time to time until we have fixed this.