[MCLEAN-102] Fast is slower on windows with many cores

[jira-importer]

Delany opened MCLEAN-102 and commented

I run a 500+ module project with 4 core Linux and experience a 20sec improvement in reactor build time when enabling the fast.

My one colleagues runs the same build with 4 cores on Windows and also sees an improvement.

But two other colleagues run the same build with 12 cores on Windows (one uses cygwin, the other powershell) and they consistently see a slowdown from 2:14 to 3:23, for example.

---

Affects: 3.2.0

[jira-importer]

Peter De Maeyer commented

There is not much to go on in this issue to effectively root-cause this, but let's see if we can hypothesize as to what might be a root cause.

Looking at the code Cleaner.delete, fast mode together with includes/excludes could have a performance impact. As soon as a file is excluded from a directory, it will cause the parentDir.delete to fail because the dir is obviously not empty, which will result in a System.gc() on Windows + 3 retries with back-off delays of 50 + 250 + 750 milliseconds, only to fail 3 times over again for the same reason. Depending on failOnError this will actually fail the build or not, but in any case it will cause an unnecessary delay. This could be fixed, by rewriting the Cleaner to leverage Files.walkFileTree(Path, FileVisitor) + being smarter about when to (not) delete a directory. MCLEAN-126 won't fix it, but is a prerequisite first step towards improving this.

[desruisseaux]

Migration to Files.walkFileTree(Path, FileVisitor) is done in #243. There is still, however, the call to System.gc() on Windows followed by retries when a file cannot be deleted. This strategy could be changed as follow: remember that a file could not be deleted and continue immediately with other files. Only after the whole tree has been walked, if there is at least one file that couldn't be deleted, call System.gc() and perform a second pass. Maybe the lock had the time to be released during the time that other files were deleted.