Memory leak when using daemon

[Dave562CZ]

When using gradle daemon registered shutdown hooks are not run because task is run inside daemon which keeps running after build is done. And also stays in memory so new shutdown hooks are added with subsequent builds and filling heap until it runs out of heap space.

[woldie]

We've got Gradle daemon disabled to avoid this shutdown hook memory snowball issue.

For anyone coming from google, if you see the following message a bunch of times on the console along with a sick/stuck Gradle process:

Expiring Daemon because JVM heap space is exhausted

Then, try adding the following to your top-level gradle.properties file to disable Gradle daemon until there's a fix:

# Turns out psxpaul's gradle-exec-plugin has a memory leak caused by a snowballing shutdown hook
# that never fires in the case of gradle running as a daemon.  Until that bug is fixed, I am
# disabling the gradle daemon to avoid the leak.
# See https://github.com/psxpaul/gradle-execfork-plugin/issues/25
org.gradle.daemon=false

Hey @psxpaul, could org.gradle.api.invocation.Gradle#buildFinished be a possible substitute for shutdown hooks in your plugin?

[psxpaul]

Thanks for the feedback @woldie. I believe I used a shutdown hook because the stop must be run when the user presses ctrl-c to kill the process.

I suspect calling Runtime.removeShutdownHook() would help, but I'm not able to reproduce this locally. Can you provide more information on how to reproduce this? Or perhaps try adding that call to AbstractExecFork.stop() and seeing if that fixes it for you?

[Dave562CZ]

Hello to replicate this issue it should be enough to run some execfork task several times with using daemon and then do heapdump of that daemon process and there should be multiple threads saved as shutdown hooks even after build is done.

[zielezin]

Hi @psxpaul,

We faced this issue on our CI infrastructure and running without a daemon is not an option.

After digging into the source code I've found that ForkTaskTerminationService implemented in #45 is already tracking the processes and cleans them up post build - this works nicely with daemon and the wrapper.

I think you could remove these lines and leave cleaning up for the service.

The only use cases it would leave started processes is:

• running the build in no daemon mode and ctrl-c the build - this is not Gradle default behavior and should be used in very rare cases.
• killing a daemon in the middle of the build started with gradlew (assuming it's not done with kill -9) - I see no reason to do this ;)

If you would really want to leave the hook in there you could register it, track it in the list and unregister in the service in the same way as tasks are tracked right now.

[psxpaul]

I've added 8328794 to try to address this. Please try 0.2.2 to see if your issues are fixed.

[zielezin]

Hi @psxpaul ,
I've used the 0.2.2 version for some time and it behaves better. The number of OOMs decreased but they still happen from time to time.
I'm trying to figure out what path leads to the thread being not registered - most probably a negative case of some sort.

[onkarjoshi]

Saw this issue being easily reproducible locally with v0.2.0. Upgraded to v0.2.2. That's made the problem go away for my use case.

Thanks.