Jupyter hangs on long-running computation

[tdaff]

Original report by Leonid Kostrykin (Bitbucket: theV0ID, GitHub: theV0ID).

---

I'm using a remote IPython kernel which I added as follows:

#!bash
remote_ikernel manage --add \
--remote-precmd="source activate /path/to/conda/envs/ipython" \
--kernel_cmd="ipython kernel -f {connection_file}" --name "remote_host/Python2" \
--interface=ssh --host="remote_host" --workdir="/some/path"

I run an arbitrary computation which takes a few hours to finish. The progress of the computations are printed in the Notebook, and the print-out updates itself every second. If I remain at the computer and keep interacting with the Jupyter notebook (mostly scrolling to see the progress) the computation finishes as expected. Contrary, If I just leave the Jupyter notebook open as-is and go away for a few hours (the browser and the notebook remain open of course) the computation always hangs. From the last print-out I see that it usually hangs after about ~300 to ~3000 seconds. I have automatic suspend/hibernate turned off on my computer. This issue doesn't occur if a local kernel is used instead.

Might that be related to a timeout of the SSH tunnels?

[tdaff]

Original comment by Tom Daff (Bitbucket: tdaff, GitHub: tdaff).

---

I've tried to replicate this but haven't managed so far. I ran a process overnight on an ssh kernel with the browser not in the foreground, but it was still writing output in the notebook in the morning. The tunnel should be happily restarting, as it did for me, but it seems as though that could be where the problem is for you.

If you encounter this issue again there are few things to check:

• If you add --verbose to the manage --add command it will make a kernel that produces more logging output (0.4.6 will also log output from tunnel commands). Was there anything in the logs in previous times about restarting kernels?
• When it is hung, do you still see the tunnel process in ps auxf. It should be attached to the kernel in the 'forest' view and it will have lots of -L host:port:host:port in the command.
• Which browser are you using? There is nothing that would be suspending the contents of tabs?