ob-session-async-R does not work as expected on macOS

[ywwry66]

Hi there, I just installed your package and tried the toy example

#+begin_src R :async :session
  Sys.sleep(5)
  "this won't hang your emacs"
#+end_src

However, I could not get the expected result, instead the following string was created:
/var/folders/3n/s_f0q14x2rs1s4kb2lph7v140000gn/T/babel-jdvNNF/R-iZ9RJs

I use emacs 27.1 and latest org-mode downloaded from https://orgmode.org/elpa/. Do you have any idea what could possibly go wrong?

Thank you.

[jackkamm]

So that string is the tmpfile where the result should be written, and is inserted as placeholder. When the result is ready after 5 seconds, it should replace that string.

Some questions:

• What OS and version are you using?
• Are you using any special emacs configuration like spacemacs?
• If you try to open the tmpfile what happens? Does it exist, and is the result there?
• Is there anything of note in the *Messages* or *R* buffers after you execute the block?
• Have you tried with emacs -Q and a minimal config?

My guess is that there's something wrong with how your Emacs tries to create tmp files on your OS, possibly to do with permissions. If you're using spacemacs perhaps this issue is related? syl20bnr/spacemacs#11409

[ywwry66]

• Previously I was on macOS Catalina, and now on Big Sur. Failed on both. However, I tried on Windows 10 today, it worked just fine.
• I used the vanilla Emacs installed by brew cask install emacs
• The file opens and contains the string "this won't hang your emacs". Its mode is -rw-r--r--. The parent directory is in mode drwx------
• Nothing suspicious in *Messages* and *R* buffers
• Yes I tried with emacs -Q --no-init and evaluating the most basic commands for package.el org.el and ob-session-async-R.el

[ywwry66]

One thing to add: when I C-c C-c on the code block, Emacs actually hangs for 5 seconds and the tmpfile path is inserted after this 5s period (not as a placeholder as I can observe in the Windows 10 test).

[jackkamm]

Hmm. I'm not on macOS so it is difficult for me to check this. But I know of other macOS users who have successfully used this project in the past.

One thought comes to mind: if you have time, could you try using non-brew Emacs and see if the problem persists?

One thing to add: when I C-c C-c on the code block, Emacs actually hangs for 5 seconds and the tmpfile path is inserted after this 5s period (not as a placeholder as I can observe in the Windows 10 test).

This is an interesting set of symptoms and I think there are some clues here. But I haven't figured them out yet. I will ponder it.

[ywwry66]

There is no official binary for macOS on Emacs download page. But I have also tried emacs-mac, with which I built from source on my machine, and it still hanged in the same way as the Brewed Emacs.

I will look into your source code when I have time.

[ywwry66]

This is more complicated than I thought. There are actually two separate issues, 1. result is not inserted from tmp-file to org buffer, and 2. emacs hangs during evaluation.

The first issue can be easily resolved:

Parse tmp-file name correctly on macOS

The tmp-file path is in such format on macOS:
"/var/folders/3n/s_f0q14x2rs1s4kb2lph7v140000gn/T/babel-jdvNNF/R-iZ9RJs"
The original ob-session-async-indicator would break the string
"ob_comint_async_R_/var/folders/3n/s_f0q14x2rs1s4kb2lph7v140000gn/T/babel-jdvNNF/R-iZ9RJs"
into "ob_comint_async_R_/var/folders/3n/s" and
"f0q14x2rs1s4kb2lph7v140000gn/T/babel-jdvNNF/R-iZ9RJs", causing
ob-session-async-filter unable to find the tmp-file.

1 file changed, 1 insertion(+), 1 deletion(-)
lisp/ob-session-async-R.el | 2 +-

modified   lisp/ob-session-async-R.el
@@ -55,7 +55,7 @@ Returns a placeholder string for insertion, to later be replaced
 by `ob-session-async-filter'."
   (ob-session-async-register
    session (current-buffer)
-   "^\\(?:[>.+] \\)*\\[1\\] \"ob_comint_async_R_\\(.+\\)_\\(.+\\)\"$"
+   "^\\(?:[>.+] \\)*\\[1\\] \"ob_comint_async_R_\\(.+?\\)_\\(.+\\)\"$"
    'org-babel-chomp
    'ob-session-async-R-value-callback)
   (cl-case result-type

The second issue is really weird. I compiled the development version of R and Emacs on my local machine, and the issue still persists. Although I don't understand the reason behind this, it seems that the R code inserted into temp-buffer

ob-session-async/lisp/ob-session-async-R.el

Lines 71 to 83 in 0d05267

	(insert
	(mapconcat
	'org-babel-chomp
	(list (format org-babel-R-write-object-command
	(if row-names-p "TRUE" "FALSE")
	(if column-names-p
	(if row-names-p "NA" "TRUE")
	"FALSE")
	".Last.value"
	(org-babel-process-file-name tmp-file 'noquote))
	(format ob-session-async-R-indicator
	"file" tmp-file))
	"\n"))

is too complicated such that R (or ESS) is overwhelmed. So I cleaned up superfluous spaces and line breaks in
org-babel-R-write-object-command, and it seems to be working now. I hope people using macOS can at least test if the hanging issue ever exists on their system.

Fix the issue that ob-session-async hangs emacs on macOS

Clean up superfluous spaces and linebreaks in
org-babel-R-write-object-command, which fixes the hanging issue on
macOS. But I don't know the reason behind this.

1 file changed, 24 insertions(+)
lisp/ob-session-async-R.el | 24 ++++++++++++++++++++++++

modified   lisp/ob-session-async-R.el
@@ -117,6 +117,30 @@ comint buffers used for asynchronous Babel evaluation."
 	 (org-babel-pick-name
 	  (cdr (assq :colname-names params)) colnames-p)))))
 
+(defconst org-babel-R-write-object-command-macos
+  "{function(object,transfer.file) {
+    object
+    invisible(
+        if (inherits(try({tfile<-tempfile()
+            write.table(object, file=tfile, sep=\"\\t\",
+                        na=\"nil\",row.names=%s,col.names=%s,
+                        quote=FALSE)
+            file.rename(tfile,transfer.file)},
+            silent=TRUE),
+            \"try-error\"))
+        {if(!file.exists(transfer.file))
+             file.create(transfer.file)})
+}}(object=%s,transfer.file=\"%s\")")
+
+(defun ob-session-async-org-babel-R-evaluate-session-macos (fun &rest args)
+  "Advise `ob-session-async-org-babel-R-evaluate-session' to use `org-babel-R-write-object-command-macos'"
+  (let ((org-babel-R-write-object-command org-babel-R-write-object-command-macos))
+    (apply fun args)))
+
+(when (eq system-type 'darwin)
+  (advice-add 'ob-session-async-org-babel-R-evaluate-session
+	      :around #'ob-session-async-org-babel-R-evaluate-session-macos))
+
 (provide 'ob-session-async-R)
 
 ;;; ob-session-async-R.el ends here

[jackkamm]

Thanks for digging into this!

Regarding issue 1, it's certainly a problem that the regex is not robust to underscore in the tempfile name. Thanks for identifying this and providing a fix. I've pushed 3c5edd1 to fix it (also in the Python and Ruby implementations).

Issue 2 is indeed bizarre. It would be good to understand why this is happening, and to see if we should push the fix upstream into org-mode itself. I wonder, if you paste the code into a regular R buffer and evaluate it with ess-eval-buffer, does the spacing still affect whether the code breaks?