json-read: JSON readtable error with python 3.6.3

[lengyueyang]

We I use python 3.6.2 and 3.6.2, when execute, occur json-read: JSON readtable error, but when use python 3.5.2, all is ok

[dcherian]

I had this problem and had to downgrade.

Can we have a section in README that mentions the latest version of python this has been tested on?

[andrewsanchez]

Neither downgrading with conda install python=3.5 (resulted in 3.5.4) nor conda install python=3.5.2 worked for me.

[dcherian]

Oh I meant downgrade ob-ipython. I haven't tried with python 3.5

[dcherian]

Can someone who's using the latest ob-ipython please comment with versions of python / ipython / jupyter they're using?

[nicholas-miklaucic]

I think I figured this out, at least on my system (latest version of ob-ipython, ipython version 6.1.0, python version 3.6.2 with Anaconda). I got this error too, and I fixed it by using exec-path-from-shell to set the PATH correctly. Install that from MELPA and put this in your init file before loading ob-ipython:

(when (memq window-system '(mac ns x))
  (exec-path-from-shell-initialize)) 

I think the reason this happens is because it uses jupyter to get a JSON output of the kernels, but it runs that via shell-command, which doesn't use the exec-path variable the rest of Emacs uses. You should be good if M-! jupyter -h returns a normal help entry and not zsh:1:command not found: 'jupyter' or equivalent.

This is probably a documentation problem moreso than a code problem, as I'm not sure how you're supposed to be able to divine where the jupyter executable is in elisp.

[dcherian]

Hmm... I've always had exec-path-from-shell configured and running. M-! jupyter -h returns a normal help entry too.

Is there a way to see the string json-read is having trouble decoding? My elisp isn't good enough to figure it out.

[nicholas-miklaucic]

We must have different errors with the same error message. Can you try invoking toggle-debug-on-errorand then running it, or if that doesn't work trying toggle-debug-on-entry with json-read?

[dcherian]

@nicholas-miklaucic Thanks!

Using debug-on-entry showed this (newlines added for clarity):

  json-read-from-string("/home/deepak/.bashrc: line 115: bind: warning: line editing not enabled\n
{\n 
 \"kernelspecs\": {\n    \"python3\": {\n     
 \"resource_dir\": \"/home/deepak/anaconda3/share/jupyter/kernels/python3\",\n      \"spec\": {\n        \"argv\": [\n         
 \"/home/deepak/anaconda3/bin/python\",\n          \"-m\",\n          \"ipykernel_launcher\",\n          \"-f\",\n          \"{connection_file}\"\n        ],\n        \"env\": {},\n       
 \"display_name\": \"Python 3\",\n        \"language\": \"python\"\n      }\n    }\n  }\n}\n")

Once I comment the line that makes bash throw a warning, things work!

@gregsexton Maybe a regexp to select the bit that begins with {\n and ends with \n") before passing it to json-read would do the trick? Perhaps you could catch the error and display the string so it's clear what json-read is choking on?

[georgek]

This isn't working at all for me. Using Python 3.6.2 and getting the same JSON read table error. This is me trying to run the example:

Debugger entered--Lisp error: (json-readtable-error)

  signal(json-readtable-error nil)
  json-read()
  (cons (json-read) acc)
  (setq acc (cons (json-read) acc))
  (while (not (= (point) (point-max))) (setq acc (cons (json-read) acc)) (forward-line))
  (let (acc) (while (not (= (point) (point-max))) (setq acc (cons (json-read) acc)) (forward-line)) (nreverse acc))
  (let ((json-array-type (quote list))) (let (acc) (while (not (= (point) (point-max))) (setq acc (cons (json-read) acc)) (forward-line)) (nreverse acc)))
  ob-ipython--collect-json()
  (if (> ret 0) (ob-ipython--dump-error (buffer-string)) (goto-char (point-min)) (ob-ipython--collect-json))
  (let ((ret (apply (quote call-process-region) code nil (ob-ipython--get-python) nil t nil (list "--" ob-ipython-client-path "--conn-file" name "--execute")))) (if (> ret 0) (ob-ipython--dump-error (buffer-string)) (goto-char (point-min)) (ob-ipython--collect-json)))
  (progn (let ((ret (apply (quote call-process-region) code nil (ob-ipython--get-python) nil t nil (list "--" ob-ipython-client-path "--conn-file" name "--execute")))) (if (> ret 0) (ob-ipython--dump-error (buffer-string)) (goto-char (point-min)) (ob-ipython--collect-json))))
  (unwind-protect (progn (let ((ret (apply (quote call-process-region) code nil (ob-ipython--get-python) nil t nil (list "--" ob-ipython-client-path "--conn-file" name "--execute")))) (if (> ret 0) (ob-ipython--dump-error (buffer-string)) (goto-char (point-min)) (ob-ipython--collect-json)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((ret (apply (quote call-process-region) code nil (ob-ipython--get-python) nil t nil (list "--" ob-ipython-client-path "--conn-file" name "--execute")))) (if (> ret 0) (ob-ipython--dump-error (buffer-string)) (goto-char (point-min)) (ob-ipython--collect-json)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))
  (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (let ((ret (apply ... code nil ... nil t nil ...))) (if (> ret 0) (ob-ipython--dump-error (buffer-string)) (goto-char (point-min)) (ob-ipython--collect-json)))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer)))))
  ob-ipython--execute-request("%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np" "default")
  (ob-ipython--eval (ob-ipython--execute-request (org-babel-expand-body:generic (encode-coding-string body (quote utf-8)) params (org-babel-variable-assignments:python params)) (ob-ipython--normalize-session session)))
  (let ((ret (ob-ipython--eval (ob-ipython--execute-request (org-babel-expand-body:generic (encode-coding-string body (quote utf-8)) params (org-babel-variable-assignments:python params)) (ob-ipython--normalize-session session))))) (if ret (progn (ob-ipython--process-response ret file result-type))))
  (let* ((file (cdr (assoc :ipyfile params))) (session (cdr (assoc :session params))) (result-type (cdr (assoc :result-type params)))) (ob-ipython--create-kernel (ob-ipython--normalize-session session) (cdr (assoc :kernel params))) (let ((ret (ob-ipython--eval (ob-ipython--execute-request (org-babel-expand-body:generic (encode-coding-string body ...) params (org-babel-variable-assignments:python params)) (ob-ipython--normalize-session session))))) (if ret (progn (ob-ipython--process-response ret file result-type)))))
  ob-ipython--execute-sync("%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np" ((:colname-names) (:rowname-names) (:result-params "replace" "drawer") (:result-type . value) (:results . "replace drawer") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:session)))
  (if (cdr (assoc :async params)) (ob-ipython--execute-async body params) (ob-ipython--execute-sync body params))
  org-babel-execute:ipython("%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np" ((:colname-names) (:rowname-names) (:result-params "replace" "drawer") (:result-type . value) (:results . "replace drawer") (:exports . "code") (:cache . "no") (:noweb . "no") (:hlines . "no") (:tangle . "no") (:session)))
  org-babel-execute-src-block(nil ("ipython" "%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np" ((:colname-names) (:rowname-names) (:result-params "drawer" "replace") (:result-type . value) (:results . "drawer replace") (:exports . "code") (:session) (:tangle . "no") (:hlines . "no") (:noweb . "no") (:cache . "no")) "" nil 714 "(ref:%s)"))
  org-ctrl-c-ctrl-c(nil)
  funcall-interactively(org-ctrl-c-ctrl-c nil)
  call-interactively(org-ctrl-c-ctrl-c nil nil)
  command-execute(org-ctrl-c-ctrl-c)

There are also seem to be problems with it killing the jupyter process. I have to kill it manually otherwise I get errors from ZMQ saying the port is already in use.

[nicholas-miklaucic]

Try invoking toggle-debug-on-entry with json-read to see what it's choking on: your error seems different as it's for running code, AFAIK.

[dcherian]

You'll also want to make sure that jupyter isn't writing any warnings also. I had to remove those too.

[georgek]

I get a buffer called *ob-ipython-kernel-default* with some output from starting a jupyter console. It seems that json-read is being run on this buffer which doesn't contain anything that looks like JSON:

Jupyter console 5.2.0

Python 3.6.2 (default, Sep 28 2017, 13:58:54) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.2.0 -- An enhanced Interactive Python. Type '?' for help.



In [1]: 

[dcherian]

@georgek That's how it looks like when it works.

Set debug-on-entry to stop at ob-ipython--collect-json. When it does hit c to continue, do this twice and it should show the string it's trying to read. That'll tell you what it's choking on

EDIT: Not sure that will work actually but you can try. I found the second answer here helpful: https://stackoverflow.com/questions/3257480/how-to-debug-elisp#3258205

[georgek]

I've tried using edebug and stepping through. The difficulty seems to be that in ob-ipython--execute-request it runs ob-ipython--collect-json in a temporary buffer. I can't figure out how to look at what's being written in the temp buffer, though.

[georgek]

OK, I finally figured it out. It's not easy to debug something that uses with-temp-buffer, but eventually I figured out I could manually repalce that with a real buffer and see the output. Turns out my ipython prints an error about being run in a virtualenv which is not proper JSON. So now I need to figure out how to suppress that warning (which I think is safe).

I succumbed to installing jupyter/ipython within each virtualenv and now there is no error and it works.