Pythoscope fails on non-ASCII python files

[onnokort]

Hi,

first of all, Pythoscope looks like an awesome idea, a pity that it seems to have been unmaintained for so long. I hope I am submitting bugs at the right location here, as the launchpad bug tracker seems to be seriously out of date?

While trying it out, I noticed it fails on encountering files with a non-ASCII encoding, or at least one with an UTF-8 encoding.

Here is an example file:
test123.py

Error message is:

INFO: Inspecting module test123.py.
ERROR: Oops, it seems that an internal Pythoscope error occurred. Please file a bug report at https://bugs.launchpad.net/pythoscope

Traceback (most recent call last):
File "/home/xxx/.local/bin/pythoscope", line 11, in
sys.exit(main())
File "/home/xxx/.local/lib/python2.7/site-packages/pythoscope/init.py", line 184, in main
init_project(project_path)
File "/home/xxx/.local/lib/python2.7/site-packages/pythoscope/init.py", line 112, in init_project
inspect_project_statically(project)
File "/home/xxx/.local/lib/python2.7/site-packages/pythoscope/inspector/init.py", line 55, in inspect_project_statically
return add_and_update_modules(project) +
File "/home/xxx/.local/lib/python2.7/site-packages/pythoscope/inspector/init.py", line 37, in add_and_update_modules
static.inspect_module(project, modpath)
File "/home/xxx/.local/lib/python2.7/site-packages/pythoscope/inspector/static.py", line 130, in inspect_module
return inspect_code(project, path, read_file_contents(path))
File "/home/xxx/.local/lib/python2.7/site-packages/pythoscope/inspector/static.py", line 135, in inspect_code
tree = parse(code)
File "/home/xxx/.local/lib/python2.7/site-packages/pythoscope/astbuilder.py", line 82, in parse
result = drv.parse_string(code, True)
File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/usr/lib/python2.7/lib2to3/pgen2/driver.py", line 60, in parse_tokens
prefix += value
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 2: ordinal not in range(128)

[franzhaas]

Hi.

I just tried your example, and it works on my machine. I used current master and Python 2.7.10, on linux What version did you use?

Best regards,
Franz

[onnokort]

Here are my versions:

$ python --version
Python 2.7.6
$ pythoscope --version
pythoscope 0.4.3

I now just updated to git master with:
pip install --user git+https://github.com/mkwiatkowski/pythoscope

Now pythoscope --version says:
pythoscope 0.5dev

The result is the same, however.

This is a Ubuntu 14.04.4 LTS system, 64bit.

[franzhaas]

I installed with
git clone https://github.com/mkwiatkowski/pythoscope
cd pythoscope/
python setup.py install

and it worked for me.

I tried the same method you used and it failed just like on your machine.

Currently no idea why. Maybe you want to try that method as well.