pip install fails on Ubuntu 18.04 under freshly installed Anaconda3

[johann-petrak]

After installing Anaconda3-2019.07-Linux-x86_64.sh for a new user, directly running pip install DAWG fails with the following:

  Building wheel for DAWG (setup.py) ... error
  ERROR: Complete output from command /home/conda/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-_lu0pouq/DAWG/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-k8u1ivdk --python-tag cp37:
  ERROR: running bdist_wheel
  running build
  running build_ext
  building 'dawg' extension
  creating build
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/src
  creating build/temp.linux-x86_64-3.7/lib
  creating build/temp.linux-x86_64-3.7/lib/b64
  gcc -pthread -B /home/conda/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ilib -I/home/conda/anaconda3/include/python3.7m -c src/dawg.cpp -o build/temp.linux-x86_64-3.7/src/dawg.o
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
  In file included from src/dawg.cpp:266:0:
  src/../lib/dawgdic/dictionary-builder.h: In member function ‘bool dawgdic::DictionaryBuilder::BuildDictionary(dawgdic::BaseType, dawgdic::BaseType)’:
  src/../lib/dawgdic/dictionary-builder.h:138:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
       if (dawg_.is_merging(dawg_child_index))
       ^~
  src/../lib/dawgdic/dictionary-builder.h:139:53: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
         link_table_.Insert(dawg_child_index, offset); {
                                                       ^
  src/dawg.cpp: In function ‘PyObject* __pyx_f_4dawg_9BytesDAWG_items(__pyx_obj_4dawg_BytesDAWG*, int, __pyx_opt_args_4dawg_9BytesDAWG_items*)’:
  src/dawg.cpp:11011:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       for (__pyx_t_10 = 0; __pyx_t_10 < __pyx_t_9; __pyx_t_10+=1) {
                            ~~~~~~~~~~~^~~~~~~~~~~
  src/dawg.cpp: In function ‘PyObject* __pyx_gb_4dawg_9BytesDAWG_24generator2(__pyx_GeneratorObject*, PyObject*)’:
  src/dawg.cpp:11485:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       for (__pyx_t_6 = 0; __pyx_t_6 < __pyx_t_5; __pyx_t_6+=1) {
                           ~~~~~~~~~~^~~~~~~~~~~
  src/dawg.cpp: In function ‘PyObject* __pyx_f_4dawg_9BytesDAWG_keys(__pyx_obj_4dawg_BytesDAWG*, int, __pyx_opt_args_4dawg_9BytesDAWG_keys*)’:
  src/dawg.cpp:11814:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       for (__pyx_t_10 = 0; __pyx_t_10 < __pyx_t_9; __pyx_t_10+=1) {
                            ~~~~~~~~~~~^~~~~~~~~~~
  src/dawg.cpp: In function ‘PyObject* __pyx_gb_4dawg_9BytesDAWG_29generator3(__pyx_GeneratorObject*, PyObject*)’:
  src/dawg.cpp:12222:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
       for (__pyx_t_6 = 0; __pyx_t_6 < __pyx_t_5; __pyx_t_6+=1) {
                           ~~~~~~~~~~^~~~~~~~~~~
  src/dawg.cpp: In function ‘int __Pyx_GetException(PyObject**, PyObject**, PyObject**)’:
  src/dawg.cpp:21523:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
       tmp_type = tstate->exc_type;
                          ^~~~~~~~
                          curexc_type
  src/dawg.cpp:21524:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
       tmp_value = tstate->exc_value;
                           ^~~~~~~~~
                           curexc_value
  src/dawg.cpp:21525:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
       tmp_tb = tstate->exc_traceback;
                        ^~~~~~~~~~~~~
                        curexc_traceback
  src/dawg.cpp:21526:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
       tstate->exc_type = local_type;
               ^~~~~~~~
               curexc_type
  src/dawg.cpp:21527:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
       tstate->exc_value = local_value;
               ^~~~~~~~~
               curexc_value
  src/dawg.cpp:21528:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
       tstate->exc_traceback = local_tb;
               ^~~~~~~~~~~~~
               curexc_traceback
  src/dawg.cpp: In function ‘void __Pyx_ExceptionSwap(PyObject**, PyObject**, PyObject**)’:
  src/dawg.cpp:21550:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
       tmp_type = tstate->exc_type;
                          ^~~~~~~~
                          curexc_type
  src/dawg.cpp:21551:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
       tmp_value = tstate->exc_value;
                           ^~~~~~~~~
                           curexc_value
  src/dawg.cpp:21552:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
       tmp_tb = tstate->exc_traceback;
                        ^~~~~~~~~~~~~
                        curexc_traceback
  src/dawg.cpp:21553:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
       tstate->exc_type = *type;
               ^~~~~~~~
               curexc_type
  src/dawg.cpp:21554:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
       tstate->exc_value = *value;
               ^~~~~~~~~
               curexc_value
  src/dawg.cpp:21555:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
       tstate->exc_traceback = *tb;
               ^~~~~~~~~~~~~
               curexc_traceback
  src/dawg.cpp: In function ‘void __Pyx_ExceptionSave(PyObject**, PyObject**, PyObject**)’:
  src/dawg.cpp:21568:21: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
       *type = tstate->exc_type;
                       ^~~~~~~~
                       curexc_type
  src/dawg.cpp:21569:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
       *value = tstate->exc_value;
                        ^~~~~~~~~
                        curexc_value
  src/dawg.cpp:21570:19: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
       *tb = tstate->exc_traceback;
                     ^~~~~~~~~~~~~
                     curexc_traceback
  src/dawg.cpp: In function ‘void __Pyx_ExceptionReset(PyObject*, PyObject*, PyObject*)’:
  src/dawg.cpp:21582:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
       tmp_type = tstate->exc_type;
                          ^~~~~~~~
                          curexc_type
  src/dawg.cpp:21583:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
       tmp_value = tstate->exc_value;
                           ^~~~~~~~~
                           curexc_value
  src/dawg.cpp:21584:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
       tmp_tb = tstate->exc_traceback;
                        ^~~~~~~~~~~~~
                        curexc_traceback
  src/dawg.cpp:21585:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
       tstate->exc_type = type;
               ^~~~~~~~
               curexc_type
  src/dawg.cpp:21586:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
       tstate->exc_value = value;
               ^~~~~~~~~
               curexc_value
  src/dawg.cpp:21587:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
       tstate->exc_traceback = tb;
               ^~~~~~~~~~~~~
               curexc_traceback
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for DAWG
  Running setup.py clean for DAWG
Failed to build DAWG
Installing collected packages: DAWG
  Running setup.py install for DAWG ... error
    ERROR: Complete output from command /home/conda/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-_lu0pouq/DAWG/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-6jwkhmli/install-record.txt --single-version-externally-managed --compile:
    ERROR: running install
    running build
    running build_ext
    building 'dawg' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/src
    creating build/temp.linux-x86_64-3.7/lib
    creating build/temp.linux-x86_64-3.7/lib/b64
    gcc -pthread -B /home/conda/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ilib -I/home/conda/anaconda3/include/python3.7m -c src/dawg.cpp -o build/temp.linux-x86_64-3.7/src/dawg.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    In file included from src/dawg.cpp:266:0:
    src/../lib/dawgdic/dictionary-builder.h: In member function ‘bool dawgdic::DictionaryBuilder::BuildDictionary(dawgdic::BaseType, dawgdic::BaseType)’:
    src/../lib/dawgdic/dictionary-builder.h:138:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation]
         if (dawg_.is_merging(dawg_child_index))
         ^~
    src/../lib/dawgdic/dictionary-builder.h:139:53: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the ‘if’
           link_table_.Insert(dawg_child_index, offset); {
                                                         ^
    src/dawg.cpp: In function ‘PyObject* __pyx_f_4dawg_9BytesDAWG_items(__pyx_obj_4dawg_BytesDAWG*, int, __pyx_opt_args_4dawg_9BytesDAWG_items*)’:
    src/dawg.cpp:11011:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         for (__pyx_t_10 = 0; __pyx_t_10 < __pyx_t_9; __pyx_t_10+=1) {
                              ~~~~~~~~~~~^~~~~~~~~~~
    src/dawg.cpp: In function ‘PyObject* __pyx_gb_4dawg_9BytesDAWG_24generator2(__pyx_GeneratorObject*, PyObject*)’:
    src/dawg.cpp:11485:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         for (__pyx_t_6 = 0; __pyx_t_6 < __pyx_t_5; __pyx_t_6+=1) {
                             ~~~~~~~~~~^~~~~~~~~~~
    src/dawg.cpp: In function ‘PyObject* __pyx_f_4dawg_9BytesDAWG_keys(__pyx_obj_4dawg_BytesDAWG*, int, __pyx_opt_args_4dawg_9BytesDAWG_keys*)’:
    src/dawg.cpp:11814:37: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         for (__pyx_t_10 = 0; __pyx_t_10 < __pyx_t_9; __pyx_t_10+=1) {
                              ~~~~~~~~~~~^~~~~~~~~~~
    src/dawg.cpp: In function ‘PyObject* __pyx_gb_4dawg_9BytesDAWG_29generator3(__pyx_GeneratorObject*, PyObject*)’:
    src/dawg.cpp:12222:35: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
         for (__pyx_t_6 = 0; __pyx_t_6 < __pyx_t_5; __pyx_t_6+=1) {
                             ~~~~~~~~~~^~~~~~~~~~~
    src/dawg.cpp: In function ‘int __Pyx_GetException(PyObject**, PyObject**, PyObject**)’:
    src/dawg.cpp:21523:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
         tmp_type = tstate->exc_type;
                            ^~~~~~~~
                            curexc_type
    src/dawg.cpp:21524:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
         tmp_value = tstate->exc_value;
                             ^~~~~~~~~
                             curexc_value
    src/dawg.cpp:21525:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         tmp_tb = tstate->exc_traceback;
                          ^~~~~~~~~~~~~
                          curexc_traceback
    src/dawg.cpp:21526:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
         tstate->exc_type = local_type;
                 ^~~~~~~~
                 curexc_type
    src/dawg.cpp:21527:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
         tstate->exc_value = local_value;
                 ^~~~~~~~~
                 curexc_value
    src/dawg.cpp:21528:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         tstate->exc_traceback = local_tb;
                 ^~~~~~~~~~~~~
                 curexc_traceback
    src/dawg.cpp: In function ‘void __Pyx_ExceptionSwap(PyObject**, PyObject**, PyObject**)’:
    src/dawg.cpp:21550:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
         tmp_type = tstate->exc_type;
                            ^~~~~~~~
                            curexc_type
    src/dawg.cpp:21551:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
         tmp_value = tstate->exc_value;
                             ^~~~~~~~~
                             curexc_value
    src/dawg.cpp:21552:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         tmp_tb = tstate->exc_traceback;
                          ^~~~~~~~~~~~~
                          curexc_traceback
    src/dawg.cpp:21553:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
         tstate->exc_type = *type;
                 ^~~~~~~~
                 curexc_type
    src/dawg.cpp:21554:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
         tstate->exc_value = *value;
                 ^~~~~~~~~
                 curexc_value
    src/dawg.cpp:21555:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         tstate->exc_traceback = *tb;
                 ^~~~~~~~~~~~~
                 curexc_traceback
    src/dawg.cpp: In function ‘void __Pyx_ExceptionSave(PyObject**, PyObject**, PyObject**)’:
    src/dawg.cpp:21568:21: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
         *type = tstate->exc_type;
                         ^~~~~~~~
                         curexc_type
    src/dawg.cpp:21569:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
         *value = tstate->exc_value;
                          ^~~~~~~~~
                          curexc_value
    src/dawg.cpp:21570:19: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         *tb = tstate->exc_traceback;
                       ^~~~~~~~~~~~~
                       curexc_traceback
    src/dawg.cpp: In function ‘void __Pyx_ExceptionReset(PyObject*, PyObject*, PyObject*)’:
    src/dawg.cpp:21582:24: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
         tmp_type = tstate->exc_type;
                            ^~~~~~~~
                            curexc_type
    src/dawg.cpp:21583:25: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
         tmp_value = tstate->exc_value;
                             ^~~~~~~~~
                             curexc_value
    src/dawg.cpp:21584:22: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         tmp_tb = tstate->exc_traceback;
                          ^~~~~~~~~~~~~
                          curexc_traceback
    src/dawg.cpp:21585:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_type’; did you mean ‘curexc_type’?
         tstate->exc_type = type;
                 ^~~~~~~~
                 curexc_type
    src/dawg.cpp:21586:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_value’; did you mean ‘curexc_value’?
         tstate->exc_value = value;
                 ^~~~~~~~~
                 curexc_value
    src/dawg.cpp:21587:13: error: ‘PyThreadState {aka struct _ts}’ has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
         tstate->exc_traceback = tb;
                 ^~~~~~~~~~~~~
                 curexc_traceback
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command "/home/conda/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-_lu0pouq/DAWG/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-6jwkhmli/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-_lu0pouq/DAWG/

[osvald-laurits]

This is related to Cython. See mcfletch/pyopengl#11 . Installing dawg in Python2.7 should work.
The patch suggested in #31 fixed this issue for me.

[hzitoun]

This is related to Cython. See mcfletch/pyopengl#11 . Installing dawg in Python2.7 should work.
The patch suggested in #31 fixed this issue for me.

@michaelhochleitner thanks for the feedback, so you are saying that there is no way to make it run in python 3 ? (macos with anaconda)

[osvald-laurits]

@hzitoun No, I'm not saying that.

Here is my tutorial on how to run dawg on python3.7 in ubuntu. Could work in macos too.

1. clone repository
   git clone https://github.com/pytries/DAWG.git
2. edit DAWG/src/dawg.pyx like described in Python 3.7 / Cython 28.5 build fails #31 :
   Remove the string " except +" in line 349. Line number could be wrong but should be approximately right.

line 349 before change:
cpdef bytes tobytes(self) except +:
Zeile 349 after change:
cpdef bytes tobytes(self):

3. execute DAWG/src/update_cpp.sh

4. install package
   Use the python executable that should run dawg.
   python setup.py install

[johann-petrak]

Is there a reason why this fix does not make it into a pypi release so that DAWG can be installed with pip?
It is a serious problem to have other python programs and packages depend on DAWG if installing DAWG requires the above steps instead of just being another dependency that gets installed during pip install. It is an even bigger problem when trying to install that python program in an automated unsupervised way.
That fix looks like a tiny change, would it be harmful to just have it in the code always?
Alternately, the setup.py program could just do the patching easily during setup I think.