Dependency issues in conda environment

[robbenmigacz]

Hello,

I am attempting to install TCR-BERT on Rocky 8. I installed the dependencies given in environment.yml, but the example scripts do not work.

I received the same error as that given in #8: packaging.version.InvalidVersion: Invalid version: '0.10.1,<0.11'. Following the suggestions given in the comments of that issue, I reinstalled the transformers package (to version 4.11.1, as suggested by a commenter). This resolved the issue I was encountering, but the scripts still do not work; now, I get the error ImportError: libffi.so.7: cannot open shared object file: No such file or directory when from neptune.experiments import Experiment runs in the model_utils module. libffi is a part of the conda environment, so I believe this is likely a result of a dependency conflict—likely because I needed to change the versions of some packages to install a compatible version of transformers. Indeed, updating transformers to a compatible version resulted in the installation of conda-forge::libffi-3.4.6-h2dba641_0 over anaconda::libffi-3.3-he6710b0_2. I am now searching for a way to install compatible dependencies, but it is proving to be challenging.

Has anyone experienced a similar issue when attempting to install this package? I believe environment.yml will need to be updated; as-is, the environment is unusable.

Thanks!

[ggruenhagen3]

I faced the same exact issues. I solved them by performing the following steps after creating the environment from the provided yml file.

conda install anaconda::packaging==21.3 
conda install anaconda::importlib_resources
conda install anaconda::libffi
conda install anaconda::cffi

This installed packaging v21.3, importlib_resources v6.4.0, libffi v3.4.4, and cffi v1.17.1.

I found that actually installing a different version of packaging rather than changing the version of transformers was key (I found that solution here). Then to solve the libffi.so.7 ImportError, the libffi package needed to be installed by conda along with cffi (I found the cff solution here).

[imerelli]

I have the some issue.
But the proposed solution is not working for me, because then I have problems with pytorch.
Any hints about this?

(tcrbert) $ python
Python 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:23:11)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from transformers import BertModel

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

Traceback (most recent call last):  File "<stdin>", line 1, in <module>
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/__init__.py", line 2310, in __getattr__
    return super().__getattr__(name)
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/file_utils.py", line 1661, in __getattr__
    value = getattr(module, name)
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/file_utils.py", line 1660, in __getattr__
    module = self._get_module(self._class_to_module[name])
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/models/bert/__init__.py", line 134, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen zipimport>", line 259, in load_module
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/models/bert/modeling_bert.py", line 25, in <module>
    import torch
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/torch/__init__.py", line 196, in <module>
    from torch._C import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/__init__.py", line 2310, in __getattr__
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/file_utils.py", line 1661, in __getattr__
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/file_utils.py", line 1660, in __getattr__
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/models/bert/__init__.py", line 134, in _get_module
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/transformers-4.4.2-py3.8.egg/transformers/models/bert/modeling_bert.py", line 25, in <module>
  File "/opt/tools/deg/miniforge3/envs/tcrbert/lib/python3.9/site-packages/torch/__init__.py", line 196, in <module>
    from torch._C import *
SystemError: X�U�IV() method: bad call flags
>>> 
Segmentation fault (core dumped)
```

[imerelli]

For the future:

pip install numpy==1.20.1
pip install numexpr==2.7.3
pip install numba==0.53.1
pip install tables==3.6.1