Installation Error

[lliligabriel]

Greetings!

I am installing the oktoberfest package manually.

It seems that some dependencies of oktoberfest require different versions of pyarrow, ie., installing different dependencies involve installing/uninstalling pyarrow==19.0.1 and pyarrow==21.0.0

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
triqler 0.9.0 requires pyarrow<20.0.0,>=19.0.1, but you have pyarrow 21.0.0 which is incompatible.
picked-group-fdr 0.9.0 requires pyarrow<20.0.0,>=19.0.1, but you have pyarrow 21.0.0 which is incompatible.

Furthermore, it seems that the dlomix.interface module has become legacy. Please correct me if I am wrong: I did not find relevant information in the dlomix source code.

File ~/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/predict/predictor.py:21
     18 logger = logging.getLogger(__name__)
     20 if TYPE_CHECKING or importlib.util.find_spec("dlomix"):
---> [21](some-folder/~/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/predict/predictor.py:21)     from .dlomix import DLomix
     23     PredictionInterface = Union[DLomix, Koina, ZeroPredictor]
...
---> [34](some-folder/~/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/predict/dlomix.py:34)     from dlomix.interface import download_model_from_github, load_keras_model, process_dataset, save_keras_model
     35     from dlomix.refinement_transfer_learning.automatic_rl_tl import AutomaticRlTlTraining, AutomaticRlTlTrainingConfig
     37 gpus = tf.config.list_physical_devices("GPU")

ModuleNotFoundError: No module named 'dlomix.interface'

This causes a broken import statement.

Thank you so much!

[lliligabriel]

I think when I installed via the bash statement, it reported failure to install the Thermo raw file parser, and forced re-install pyarrow==21.0.0

I tried to install dlomix again just now, and below it forced re-install pyarrow==21.0.0

pip install dlomix
Installing collected packages: pyarrow
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 19.0.1
    Uninstalling pyarrow-19.0.1:
      Successfully uninstalled pyarrow-19.0.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
triqler 0.9.0 requires pyarrow<20.0.0,>=19.0.1, but you have pyarrow 21.0.0 which is incompatible.
picked-group-fdr 0.9.0 requires pyarrow<20.0.0,>=19.0.1, but you have pyarrow 21.0.0 which is incompatible.
Successfully installed pyarrow-21.0.0

If I try to manually install pyarrow==19.0.1, it reports:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 4.1.1 requires pyarrow>=21.0.0, but you have pyarrow 19.0.1 which is incompatible.
Successfully installed pyarrow-19.0.1

Thanks!

[picciama]

Hi @lliligabriel,

please let me know which operating system, python version, dependency manager, and oktoberfest version (or branch) you are using to install. I need to investigate this. At the moment, dlomix integration is still in beta, and we are actively developing the interface, unfortunately, other things had to be prioritized.

[lliligabriel]

Thank you so much!

OS:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian

Python version:

$ python --version
Python 3.11.0

Dependency manager: I did not intentionally use any dependency managers. I followed the documentation of managing the Oktoberfest environment with conda.

Oktoberfest version:

pip show oktoberfest
Name: oktoberfest
Version: 0.10.0

Thank you so much! I really appreciate your time:)

[picciama]

The pyarrow issue arises from picked-group-fdr==0.9.0 having triqler as a dependency, which requires pyarrow <20.0.0 but dlomix is installing datasets==4.1.1 which requires pyarrow>=21.0.0. triqler does not have a new release, so we cannot update this in picked-group-fdr. @MatthewThe @omsh please advise. It seems dlomix is using no version constraints at the moment installing the latest packages(?) in setup.py.

@lliligabriel For me using pip install oktoberfest dlomix works in a clean mamba / conda environment with python=3.11. This will ensure datasets==4.0.0 which is working with pyarrow <20.0.0

[picciama]

I think when I installed via the bash statement, it reported failure to install the Thermo raw file parser
Which bash statement are you referring to? What error message do you get?

[lliligabriel]

Thank you so much!
I cannot replicate the message I have right now - the pip install oktoberfest dlomix works! I think it was originally regarding ThermoRawFileParser.

However, the package still cannot be run correctly, due to the missing dlomix.interface module.

In command line:

$ oktoberfest --help
Traceback (most recent call last):
  File "/new_home/xinyuegu/miniforge3/envs/oktoberfest/bin/oktoberfest", line 3, in <module>
    from oktoberfest.__main__ import main
  File "/new_home/xinyuegu/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/__init__.py", line 14, in <module>
    from oktoberfest import predict as pr
  File "/new_home/xinyuegu/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/predict/__init__.py", line 6, in <module>
    from .predictor import Predictor
  File "/new_home/xinyuegu/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/predict/predictor.py", line 21, in <module>
    from .dlomix import DLomix
  File "/new_home/xinyuegu/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/predict/dlomix.py", line 34, in <module>
    from dlomix.interface import download_model_from_github, load_keras_model, process_dataset, save_keras_model

In python import:

Version:1.0 StartHTML:0000000105 EndHTML:0000012522 StartFragment:0000000121 EndFragment:0000012493 ----> 1 import oktoberfest as ok
      2 import json

File ~/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/__init__.py:14
     11 import sys
     13 from oktoberfest import plotting as pl
---> 14 from oktoberfest import predict as pr
     15 from oktoberfest import preprocessing as pp
     16 from oktoberfest import rescore as re

File ~/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/predict/__init__.py:6
      3 import importlib
      5 from .koina import Koina
----> 6 from .predictor import Predictor
      8 if importlib.util.find_spec("dlomix"):
      9     from .dlomix import DLomix, create_dlomix_dataset, refine_intensity_predictor

File ~/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/oktoberfest/predict/predictor.py:21
     18 logger = logging.getLogger(__name__)
     20 if TYPE_CHECKING or importlib.util.find_spec("dlomix"):
---> 21     from .dlomix import DLomix
     23     PredictionInterface = Union[DLomix, Koina, ZeroPredictor]
...
---> 34     from dlomix.interface import download_model_from_github, load_keras_model, process_dataset, save_keras_model
     35     from dlomix.refinement_transfer_learning.automatic_rl_tl import AutomaticRlTlTraining, AutomaticRlTlTrainingConfig
     37 gpus = tf.config.list_physical_devices("GPU")

ModuleNotFoundError: No module named 'dlomix.interface'

I really appreciate your inputs!

[MatthewThe]

I will create a new triqler release without the upper version constraint on pyarrow.

[picciama]

I will create a new triqler release without the upper version constraint on pyarrow.

Cheers Matthew

ModuleNotFoundError: No module named 'dlomix.interface'

This can also be an artifact of resolving python dependencies. By any chance, do you execute oktoberfest imports from a folder that contains a dlomix subfolder?
Python checks for packages in the current working directory first, and treats them as an importable module, i.e. this line:

20 if TYPE_CHECKING or importlib.util.find_spec("dlomix"):

will return true if you have a dlomix folder in your current working directory (that, of course, doesn't have the expected modules).
If you don't need dlomix, then this is actually a misleading error message and installing dlomix won't resolve the error since python still prefers the subfolder to a package installed in your site-packages folder in your environment.

I will add a warning in the next release, asking users to check if they have a dlomix subfolder and to delete it first.

[MatthewThe]

I just release triqler v0.9.1 which no longer has the upper version limit on pyarrow

[lliligabriel]

Thank you so so much Matthew:)))!!!

and Greetings @picciama!
I am executing oktoberfest directly from the oktoberfest conda environment - ie., my root directory in the server. So no, there is no dlomix subfolder in this root server, and the dlomix that Python may try to use is indeed ~/miniforge3/envs/oktoberfest/lib/python3.11/site-packages/dlomix (pip show dlomix)

I tried to copy and paste the dlomix folder into my root directory (not the best practice) and re-ran the import statement, but still the same error message.

[picciama]

@omsh how far are we with the integration? I can reproduce the error. This is likely because the required changes are not ready yet? The branch in dlomix where refinement learning was done with is likely https://github.com/wilhelm-lab/dlomix/tree/feature/bmpc

@lliligabriel I am not sure if this is working though... As I said, it is in beta and hence, we haven't yet documented it. We need to restructure some things but @omsh is likely the best person to ask for support concerning dlomix.

[lliligabriel]

Thanks! For now, are you knowledgeable of whether there could be a way that I could use Oktoberfest without dlomix? (Ie., importing Oktoberfest successfully, circumventing the use of dlomix for now)

[picciama]

Of course, you can uninstall dlomix again. As long as it isn't installed, python doesn't find it and won't attempt to import it.
Worst case, make a new clean conda environment where you only use pip install oktoberfest

[lliligabriel]

Thanks! Manually removed dlomix and the import statement worked for now. I will let you know about my progress:)) Have a lovely weekend!