Naomi Install fails on master branch/stable version

[aczire]

Since the master branch is still dependent on sklearn, installation fails at python requirements install.

[aaronchantrill]

Yes, the master branch has not been updated in quite a while. I think the workflow idea was to have master be the release version when a new version is released. Did you install it by selecting the stable version from the installer? You are correct that this won't work with the updated scikit-learn library. I'll try a stable install later today and see if I need to fix anything else.

[aczire]

Correct, I tried to install by selecting 'stable' from the installer menu. naomi-dev has the correct scikit package, so it's good.

[aaronchantrill]

I'm checking in with the group on Discord to see how they want to handle that. It's been almost four years since the last update to stable.

[aaronchantrill]

I fixed the sklearn requirement a couple of weeks ago. Now I'm getting this:

Processing triggers for libgdk-pixbuf-2.0-0:arm64 (2.42.10+dfsg-1+deb12u1) ...
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    For more information visit http://rptl.io/venv

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
sourcing virtualenvwrapper.sh
-bash: /home/pi/.local/bin/virtualenvwrapper.sh: No such file or directory
checking if Naomi virtualenv exists
Naomi virtualenv does not exist. Creating.
-bash: mkvirtualenv: command not found
-bash: workon: command not found
Notice: Something went wrong, not in virtual environment... 

I think this is because virtualenvwrapper is being installed by pip, but "sudo pip install ..." does not work anymore in the primary environment. In newer versions of Naomi, we are using apt to install the python3-virtualenvwrapper package, which is fine.

[aaronchantrill]

The virtualenvwrapper issue affects all three types of install, stable, milestone and nightly. I don't understand why it is trying to access /home/pi/.local/bin/virtualenvwrapper.sh. The installer is being explicitly instructed to use /usr/share/virtualenvwrapper/virtualenvwrapper.sh.

There is a line here:

PATH=$PATH:~/.local/bin mkvirtualenv -p $VIRTUALENVWRAPPER_PYTHON Naomi

which may explain it. I'm looking into it and trying to determine what changed, exactly.

[aaronchantrill]

So, I just tried to manually follow the same process that the Naomi installer uses to install and set up virtualenvwrapper on a brand new Raspberry Pi OS Lite (64-bit) running on a Raspberry Pi 5 and it seemed to work fine:

pi@naomi:~ $ export WORKON_HOME=$HOME/.virtualenvs
pi@naomi:~ $ echo $WORKON_HOME
/home/pi/.virtualenvs
pi@naomi:~ $ export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
pi@naomi:~ $ echo $VIRTUALENVWRAPPER_PYTHON
/usr/bin/python3
pi@naomi:~ $ which python
/usr/bin/python
pi@naomi:~ $ which python3
/usr/bin/python3
pi@naomi:~ $ ls -l /usr/bin/python
lrwxrwxrwx 1 root root 7 Jun 17  2024 /usr/bin/python -> python3
pi@naomi:~ $ source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/premkproject
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/postmkproject
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/initialize
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/premkvirtualenv
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/postmkvirtualenv
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/prermvirtualenv
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/postrmvirtualenv
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/predeactivate
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/postdeactivate
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/preactivate
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/postactivate
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/get_env_details
pi@naomi:~ $ PATH=$PATH:~/.local/bin mkvirtualenv -p $VIRTUALENVWRAPPER_PYTHON Naomi
created virtual environment CPython3.11.2.final.0-64 in 215ms
  creator CPython3Posix(dest=/home/pi/.virtualenvs/Naomi, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/pi/.local/share/virtualenv)
    added seed packages: pip==23.0.1, setuptools==66.1.1, wheel==0.38.4
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/Naomi/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/Naomi/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/Naomi/bin/preactivate
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/Naomi/bin/postactivate
virtualenvwrapper.user_scripts creating /home/pi/.virtualenvs/Naomi/bin/get_env_details
(Naomi) pi@naomi:~ $ workon Naomi
(Naomi) pi@naomi:~ $ 

So, I'm a little confused about what exactly is failing here. I'm going to add some additional status messages and try the installer again. The libgtk-pixbuf package mentioned just before the error appears to be a requirement of the libeigen3-dev package which is required for compiling KenLM, and it appears to have installed fine. What appears to fail is the pip install -r line, which should only be called if the workon Naomi call succeeds and after printing the offer to set virtualenvwrapper to automatically start, which is not happening in the output above, so it is confusing what exactly is happening here.

[aaronchantrill]

I just tried installing by cloning the project from github, then running the naomi-setup.sh script and it installed correctly. I saved the copy of naomi-setup.sh downloaded using the link from the one-liner (https://installers.projectnaomi.com/naomi-setup.sh) with the version from naomi-dev, and here are the changes:

$ diff Downloads/naomi-setup.sh Naomi/installers/naomi-setup.sh 
174c174
< 	  printf "${B_W}yum found${NL}"
---
>       printf "${B_W}yum found${NL}"
182c182
< 	  printf "${B_W}apt found${NL}"
---
>       printf "${B_W}apt found${NL}"
204c204
< 	  printf "${B_W}yum found${NL}"
---
>       printf "${B_W}yum found${NL}"
212c212
< 	  printf "${B_W}apt found${NL}"
---
>       printf "${B_W}apt found${NL}"
248c248
<       cd ~/Naomi
---
>       cd ~/.config/naomi
277c277
<       cd ~/Naomi
---
>       cd ~/.config/naomi
321c321
<       cd ~/Naomi
---
>       cd ~/.config/naomi
350c350
<       cd ~/Naomi
---
>       cd ~/.config/naomi
729c729
< 	naomi_uninstall
---
>     naomi_uninstall

The major important changes seem to have to do with performing certain actions in the ~/Naomi or ~/.config/naomi directories. This would likely be about creating the script for launching Naomi. I'm still not sure why it's causing the installer to fail. I'm also not sure where the one-liner version of the script is coming from. I wonder if it would be possible to change the one-liner to:
. <( wget -O - https://raw.githubusercontent.com/NaomiProject/Naomi/refs/heads/naomi-dev/installers/naomi-setup.sh )
I will try that next.

[aaronchantrill]

Trying the one-liner with the direct link to the raw github file did not work. I still got the error about being in a managed environment. So I think there must be something to do with the way we are sourcing the file.