DietPi-Software | Install all Python software options into venvs

[miloit]

Describe the bug

When installing Mopidy via DietPi-Software, the installation fails with an error related to pip:

Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 25.1.1
error: uninstall-no-record-file

× Cannot uninstall pip 25.1.1
╰─> The package's contents are unknown: no RECORD file was found for pip.

hint: The package was installed by Debian. You should check if it can uninstall the package.
[FAILED] DietPi-Software | python3 get-pip.py

Steps to reproduce

1. Run DietPi-Software on a fresh DietPi install.
2. Attempt to install Mopidy.
3. Observe the error above.

Expected behavior

Mopidy should install correctly without attempting to uninstall the system pip package, respecting the Debian-provided pip installation.

Actual behavior

DietPi attempts to uninstall the system pip using get-pip.py, which fails because Debian pip has no RECORD file.

System information

DietPi version: latest

Reference code: 27b12307-94fd-435d-a727-00f0f85c2fab

The issue seems related to DietPi calling get-pip.py unnecessarily when pip is already installed via Debian packages.

[MichaIng]

For now, uninstall the APT package:

apt autopurge python3-pip

Switching all Python software options to venv's is planned for DietPi v10.0 in January, or maybe v10.1 in February. I hope there is a way to install pip and e.g. setuptools into the venv directly, to basically not have any system instance anymore that could conflict with Debian packages.

[miloit]

For now, uninstall the APT package:

apt autopurge python3-pip
Switching all Python software options to venv's is planned for DietPi v10.0 in January, or maybe v10.1 in February. I hope there is a way to install pip and e.g. setuptools into the venv directly, to basically not have any system instance anymore that could conflict with Debian packages.

I have done this....

Do I need than to reinstall all SW packages?

[MichaIng]

The ones you selected for this run, yes. If dietpi-software aborts in the middle, the software titles selected in this run are not marked as installed.

[MichaIng]

Background to the issue: Since v25, pip tries to uninstall even distribution packages of all Python modules tried to be installed, including pip itself and other toolkit packages like setuptools. Those were previously ignored. E.g. if python3-pip and/or python3-setuptools were installed, at the fixed /usr/lib/python3* location, pip just ignored it, since it installs to /usr/local/lib/python3* instead, so the paths do not collide.

Already with earlier pip versions, the externally-managed option was added for distribution to prevent pip from installing any module outside of venv's. But this could be overridden with another option, and worked well until v25. Now installing anything with pip outside of venv's has been made very inpracticable. So we need to change things our end to avoid these APT package conflicts.

[miloit]

The ones you selected for this run, yes. If dietpi-software aborts in the middle, the software titles selected in this run are not marked as installed.

So every python software needs to be reinstalled?

[MichaIng]

So every python software needs to be reinstalled?

If the dietpi-software aborted in the middle, and Python 3 was not installed separately before, yes. But just select again Mopidy as you did before. Python 3 as dependencies is anyway (re)installed in any case.

[MichaIng]

Let's see how well Copilot can deal with such a task:

@Copilot I leave here some info and instructions for this task:

• In https://github.com/MichaIng/DietPi/blob/dev/dietpi/dietpi-software we install a lot of Python software via pip3 and break-system-packages=true as root user into the system-wide /usr/local/lib/python3.x/dist-packages Python site.
• pip started to require break-system-packages=true for this already since a while, but recently is additionally tries to uninstall distro Python packages installed to /usr/lib/python3/dist-packages before installing modules with the same name to /usr/local/lib/python3.x/dist-packages. And this uninstall attempt fails, as can be seen in OP.
• We hence need to move our Python software installation options into virtualenv or venv environments.
• These venvs should be located below /opt/<software name> or directly at /opt/<software name> if not otherwise used. If the directory exists already, it could be checked whether the venv is still for the right Python version, else it would need to be recreated.
• The systemd service would need to load the respective venv, same with the CLI shell alias we generate in some cases.
• There are some exceptions:
  • OctoPrint is installed on user-level. It needs to become a venv as well, but that venv needs to be owned by the octoprint user, as its GUI allows to install further modules. The venv should be located below /mnt/dietpi_userdata/octoprint as well, since additionally installed modules are treated as user data, rather than fixed execution files which we usually install to /opt/<software name>.
  • ADS-B Feeder and Mopidy are not themselves installed via pip, but we install additional Python modules for them to use. We do not control the system services for those two software titles, and especially Mopidy is itself installed to /usr/lib/python3/dist-packages as APT package, hence, AFAIK, cannot load modules additionally from a venv. So we need to keep installing the additional modules to the system site. But is is probably okay to do this via explicit --break-system-packages for the particular pip3 call, rather than setting this flag in /etc/pip.conf. We need to check back with upstream developers about how to solve it best in those two cases.
  • For motionEye, the motioneye_init command installs a systemd service to /etc/systemd/system/motioneye.conf which calls ExecStart=/usr/local/bin/meyectl. So that service needs to be seded to call the right instance from the venv.
  • Home Assistant and microblog.pub are installed via pyenv into their own isolated Python instances. Hence no venv migration needs to be done for those.
• As of our coding standard, keep the changes minimal, generate and use the venv with minimal amount of code to just work. Those are not intended as development environments, but as mostly fixed end user environments, which do not change until the software is reinstalled (performing an update), with OctoPrint being one exception.