Install Python software into venvs to avoid pip v25+ conflicts

[Copilot]

Migration Plan: Python Software to Virtual Environments

Analysis Complete

• Identify all Python software installations in dietpi-software
• Understand current installation patterns
• Review special cases and exceptions

Global changes:

• Remove break-system-packages from /etc/pip.conf
• Add python3-venv and python3-pip to dependencies
• Set PYTHON_VERSION as global variable
• Remove root-user-action=ignore from pip.conf
• Add --root-user-action=ignore to system pip calls

Software migrated to venv:

• ID 125: Synapse (Matrix server) - /opt/synapse
• ID 136: motionEye - /opt/motioneye, sed systemd service (robust pattern)
• ID 139: SABnzbd - /opt/sabnzbd
• ID 180: Bazarr - /opt/bazarr/venv
• ID 155: HTPC Manager - /opt/htpc-manager
• ID 153: OctoPrint - /mnt/dietpi_userdata/octoprint/venv, owned by octoprint user

Exceptions (kept system install with --break-system-packages):

• ID 118: Mopidy - APT package, additional modules with --break-system-packages --root-user-action=ignore
• ID 141: ADS-B Feeder - Uses --break-system-packages --root-user-action=ignore

Already using isolated Python (skipped):

• ID 157: Home Assistant - Uses pyenv
• ID 16: microblog.pub - Uses pyenv

Code improvements:

• Compressed venv creation to single line: [[ -d '/opt/foo' ]] || G_EXEC python3 -m venv --system-site-packages /opt/foo
• Use global PYTHON_VERSION variable instead of calculating per-software
• Python_Deps no longer unsets PYTHON_VERSION
• System pip calls use --root-user-action=ignore flag

All tasks completed! ✅

Original prompt

---

This section details on the original issue you should resolve

<issue_title>DietPi-Software | Install all Python software options into venvs</issue_title>
<issue_description>### Describe the bug
When installing Mopidy via DietPi-Software, the installation fails with an error related to pip:

Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 25.1.1
error: uninstall-no-record-file

× Cannot uninstall pip 25.1.1
╰─> The package's contents are unknown: no RECORD file was found for pip.

hint: The package was installed by Debian. You should check if it can uninstall the package.
[FAILED] DietPi-Software | python3 get-pip.py

Steps to reproduce

1. Run DietPi-Software on a fresh DietPi install.
2. Attempt to install Mopidy.
3. Observe the error above.

Expected behavior

Mopidy should install correctly without attempting to uninstall the system pip package, respecting the Debian-provided pip installation.

Actual behavior

DietPi attempts to uninstall the system pip using get-pip.py, which fails because Debian pip has no RECORD file.

System information

DietPi version: latest

Reference code: 27b12307-94fd-435d-a727-00f0f85c2fab

The issue seems related to DietPi calling get-pip.py unnecessarily when pip is already installed via Debian packages.</issue_description>

<agent_instructions>I leave here some info and instructions for this task:

• In https://github.com/MichaIng/DietPi/blob/dev/dietpi/dietpi-software we install a lot of Python software via pip3 and break-system-packages=true as root user into the system-wide /usr/local/lib/python3.x/dist-packages Python site.
• pip started to require break-system-packages=true for this already since a while, but recently is additionally tries to uninstall distro Python packages installed to /usr/lib/python3/dist-packages before installing modules with the same name to /usr/local/lib/python3.x/dist-packages. And this uninstall attempt fails, as can be seen in OP.
• We hence need to move our Python software installation options into virtualenv or venv environments.
• These venvs should be located below /opt/<software name> or directly at /opt/<software name> if not otherwise used. If the directory exists already, it could be checked whether the venv is still for the right Python version, else it would need to be recreated.
• The systemd service would need to load the respective venv, same with the CLI shell alias we generate in some cases.
• There are some exceptions:
  • OctoPrint is installed on user-level. It needs to become a venv as well, but that venv needs to be owned by the octoprint user, as its GUI allows to install further modules. The venv should be located below /mnt/dietpi_userdata/octoprint as well, since additionally installed modules are treated as user data, rather than fixed execution files which we usually install to /opt/<software name>.
  • ADS-B Feeder and Mopidy are not themselves installed via pip, but we install additional Python modules for them to use. We do not control the system services for those two software titles, and especially Mopidy is itself installed to /usr/lib/python3/dist-packages as APT package, hence, AFAIK, cannot load modules additionally from a venv. So we need to keep installing the additional modules to the system site. But is is probably okay to do this via explicit --break-system-packages for the particular pip3 call, rather than setting this flag in /etc/pip.conf. We need to check back with upstream developers about how to solve it best in those two cases.
  • For motionEye, the motioneye_init command installs a systemd service to /etc/systemd/system/motioneye.conf which calls ExecStart=/usr/local/bin/meyectl. So that service needs to be seded to call the right instance from the venv.
  • Home Assistant and microblog.pub are installed via pyenv into their own isolated Python instances. Hence no venv migration needs to be done for those.
• As of our coding standard, keep the changes minimal, generate and use the venv with minimal amount of code to just work. Those are not intended as development environments, but as mostly fixed end user environments, which do not change until the software is reinstalled (performing an update), with OctoPrint being one exception.</agent_instructions>

Comments on the Issue (you are @copilot in this section)

@MichaIng For now, uninstall the APT package: ```sh apt autopurge python3-pip ``` Switching all Python software options to venv's is planned for DietPi v10.0 in January, or maybe v10.1 in February. I hope there is a way to install `pip` and e.g. `setuptools` into the venv directly, to basically not have any system instance anymore that could conflict with Debian packages. @MichaIng The ones you selected for this run, yes. If `dietpi-software` aborts i...

• Fixes DietPi-Software | Install all Python software options into venvs #7859

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[Joulinar]

Lets see how copilot is dealing with this 😄

[MichaIng]

From the initial comments and title it misinterpreted the issue. I changed the issue title and added clarification, also explicitly via agent instructions, before assigning Copilot to the issue, so I hope it gets things right after reading those and does not remove get-pip.py or so 😄.

[Joulinar]

Would we need some migration for existing installations?

[MichaIng]

Hah, it did get it right. Impressive, isn't it? First commit pretty flawless. I'll do some format changes, but so far did not see a mistake which would not work. Also, where /opt/foo exists, it uses /opt/foo/venv, else /opt/foo directly. Also for OctoPrint write access preserved, otherwise following my instructions/hints.

... ah, one mistake: It did remove get-pip.py, as it would indeed also still fail if python3-pip was installed. But python3-pip is not installed. So pip will be currently missing for ADS-B Feeder and Mopidy installs, if I am not mistaken.

Also nice to see the iterations it does, adding uninstall adjustments afterwards (I did not mention this anywhere, I just got the idea), code hardening.

Would we need some migration for existing installations?

Yes. We should probably do this on DietPi update to not have all that cleanup code in dietpi-software.

[MichaIng]

Test installs: https://github.com/MichaIng/DietPi/actions/runs/22114553332

Further tasks:

• Python_Deps cleanup, as PYTHON_VERSION is now always available
• Migration in DietPi update:
  • Uninstall system instance of main Python modules
  • Do we want to cleanup /usr/local/lib/python3.x/dist-packages? We would risk to remove modules users installed manually and intentionally. Probably we can ask via dialogue.
  • Migrate environment from OctoPrint into the venv. Need to test whether this is seamlessly possible.
• Remove --system-site-packages? It is not needed for pip to be available in a venv, as it is added automatically. --system-site-packages can reduce the need of additionally installed modules in the venv, maybe also pip is not fully installed in that case, but a wrapper around system pip or so. But it also means that dependency updates for modules available on system-level are only done if required to satisfy versioned dependency declarations. Not necessarily an issue, but we might want to allow venvs to always use latest available versions (at install time), instead of sticking with possibly outdated system versions of dependencies.

[MichaIng]

• Mopidy on Forky fails for a different reason: https://salsa.debian.org/python-team/packages/mopidy/-/merge_requests/3
• cryptography (maturin backend) builds on RISC-V also fail for a different reason: ci: set crt-static for riscv64 and loongarch64 musl targets PyO3/maturin#3009
• Forky on ARMv6 fails because the Raspbian repository once again is in an inconsistent state, with several new merged FFmpeg and PostgreSQL packages depending new dependencies which have not been merged. Regular issue with Raspbian testing, with actually a very simple solution (pulling from snapshots.debian.org): https://bugs.launchpad.net/raspbian/+bug/2052306

[MichaIng]

Okay, I implemented the migration for Synapse, motionEye, and OctoPrint:

• Synapse and motionEye were themselves installed via pip, hence the system-site package is uninstalled.
• OctoPrint was installed into the user-site (which makes no difference regarding pip breakage), as well as plugins installed with its UI. The migration moves the user site to the venv path, and turns that into a venv. It is not 100% straight-forward with python -m venv, since executable wrappers below venv/bin still have the old #!/usr/bin/python3 shebang, so they fail to load any other module. Understanding how this works, without forcing a reinstall of all modules, the shebang is just replaced with the venv/bin/python3 path. This is exactly what a (re)install into the venv does. Works well, all plugins remain available, and new plugins can be installed. OctoPrint recognizes the venv well, like it did recognized the prior user site, to install plugins correctly.

SABnzbd, Bazarr, and HTPC Manager are left untouched. They are installed as source tarball, any only their dependencies installed via pip3 -r requirements.txt. They are not designed for/cannot be installed themselves as Python packages. Since we have no real way to know whether a module at /usr/local/lib/python3.*/dist-packages is a dependency for one of our software options only, and if so for which, we cannot do a cleanup anyway. So no harm to keep software using those. A reinstall will make them use a venv without any additional migration steps needed.

ADS-B Feeder and Mopidy do still require /usr/local/lib/python3.*/dist-packages, though we need to find out whether/how to get rid of it.

Once none of our software options makes any use of /usr/local/lib/python3.*/dist-packages anymore, we can offer a complete uninstall of all packages there, and removal of leftovers.