Implement uv backend 🚀

[danielhollas]

tl;dr with uv (and warm uv cache), creating a new aiida project takes around 3 seconds (unlike 30-40 seconds with pip). You can finally have a tagline "Create AiiDA project in a jiffy!"

After more testing I think it would be reasonable for this to be a new default, but would keep pip as a fallback option (we've been using uv in CI for a while and haven't had much issues).

Closes #27

[GeigerJ2]

See my comments. Maybe the refactor is worth it :) ping @mbercx

[danielhollas]

Thanks for the review @GeigerJ2! I'll wait a bit for @mbercx taking a look before doing further work here.

[mbercx]

Thanks for the ping guys. I'm focussing entirely on aiida-quantumespresso at the moment, however, and am trying to avoid any distractions. If it's not urgent, I'll cycle back to this once I'm (more or less) happy with the state of that package, if that day will ever come. 😅

[mbercx]

@danielhollas at this stage, do you see any issue with just using uv in the VenvProject? Following the KISS and "less is more" principles, unless there is a good reason to keep plain pip, I would just replace it.

[danielhollas]

@mbercx no particular reason other than the fact that this is largely untested and we could run into unforeseen issues (in which case having a plain pip as an escape hatch would have helped). But I am happy to simplify this if you think it's a better path forward (and any issues can be fixed in subsequent PRs). LMK what you think.

[GeigerJ2]

@danielhollas at this stage, do you see any issue with just using uv in the VenvProject? Following the KISS and "less is more" principles, unless there is a good reason to keep plain pip, I would just replace it.

Hmmm, would the simplifications to the code really be worth dropping the most standard tool for this? I'm a bit hesitant about this... I know for the user outside, it would look basically the same: make venv install packages with pip/uv, the latter they don't really care about. However, pip one always has with modern Python out of the box, while uv needs to be installed additionally. I don't think pip is something we can just drop, in my opinion, and, in any case, the simplifications we'd get by dropping it are not worth it. What do you think, guys?

[danielhollas]

@GeigerJ2 I don't have a strong opinion about this but note that we install uv into the venv, so the user should not even notice there is a difference, besides the speed.

One thing I did not consider in this PR, of we fully switch to uv, we might need some extra code to handle existing projects.

[mbercx]

Maybe we can just create the environment with the regular venv module? This doesn't take a lot of time, right?

[danielhollas]

Well, in takes a whole second on my machine (kind of surprising), which is actually a lot in terms of uv speed. I don't think there's a good reason NOT to use uv here tbh.

[mbercx]

Warning

The message below is off-the-cuff, and may contain mistakes. Also I just found out that you can use these admonitions on GitHub and I love it. 😍

Hmmm, would the simplifications to the code really be worth dropping the most standard tool for this?

Frankly, I think so, yes. Working on AiiDA has made me very wary of:

1. Trying to anticipate issues/extenstions too much.
2. Too much class inheritance.
3. Adding more and more options other users are confronted with because there might just be one out there that possibly needs it.
4. Having too much lines code to maintain. And little bits add up over time.

I would:

1. Make uv a dependency of aiida-project. Users don't have to install anything extra. We explicitly use that one instead of which so we can control the version. We can pin it because this is a tool that should be installed in a separate environment and never be a dependency.
2. We use that uv path to install aiida-core into the virtual environment. @danielhollas I suppose the big time saver is this one, not the creation of the virtual environment?
3. If - and only if - at some point we realise that something goes wrong with the uv pip install of aiida-core in the new environment, we can add a config option to set the venv package manager to regular pip. At that point we can still consider the structure you proposed, @GeigerJ2. But if we can't install aiida-core with uv pip install, I think we'll find out pretty quickly.

Note: I know we can also install other packages with the aiida-project create command. I'm wondering if we don't want to remove this feature (see #30). I don't use it in the end to be honest. You can just install the packages after you enter the environment. The scope of aiida-project for now should just be: "get the user to verdi".

we might need some extra code to handle existing projects.

If we do the changes I propose above, would we? The only thing we'd change is installing aiida-core with uv upon creation of a project. Existing projects would be unaffected, unless I'm missing something.

[danielhollas]

Note

I totally agree with your excitement about the admonitions. 🤣

Make uv a dependency of aiida-project. Users don't have to install anything extra. We explicitly use that one instead of which so we can control the version. We can pin it because this is a tool that should be installed in a separate environment and never be a dependency.

Indeed, that's already done in this PR.

Note: I know we can also install other packages with the aiida-project create command. I'm wondering if we don't want to remove this feature (see #30).

I mean, uv should work perfectly fine for installing whatever package so I don't think we need to remove this feature, at least it's not pre-requisite to merging this PR.

But if we can't install aiida-core with uv pip install, I think we'll find out pretty quickly.

Yeah, that really should not happen, as we're using uv in aiida-core CI.

If we do the changes I propose above, would we? The only thing we'd change is installing aiida-core with uv upon creation of a project. Existing projects would be unaffected, unless I'm missing something.

Right. I am not that familiar with this project --- but indeed if the package installation only happens at project creation then we should be fine. We just need to be mindful of the fact that the old venvs will not have uv installed in them --- I guess this is fine at this point, but if we ever added a feature that would be touch existing venv we would have to think about that.

I agree with the general sentiment of keeping things simple. When I cooked up this PR during the last year coding week, I wasn't even sure if this feature would be wanted, which is why I went with making it optional. But I agree that the risk here should be small if we test things enough. I'll work on simplifying this PR this week.

[mbercx]

I mean, uv should work perfectly fine for installing whatever package so I don't think we need to remove this feature, at least it's not pre-requisite to merging this PR.

Completely agree! I also wouldn't expect issues, to be honest. But #30 describes several other reasons why I think we should remove that feature.

I'll work on simplifying this PR this week.

If you have time, great! I'm also back to coding quite a bit more lately, so happy to help if needed.

[GeigerJ2]

Warning

The message below is off-the-cuff, and may contain mistakes. Also I just found out that you can use these admonitions on GitHub and I love it. 😍

Just note that they don't appear properly upon "quote reply" :D

Hmmm, would the simplifications to the code really be worth dropping the most standard tool for this?

Frankly, I think so, yes. Working on AiiDA has made me very wary of:

1. Trying to anticipate issues/extenstions too much.
2. Too much class inheritance.
3. Adding more and more options other users are confronted with because there might just be one out there that possibly needs it.
4. Having too much lines code to maintain. And little bits add up over time.

There goes my excitement of learning about the Strategy pattern, but, fair enough, we all know this pain all too well.

I would:

1. Make uv a dependency of aiida-project. Users don't have to install anything extra. We explicitly use that one instead of which so we can control the version. We can pin it because this is a tool that should be installed in a separate environment and never be a dependency.
2. We use that uv path to install aiida-core into the virtual environment. @danielhollas I suppose the big time saver is this one, not the creation of the virtual environment?
3. If - and only if - at some point we realise that something goes wrong with the uv pip install of aiida-core in the new environment, we can add a config option to set the venv package manager to regular pip. At that point we can still consider the structure you proposed, @GeigerJ2. But if we can't install aiida-core with uv pip install, I think we'll find out pretty quickly.

OK, if we can guarantee that uv will be available through that route, then it is fine for me to fully drop pip to simplify the code, as you suggested @mbercx. Thanks, both, for your extensive replies, I'm on board 🚀 Feel free to ping me for another review!

PS: Sorry for closing the PR, ended up clicking that button with my touchpad on page reload...

[mbercx]

Just note that they don't appear properly upon "quote reply" :D

Don't you dare rain on my parade! 😤

There goes my excitement of learning about the Strategy pattern, but, fair enough, we all know this pain all too well.

Haha, I'm all too familiar with the trap of wanting to over-engineer something. So I won't cast the first stone.

[danielhollas]

Alright, this is ready for a next round of review. The PR is now delightfully simple. 👼

One open question: The create command can be now run even if a project with a given name already exists. Is that an intended behaviour? If it is, what is the rationale and intented use case?

If it is not, we should stop early with a notification message (that seems to be the most reasonable to me).

This is important for this PR because uv venv by default removes the existing venv, which is probably not what we want here and we should guard against it.

[mbercx]

One open question: The create command can be now run even if a project with a given name already exists. Is that an intended behaviour? If it is, what is the rationale and intented use case?

I wouldn't say it's intended. This package was mostly my personal tool at first, so safety and UX wasn't too much of a concern. That said, I have used this in the past to overwrite the virtual environment while keeping the project directory (and the .aiida directory inside it) intact. But that use case can be supported with e.g. a -f/--force or -o/--overwrite option or something similar.

As usual: I'll write my OCD commit messages as I review and understand the changes, ok if I reorganise? :)

[danielhollas]

CI timings on this PR are instructive 🚀

As usual: I'll write my OCD commit messages as I review and understand the changes, ok if I reorganise? :)

👍

[mbercx]

CI timings on this PR are instructive 🚀

Very sexy indeed! But errmmm, wasn't it getting late there? I feel I have to stop reviewing your PRs to help improve your sleeping pattern. 😅

[GeigerJ2]

Just a note here that, if I pip install -e aiida-project in my system python from this PR, I get the exception below. Not sure if this is a common setup scenario, however, I typically have aiida-project installed globally, not in any venv, as I use it exclusively to handle my aiida-work related envs (my system might be fucked up, also, as I previously had uv installed via pipx). Still, maybe we should err out nicer?

~/dev                                                                                                                                                                                                              10:40:30
❯ pip install -e aiida-dev/aiida-project/
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/geiger_j/dev/aiida-dev/aiida-project
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: pydantic~=2.7 in /home/geiger_j/.local/lib/python3.10/site-packages (from aiida-project==0.6.0) (2.11.9)
Collecting uv~=0.8.20
  Using cached uv-0.8.22-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (21.3 MB)
Requirement already satisfied: python-dotenv~=1.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from aiida-project==0.6.0) (1.0.1)
Requirement already satisfied: typer[all]~=0.9 in /home/geiger_j/.local/lib/python3.10/site-packages (from aiida-project==0.6.0) (0.17.4)
Requirement already satisfied: pyyaml~=6.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from aiida-project==0.6.0) (6.0.1)
Requirement already satisfied: pydantic-settings~=2.2 in /home/geiger_j/.local/lib/python3.10/site-packages (from aiida-project==0.6.0) (2.7.1)
Requirement already satisfied: pydantic-core==2.33.2 in /home/geiger_j/.local/lib/python3.10/site-packages (from pydantic~=2.7->aiida-project==0.6.0) (2.33.2)
Requirement already satisfied: typing-inspection>=0.4.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from pydantic~=2.7->aiida-project==0.6.0) (0.4.1)
Requirement already satisfied: typing-extensions>=4.12.2 in /home/geiger_j/.local/lib/python3.10/site-packages (from pydantic~=2.7->aiida-project==0.6.0) (4.15.0)
Requirement already satisfied: annotated-types>=0.6.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from pydantic~=2.7->aiida-project==0.6.0) (0.6.0)
WARNING: typer 0.17.4 does not provide the extra 'all'
Requirement already satisfied: click>=8.0.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from typer[all]~=0.9->aiida-project==0.6.0) (8.1.7)
Requirement already satisfied: rich>=10.11.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from typer[all]~=0.9->aiida-project==0.6.0) (13.7.1)
Requirement already satisfied: shellingham>=1.3.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from typer[all]~=0.9->aiida-project==0.6.0) (1.5.4)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from rich>=10.11.0->typer[all]~=0.9->aiida-project==0.6.0) (2.18.0)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/geiger_j/.local/lib/python3.10/site-packages (from rich>=10.11.0->typer[all]~=0.9->aiida-project==0.6.0) (3.0.0)
Requirement already satisfied: mdurl~=0.1 in /home/geiger_j/.local/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer[all]~=0.9->aiida-project==0.6.0) (0.1.2)
Building wheels for collected packages: aiida-project
  Building editable for aiida-project (pyproject.toml) ... done
  Created wheel for aiida-project: filename=aiida_project-0.6.0-py3-none-any.whl size=4791 sha256=2e963bd28ad59f19f3860d2b3dceceffdbc9e6e447674c608ea3542e824bb70c
  Stored in directory: /tmp/pip-ephem-wheel-cache-5qs119dt/wheels/65/ad/74/f2ea77111c51bfb26fef6d59415ae796d2bcdb399072bda2bb
Successfully built aiida-project
Installing collected packages: uv, aiida-project
Successfully installed aiida-project-0.6.0 uv-0.8.22

~/dev                                                                                                                                                                                                           5s 10:40:35
❯ aiida-project create test-uv
✨ Creating the project directory and environment using the Python binary:
   /usr/bin/python3.10
╭─────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ───────────────────────────────────────────────────────────────────────────────────────────╮
│ /home/geiger_j/dev/aiida-dev/aiida-project/aiida_project/commands/main.py:147 in create                                                                                                                                 │
│                                                                                                                                                                                                                         │
│   144 │   )                                                                                                                                                                                                             │
│   145 │                                                                                                                                                                                                                 │
│   146 │   try:                                                                                                                                                                                                          │
│ ❱ 147 │   │   project.create(python_path=python_path)                                                                                                                                                                   │
│   148 │   except CalledProcessError as e:                                                                                                                                                                               │
│   149 │   │   print("[bold red]Error:[/bold red] Python environment creation failed!")                                                                                                                                  │
│   150 │   │   typer.echo(e)                                                                                                                                                                                             │
│                                                                                                                                                                                                                         │
│ /home/geiger_j/dev/aiida-dev/aiida-project/aiida_project/project/venv.py:43 in create                                                                                                                                   │
│                                                                                                                                                                                                                         │
│   40 │   │   │   f"{python_path.resolve()}",                                                                                                                                                                            │
│   41 │   │   │   str(self.venv_path),                                                                                                                                                                                   │
│   42 │   │   ]                                                                                                                                                                                                          │
│ ❱ 43 │   │   subprocess.run(venv_command, check=True, capture_output=True)                                                                                                                                              │
│   44 │                                                                                                                                                                                                                  │
│   45 │   def destroy(self):                                                                                                                                                                                             │
│   46 │   │   """Destroy the project."""                                                                                                                                                                                 │
│                                                                                                                                                                                                                         │
│ /usr/lib/python3.10/subprocess.py:503 in run                                                                                                                                                                            │
│                                                                                                                                                                                                                         │
│    500 │   │   kwargs['stdout'] = PIPE                                                                                                                                                                                  │
│    501 │   │   kwargs['stderr'] = PIPE                                                                                                                                                                                  │
│    502 │                                                                                                                                                                                                                │
│ ❱  503 │   with Popen(*popenargs, **kwargs) as process:                                                                                                                                                                 │
│    504 │   │   try:                                                                                                                                                                                                     │
│    505 │   │   │   stdout, stderr = process.communicate(input, timeout=timeout)                                                                                                                                         │
│    506 │   │   except TimeoutExpired as exc:                                                                                                                                                                            │
│                                                                                                                                                                                                                         │
│ /usr/lib/python3.10/subprocess.py:971 in __init__                                                                                                                                                                       │
│                                                                                                                                                                                                                         │
│    968 │   │   │   │   │   self.stderr = io.TextIOWrapper(self.stderr,                                                                                                                                                  │
│    969 │   │   │   │   │   │   │   encoding=encoding, errors=errors)                                                                                                                                                    │
│    970 │   │   │                                                                                                                                                                                                        │
│ ❱  971 │   │   │   self._execute_child(args, executable, preexec_fn, close_fds,                                                                                                                                         │
│    972 │   │   │   │   │   │   │   │   pass_fds, cwd, env,                                                                                                                                                              │
│    973 │   │   │   │   │   │   │   │   startupinfo, creationflags, shell,                                                                                                                                               │
│    974 │   │   │   │   │   │   │   │   p2cread, p2cwrite,                                                                                                                                                               │
│                                                                                                                                                                                                                         │
│ /usr/lib/python3.10/subprocess.py:1863 in _execute_child                                                                                                                                                                │
│                                                                                                                                                                                                                         │
│   1860 │   │   │   │   │   │   err_filename = orig_executable                                                                                                                                                           │
│   1861 │   │   │   │   │   if errno_num != 0:                                                                                                                                                                           │
│   1862 │   │   │   │   │   │   err_msg = os.strerror(errno_num)                                                                                                                                                         │
│ ❱ 1863 │   │   │   │   │   raise child_exception_type(errno_num, err_msg, err_filename)                                                                                                                                 │
│   1864 │   │   │   │   raise child_exception_type(err_msg)                                                                                                                                                              │
│   1865                                                                                                                                                                                                                  │
│   1866                                                                                                                                                                                                                  │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/uv'

~/dev                                                                                                                                                                                                              10:40:40
❯ which python
/usr/bin/python

~/dev                                                                                                                                                                                                              10:40:46
❯ which uv
/home/geiger_j/.local/bin/uv

[danielhollas]

I wouldn't say it's intended. This package was mostly my personal tool at first, so safety and UX wasn't too much of a concern. That said, I have used this in the past to overwrite the virtual environment while keeping the project directory (and the .aiida directory inside it) intact. But that use case can be supported with e.g. a -f/--force or -o/--overwrite option or something similar.

Okay, for now I've added code that simply returns early if a project exists already. Users can always nuke their venv manually if needed.

Caution

The main scary thing here is that if you happen to provide a wrong path uv venv, it will not care and remove everything at that path! At least that's what happened in older versions, now there is a confirmation dialog by default. And this is not a theoretical concern --- on main you can pass an empty string as a project name, which would in turn nuke all of your venvs. ❗ ⚠️

[danielhollas]

Just a note here that, if I pip install -e aiida-project in my system python from this PR, I get the exception below.

😝

But fair point, I've added a fallback using shutil.which and print an error if that fails. PTAL

Not sure if this is a common setup scenario, however, I typically have aiida-project installed globally,

Yeah, that's what uv tool or pipx are for. :-) For local development I would always create a separate environment and activate it.

[mbercx]

The main scary thing here is that if you happen to provide a wrong path uv venv, it will not care and remove everything at that path! At least that's what happened in older versions, now there is a confirmation dialog by default. And this is not a theoretical concern --- on main you can pass an empty string as a project name, which would in turn nuke all of your venvs. ❗ ⚠️

First of all I would like to reiterate how much I love these admonitions. 😍

But yeah, fair point. You'd have to pass '', because else typer will complain that the NAME argument is missing, but clearly here be dragons. Let me review this PR and save ourselves the potential heart ache.

[mbercx]

Sweet, great work @danielhollas! Testing this locally has been an absolute bliss with the improved speed, can't wait to get this released. 🚀

Just left a few nitpicks, but then this is good to go for me! I'd split up the commits as usual, and then have you double check the messages, if that's ok?

[mbercx]

I didn't catch this before, but have you tried source-ing the $HOME/.bashrc? The aiida-project init command will add the cda command there, but the user always has to source or open a new terminal.

[danielhollas]

Yes, I did try and couldn't make it to work.

[mbercx]

Strange... Oh well.

[danielhollas]

Just left a few nitpicks, but then this is good to go for me! I'd split up the commits as usual, and then have you double check the messages, if that's ok?

Yep, please feel free to finish this, no need for me to double check, I will not have time to review this evening and will be glad to have this from my plate.

Testing this locally has been an absolute bliss with the improved speed, can't wait to get this released. 🚀

Yes, it's very nice. 🎉

[mbercx]

Yep, please feel free to finish this, no need for me to double check, I will not have time to review this evening and will be glad to have this from my plate.

I understand, you've been pumping quite a bit of time into aiida-project, and I appreciate it! 🙏

[mbercx]

Thanks again @danielhollas! Time for a release, I'd say. :) 🚀