Include current project when subsetting --venv-repository with --requirements

[matthieucx]

Hello @jsirois !

I have a bit of a complicated use-case.

I work in a data engineering team. Among other things, we write internal tooling for data scientists. I use PEX to package python environments to use on a Spark cluster, and wrote a CLI tool around it.

Data scientists can use uv or poetry to manage their projects, so resolutions is handled by those tools; I use --venv-repository.

In order to package only the dependencies useful in a spark session, I use poetry/uv to generate a lockfile, including dependency groups specified by the users through the CLI. I then use --requirements.

Now my issue: users can request that the current project be included in the PEX. As I subset the venv through the lockfile, I need to explicitly tell PEX that I need the project.

• Projects may or may not use a src/ layout, so I can't rely on this, which renders the use of --package/--module complicated, I think?
• "-e ." cannot be resolved when using --venv-repository, if I understand correctly
• --project seems to reintroduce a resolution step, slowing things done. I also ran into issues that I cannot easily reproduce with a minimal example yet when using --venv-repository, --requirements and --project together. Here's one such issue:

pex: Building pex :: Adding distributions built from local projects and collecting their requirements: /home/[...]/work/project :: Resolving requirements. :: Resolving for:
pex: Hashing pex                                                                                                                                                                                                                           pex: Hashing pex: 104.5ms
pex: Isolating pex: 0.1ms
pex: Building pex :: Adding distributions built from local projects and collecting their requirements: /home/[...]/work/project :: Resolving requirements. :: Building distributions for:
BuildRequest(download_target=DownloadTarget(target=LocalInterpreter('/home/[...]/work/project/.venv/bin/python'), universal_target=None), source_path='/home/[...]/work/project', finpex: Building /home/[...]/work/project to /tmp/tmpzcwmw226/build
pex: Building pex :: Adding distributions built from local projects and collecting their requirements: /home/[...]/work/project :: Resolving requirements. :: Calculating project names for direct requirements:
pex: Building pex :: Resolving distributions for requirements: anywidget<0.10.0,>=0.9.14 corr_module>=1.0.0 marketdata<3.0.0,>=2.0.0 matplotlib<4.0.0,>=3.7.0 numpy<3.0.0,>=2.2.3 openpyxl<4.0.0,>=3.1.5 plotly<6.0.0,>=5.24.1 popex: Building pex :: Resolving distributions for requirements: anywidget<0.10.0,>=0.9.14 corr_module>=1.0.0 marketdata<3.0.0,>=2.0.0 matplotlib<4.0.0,>=3.7.0 numpy<3.0.0,>=2.2.3 openpyxl<4.0.0,>=3.1.5 plotly<6.0.0,>=5.24.1 polars<2.0.0,>=1.32.3 pyyaml<7.0.0,>=6.0.2 scikit-learn<2.0.0,>=1.6.1 scipy<2.0.0,>=1.15.0 statsmodels<0.15.0,>=0.14.4 tsm<3.0.0,>=2.0.11 utils-internal==1.0.11 /home/[...]/work/project/archive-requirements.txtpex: Building pex :: Resolving distributions for requirements: anywidget<0.10.0,>=0.9.14 corr_module>=1.0.0 marketdata<3.0.0,>=2.0.0 matplotlib<4.0.0,>=3.7.0 numpy<3.0.0,>=2.2.3 openpyxl<4.0.0,>=3.1.5 plotly<6.0.0,>=5.24.1 polars<2.0.0,>=1.32.3 pyyaml<7.0.0,>=6.0.2 scikit-learn<2.0.0,>=1.6.1 scipy<2.0.0,>=1.15.0 statsmodels<0.15.0,>=0.14.4 tsm<3.0.0,>=2.0.11 utils-internal==1.0.11 /home/[...]/work/project/archive-requirements.txtException in thread Thread-3 (_handle_results):e/[...]/work/project/.venv. :: Using 4 parallel jobs to process 178 items
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.10/multiprocessing/pool.py", line 579, in _handle_results
task = get()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "/home/[...]/work/project/.venv/lib/python3.10/site-packages/pex/sorted_tuple.py", line 63, in new
sorted(iterable, key=key, reverse=reverse),  # type: ignore[arg-type, type-var]
File "/home/[...]/work/project/.venv/lib/python3.10/site-packages/pex/vendor/_vendored/attrs/attr/_make.py", line 1848, in lt
return attrs_to_tuple(self) < attrs_to_tuple(other)
TypeError: '<' not supported between instances of 'SpecifierSet' and 'SpecifierSet'

I'll open a proper issue for this if I manage to reproduce it in a barebone example

Finally, I tried adding the project name to the lockfile directly after its generation, which seems to work. But I ran into issues for package names using hyphens. Should I normalize the name to what's used for wheels (ie what is present inside the venv) ?

I would gladly take any advice or comment regarding this setup. Thanks for working on PEX, it's great!

[jsirois]

Thanks @matthieucx , that's probably enough details for me to experiment this afternoon and suggest a workaround or else introduce some Pex fixes to handle this case, which would seem common enough to me.

[jsirois]

Another note to self, Pex uses Pip to build local projects, and even though Pex uses pip wheel --no-deps ., it seems that is very much slower than directly doing so. For example:

:; time pip wheel -w /tmp/wheels . --no-deps
Processing /home/jsirois/support/pex/discussions/2979/project
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: project
  Building wheel for project (pyproject.toml) ... done
  Created wheel for project: filename=project-0.1.0-py3-none-any.whl size=1276 sha256=498cfde1b151ef42c873911f86730591352449e0f7e72b03d20802f54e714518
  Stored in directory: /tmp/pip-ephem-wheel-cache-7dvoqmo8/wheels/9a/5d/1f/835c4fa462d3d92ba842c9ddb3288aebc9957edd43f163d884
Successfully built project

[notice] A new release of pip is available: 25.2 -> 25.3
[notice] To update, run: pip install --upgrade pip

real	0m0.871s
user	0m0.696s
sys	0m0.145s

:; time uv build
Building source distribution (uv build backend)...
Building wheel from source distribution (uv build backend)...
Successfully built dist/project-0.1.0.tar.gz
Successfully built dist/project-0.1.0-py3-none-any.whl

real	0m0.007s
user	0m0.003s
sys	0m0.004s

[jsirois]

Alright @matthieucx I was able to reproduce the bug you found and isolate it. That fix is in #2981 and will be released in Pex 2.67.1 later this evening. I'm going to see if I can do anything about the --project ... build time. That is not doing a full resolve, just a resolve for the [build-system] requires in order to build the project, but that is using PyPI and not the --venv-repository. Generically, the --venv-repostory will not contain [build-system] requires, but just [project] dependencies, etc; so this is not a clean problem.

[jsirois]

Tinkering with editable installs in --venv-repositorys, I found a further bug: #2982

I'll try to knock that one out this evening.

[jsirois]

Alright @matthieucx, with #2984 released as Pex 2.67.2 later this evening, Pex will be able to resolve editables from venvs. To do so, it will effectively convert the editable into a --project ... argument.

Since we're in different time zones (I assume), can you gather the following information?:

• With Pex 2.67.2 try pex -v --venv-repository <venv dir> <project name> -o <project name>.pex. That should now be all you need to PEX up a project from its venv. The -v output will include high level timings and separate the 3 major bits: 1. project build time, 2. venv resolve time, 3. PEX zipping time.
• Timings for how long it takes to create a wheel for a project; i.e.: uv build or poetry ... (I don't know poetry!). One option to speed up the project build phase is to not let Pip do this but let you do it externally 1st and then modify Pex's --project option to accept wheel files.

And, by the way, be sure to use modern Pip! Perhaps easiest is to use --pip-version latest-compatible. Pip has gotten faster over the years and Pex defaults to a very old Pip (so it doesn't break existing users).

[matthieucx]

Wow, thanks a lot!

You assumed correctly, I'll try it out as soon as the release passes and let you know. Thanks for the tips regarding pip, I wasn't aware of it.

[matthieucx]

Regarding point 2, for the project I am compiling with PEX:

build-backend = "poetry.core.masonry.api"

❯ time poetry build --no-cache --quiet

________________________________________________________
Executed in    1.21 secs      fish           external
   usr time  921.36 millis    0.98 millis  920.38 millis
   sys time  235.51 millis    1.19 millis  234.32 millis

❯ time poetry build --quiet

________________________________________________________
Executed in    1.16 secs      fish           external
   usr time  805.91 millis  207.00 micros  805.70 millis
   sys time  274.12 millis  591.00 micros  273.53 millis

build-backend = "uv_build"

❯ time uv build --no-cache --quiet

________________________________________________________
Executed in  719.56 millis    fish           external
   usr time  168.06 millis  783.00 micros  167.27 millis
   sys time   80.53 millis    0.00 micros   80.53 millis

❯ time uv build --quiet

________________________________________________________
Executed in  372.20 millis    fish           external
   usr time  113.49 millis  317.00 micros  113.18 millis
   sys time   44.84 millis  321.00 micros   44.52 millis

I ran each command a dozen times and those are representative

[matthieucx]

Thanks @jsirois, seems like everything is working fine, the pex builds and I can import modules from my project!

First invocation:

❯ poetry run pex -v --venv-repository .venv/ airo --pip-version latest-compatible -o airo.pex
pex: Building pex :: Resolving distributions for requirements: airo :: Resolving requirements from 1 venv at: .venv/. :: Resolving for:
pex: Hashing pex                                                                                                                                           pex: Hashing pex: 103.4ms                                                                          
pex: Isolating pex: 26.2ms                                                                                                              
pex:   Extracting pex to /home/[...]/.cache/pex/isolated/0/9e876d36eab98243b45935293aa68b68a45426c2: 25.0ms
pex: Resolving for:
pex: Re-writing /home/[...]/.cache/pex/venvs/3/9a280f4e87ab18a2babcee585180f5984dafdd6b/5985ed09b49a653d6596b0e14d134c5456cf1a9f.lck.work/bin/activate
pex: Re-writing /home/[...]/.cache/pex/venvs/3/9a280f4e87ab18a2babcee585180f5984dafdd6b/5985ed09b49a653d6596b0e14d134c5456cf1a9f.lck.work/bin/activate.csh
pex: Re-writing /home/[...]/.cache/pex/venvs/3/9a280f4e87ab18a2babcee585180f5984dafdd6b/5985ed09b49a653d6596b0e14d134c5456cf1a9f.lck.work/bin/activate.fish
pex: Resolving for:
  cp310-cp310-manylinux_2_39_x86_64 interpreter at /home/[...]/work/project/.venv/bin/python: 6577.7ms
pex: Calculating project names for direct requirements:
  pip==25.3
  setuptools==80.9.0
pex: Calculating project names for direct requirements:
  pip==25.3
  setuptools==80.9.0
  wheel==0.45.1: 0.2ms
pex: Installing 3 distributions: 2404.0ms
pex: Checking install: 2.1ms
pex: Re-writing /home/[...]/.cache/pex/venvs/3/2c21e7272badf11cb8ac2eaee51f21f6291e53cc/5985ed09b49a653d6596b0e14d134c5456cf1a9f.lck.work/bin/activate
pex: Re-writing /home/[...]/.cache/pex/venvs/3/2c21e7272badf11cb8ac2eaee51f21f6291e53cc/5985ed09b49a653d6596b0e14d134c5456cf1a9f.lck.work/bin/activate.csh
pex: Re-writing /home/[...]/.cache/pex/venvs/3/2c21e7272badf11cb8ac2eaee51f21f6291e53cc/5985ed09b49a653d6596b0e14d134c5456cf1a9f.lck.work/bin/activate.fish
pex: Building pex :: Resolving distributions for requirements: airo :: Resolving requirements from 1 venv at: .venv/. :: Building distributions for:
  BuildRequest(download_target=DownloadTarget(target=LocalInterpreter('/home/[...]/work/project/.venv/bin/python'), universal_tarpex: Building /home/[...]/work/project to /tmp/tmpprvj9f9a/build                                                                                                                                                                                                                                                                                                                                       
pex: Building pex :: Resolving distributions for requirements: airo :: Resolving requirements from 1 venv at: .venv/. :: Calculating project names for direct requirements:
pex: Building pex :: Resolving distributions for requirements: airo :: Resolving requirements from 1 venv at: .venv/. :: Installing 1 distributions        pex: Building pex :: Resolving distributions for requirements: airo :: Resolving requirements from 1 venv at: .venv/. :: Using 4 parallel jobs to process 1pex: Elapsed time per process job:
  1) [882832] 0.98s 37 items
  2) [882833] 0.98s 43 items
  3) [882835] 0.89s 47 items
  4) [882834] 0.89s 43 items             
pex: Building pex: 21008.8ms                     
pex:   Adding distributions from pexes: : 0.1ms
pex:   Adding distributions built from local projects and collecting their requirements: : 0.6ms
pex:     Resolving requirements.: 0.5ms
pex:   Resolving distributions for requirements: airo: 20958.7ms
pex:     Resolving requirements from 1 venv at: .venv/.: 20951.5ms
pex:       Resolving for:
  cp310-cp310-manylinux_2_39_x86_64 interpreter at /home/[...]/work/project/.venv/bin/python: 14922.8ms
pex:       Building distributions for:
  BuildRequest(download_target=DownloadTarget(target=LocalInterpreter('/home/[...]/work/project/.venv/bin/python'), universal_target=None), source_path='/home/[...]/work/project', fingerprint=None, subdirectory=None): 2315.3ms
pex:       Calculating project names for direct requirements:
  airo @ file:///home/[...]/work/project: 0.4ms
pex:       Installing 1 distributions: 90.5ms
pex:       Using 4 parallel jobs to process 170 items: 3196.2ms
pex:   Configuring PEX dependencies: 23.9ms
Saving PEX file to airo.pex
pex: Zipping PEX file.: 225421.7ms

Second invocation:

pex: Building pex :: Resolving distributions for requirements: airo :: Resolving requirements from 1 venv at: .venv/. :: Resolving for:
pex: Hashing pex                                                                                                                                           pex: Hashing pex: 108.8ms                                                                          
pex: Isolating pex: 0.1ms
pex: Building pex: 2522.3ms                      
pex:   Adding distributions from pexes: : 0.1ms
pex:   Adding distributions built from local projects and collecting their requirements: : 0.7ms
pex:     Resolving requirements.: 0.4ms
pex:   Resolving distributions for requirements: airo: 2520.1ms
pex:     Resolving requirements from 1 venv at: .venv/.: 2520.0ms
pex:       Resolving for:
  cp310-cp310-manylinux_2_39_x86_64 interpreter at /home/[...]/work/project/.venv/bin/python: 2383.6ms
pex:   Configuring PEX dependencies: 0.1ms
Saving PEX file to airo.pex
pex: Zipping PEX file.: 186.9ms

[jsirois]

Thanks for the details @matthieucx . Your second run "Zipping PEX file" timing is hard to believe. That's the same exact command line as the 1st run? Zipping is not cached in any way and so that step should always take the same time.