[strawman] Add UvPipCompile #40

bollwyvl · 2026-01-27T03:09:26Z

References

RFC 3: using a dependency solver #10

Changes

add pyodide-lock[uv]
- the dependency on uv 0.9.27 is fairly bleeding-edge, but this version fixes a fairly serious inconsistency around --python-version
add UvPipCompile and helper classes
add tests
- well, technically just one, but with a fair number of parameterized inputs encountered in JupyterLite situations, but representative of a number of classes of issues
add minimal example to README

Future work

potentially out-of-scope for this PR

add a CLI
add docs
fixing PEP-420 import name discovery

ryanking13 · 2026-01-29T05:36:42Z

Thanks for the PoC @bollwyvl. Overall, I am +1 for the direction to provide an easy way to create a lockflie with custom dependencies, and it is great that we can leverage uv pip compile for that.

In the long term, I would prefer to create a lockfile from scratch instead of passing pyodide-lock.json as an input. This should be available when PEP 783 is accepted, or it is even possible now by using alternative package indices.

We are also considering adopting pylock.toml format and replace pyodide-lock.json with it for better compatibility (pyodide/pyodide#5617), thogh it would require some more work.

bollwyvl · 2026-01-29T14:30:04Z

In terms (ha!) of schedule: sure, it certainly has taken a long time for all of the upstream features and specs to line up for:

an platform-correct solve
using only pip-installable tools
including local wheels
in under a second
aware of the constraints of a host pyodide site

But great as the enemy of the good and all: I'm very motivated to do whatever it takes to get this landed here. pyodide-lock.json is what we have to work with right now, and I don't know of a better place to land this feature set which is sorely lacking for folks that want to offer their users a better, faster Pyodide experience.

PEP 783

Yep, that and many other great PEPs (e.g. PEP-794) are coming soon to an ecosystem near you... in the long term. Perhaps pessimistically, I'm assuming "use anything from PyPI" will need to be tweaked for a while yet to work properly in the harsh climate of the browser.

Further, I don't know if PyPI really wants to be signing up for it to be easier for potentially hundreds of megabytes of downloads per-page-view. Developers are already pretty cavalier about caching in e.g. CI; browser users might not even know they are using Python.

As a side effect of having to backfill PEP-764 by inspection, the proposed API encourages self-hosted wheels. In previous attempts at these features, I actually wired up a local caching proxy to intercept all wheel and API calls to avoid the second hit to download the file, as uv doesn't provide a nice way to get/reconstruct the actual wheels it downloads. But more subprocess, more problems, and 2 * whl < uncached_page_views_forever * whl.

require some more work.

Right: TOML not being part of the web "standard library", I'm confident a JS-based parser can be found, workarounds for bespoke, not-wheel files engineered, etc... in the long term.

lockfile from scratch

Assuming a hard switch that breaks downstream workflows is being considered, the proposed API could be adapted to make that easier. Pep751Toml could become a more full read/write, validating model, maybe in a separate source file unless a good, lightweight (ideally no pydantic), preferrably PyPA-supported one appears.

Indeed, there's probably value in this repository providing a path (ha!) from an existing pyodide-lock.json to a pylock.toml if the output file ends in .toml, and anything in-between.

hoodmane · 2026-01-31T11:50:52Z

Seems like pypi is interested in serving our packages. The main blocker on pep 783 is the Donald Stufft problem -- he's supposed to decide on whether to approve the pep or not but he does not read it. I have a meeting with the steering council in a few weeks and hopefully they'll come up with an alternative for me.

bollwyvl · 2026-01-31T14:34:04Z

pypi is interested in serving our packages

Great! But this doesn't change the fact that there are currently zero wheels on PyPI or any Pyodide distribution that would be marked properly. The use case of:

add a locally-built, unpublished wheel to the existing auto import mechanism provided by pyodide-lock.json with (potentially modified) PyPI dependencies

... is still an enormous pain for downstream users, and therefore downstream maintainers, such as myself.

Perhaps #10, or a brand-new issue, would be a better place to discuss future PEPs and future breaking changes. In the meantime, what would take for the capability proposed here to land in pyodide-lock 0.1.2? Again, happy to make whatever changes are needed, or pursue another approach.

ryanking13 · 2026-02-02T05:04:48Z

Sure, even after the PEP 783 is landed, it will take a lot of time for the package maintainers to adopt it and publish wheels. So having a feature to generate a lockfile from local wheels, wheels hosted from other places would still be useful.

Let me take a look at this PR soon and see what kind of changes we want before landing, and sorry for the delay. I've been busy these days and didn't have time to think about the lockfile.

pyodide_lock/uv_pip_compile.py

ryanking13 · 2026-02-02T05:58:11Z

pyodide_lock/uv_pip_compile.py

+                return
+            yield self.fetch_wheel(wheel_dir, archive["url"])
+        elif wheels:
+            wheel = wheels[0]


Could you please add a comment when there can be multiple wheels and why we are picking the first one?

f4b2355 added an exception and comment; I don't know how it would be possible, given --python-platform and --python-version to get multiple wheels, but the spec is the spec, and wheels is a list.

ryanking13 · 2026-02-02T05:59:17Z

pyodide_lock/uv_pip_compile.py

+        work: Path,
+        wheel_dir: Path,
+        lock_spec: PyodideLockSpec,
+    ) -> Iterator[tuple[Path, str | None]]:


Why is this function returning an interator, regarding that there is no iteration at all?

The iterator allows for having a single return type, with {0,1} findings: if the source is already in the destination place, nothing needs to be copied, but should not fail.

ryanking13 · 2026-02-02T06:15:40Z

pyodide_lock/uv_pip_compile.py

+                    continue
+                if Path(self.input_path.parent, file_name).is_file():
+                    continue
+                url = f"{self.base_url_for_missing.removesuffix('/')}/{file_name}"


How about preprocessing the base_url_for_missing at the constructor, so that we don't need to remove the suffix here.

I guess my goal is to change the input lock as little as possible, and isolate all rewriting in the post-processing step.

ryanking13 · 2026-02-02T06:18:37Z

README.md

+lock_spec.to_json(lock_path)
+```
+
+### Adding pre-built wheels and dependencies


I would like to hide the Python API and only expose it through the CLI (pyodide lock) so that we can safely change the API more easily.

I will just take this out at present, and focus on the code. The state of the existing API/CLI docs is probably out of scope.

As a downstream, I want a documented Python API, and don't want a click/typer dependency.

Removed in f4b2355

bollwyvl added 2 commits January 26, 2026 21:06

add UvPipCompile

8b00b83

mypy

fef1e30

bollwyvl changed the title ~~Add UvPipCompile~~ [strawman] Add UvPipCompile Jan 27, 2026

bollwyvl marked this pull request as ready for review January 27, 2026 03:17

bollwyvl mentioned this pull request Jan 27, 2026

RFC 3: using a dependency solver #10

Open

bollwyvl added 3 commits January 27, 2026 07:16

more linter

19a8751

add full base URL replacement

0e1a214

make missing URL replacer more general

2bcfbdf

ryanking13 reviewed Feb 2, 2026

View reviewed changes

bollwyvl added 2 commits February 2, 2026 08:43

address review comments

f4b2355

check if wheel archive exists

4dab793

bollwyvl requested a review from ryanking13 February 2, 2026 14:54

[strawman] Add UvPipCompile #40

Are you sure you want to change the base?

[strawman] Add UvPipCompile #40

Conversation

bollwyvl commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

References

Changes

Future work

Uh oh!

ryanking13 commented Jan 29, 2026

Uh oh!

bollwyvl commented Jan 29, 2026

Uh oh!

hoodmane commented Jan 31, 2026

Uh oh!

bollwyvl commented Jan 31, 2026

Uh oh!

ryanking13 commented Feb 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ryanking13 Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

bollwyvl Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

ryanking13 Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

bollwyvl Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ryanking13 Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

bollwyvl Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

ryanking13 Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

bollwyvl Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bollwyvl Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

bollwyvl commented Jan 27, 2026 •

edited

Loading

bollwyvl Feb 2, 2026 •

edited

Loading

bollwyvl Feb 2, 2026 •

edited

Loading