725: handling of package variations

[tobiasdiez]

How should one specify the "flavor" of a package that one is interested in? For example, gap on conda has gap-core, gap-defaults and gap (full), mostly differing by what kind of gap packages are installed. But not every distribution may make the same distinction.

Another example is the seadata db of pari (https://pari.math.u-bordeaux.fr/packages.html), which exists in different sizes.

https://peps.python.org/pep-0725/#canonical-names-of-dependencies-and-dev-el-split-packages discusses split devel packages, and the advice given there makes sense for devel-splits (since essentially the tool reading the metadata can make an educated guess on which kind of split package to load depending on the task). However, this is less so in the context of other split packages/db's.

[jaimergp]

There can be several mappings for the same PURL, and then the user might be able to choose what to provide at the system level.

[jaimergp]

I'll keep this issue open for now because we need to make sure it is covered in the "725b" PEP (we don't have a number yet). I think we have the technical solution, but maybe it needs some explicit discussion in there so this question is addressed.

For now, let me elaborate on what I meant with my previous comment with some actual examples.

We have the following components:

• external-metadata-mappings: the central registry plus some example mappings for different ecosystems. You can explore the mappings via this Streamlit app.
• pyproject-external: the library that parses the [external] table and maps them to your package manager names.

In external-metadata-mappings, a given canonical identifier (let's say dep:generic/libjpeg) can have several mapping entries. For example, in conda-forge:

    {
      "id": "dep:generic/libjpeg",
      "description": "The free library for JPEG image compression by the Independent JPEG Group",
      "specs": "jpeg",
      "urls": {
        "feedstock": "https://github.com/conda-forge/jpeg-feedstock"
      }
    },
    {
      "id": "dep:generic/libjpeg",
      "description": "IJG JPEG compliant runtime library with SIMD and other optimizations",
      "specs": "libjpeg-turbo",
      "urls": {
        "feedstock": "https://github.com/conda-forge/libjpeg-turbo-feedstock"
      }
    },
    {
      "id": "dep:generic/libjpeg",
      "description": "libjpeg-turbo, repackaged from mingw w64",
      "specs": "m2w64-libjpeg-turbo",
      "urls": {
        "feedstock": "https://github.com/conda-forge/msys2-recipes"
      }
    },

It is up to the consuming tool to present these options to the end-user. Right now, in pyproject-external, since it's only a demo, we are defaulting to the first non-null entry found for the dep: identifier, but ideally you'd get prompted for a choice.

I hope this clarifies the question, but let me know if you had something else in mind!

[rgommers]

There will be a lot of judgement calls to make when well-known libraries get rewritten, forked, etc. I think from our design perspective there are a couple of possible answers to make. We really should avoid the "let the end user choose" though.

Let's go through a couple of the examples here:

How should one specify the "flavor" of a package that one is interested in? For example, gap on conda has gap-core, gap-defaults and gap (full), mostly differing by what kind of gap packages are installed. But not every distribution may make the same distinction.

I'd always want to go with what the package authors want first. So I looked at https://www.gap-system.org/install/ and https://github.com/gap-system/PackageDistro, and I don't see a split there that the authors define/desire. They seem to just be talking about "GAP" and not about gap-core or anything else. Although on the from-source build instructions, they have gap-minimal and gap-full: https://github.com/gap-system/gap?tab=readme-ov-file#obtaining-the-gap-package-distribution, and it says "In contrast to the GAP stable releases, the development version does not come bundled with all the GAP packages."

From that I conclude that:

• dep:generic/gap or dep:/github/gap-system/gap should map to gap-full.
• There may be a case for gap-minimal as a separate entry, since it exists upstream. Perhaps that maps to gap-core as a package name elsewhere, and if so then that can be defined in the ecosystem mappings.

Ecosystem-specific package names that are not recognized by upstream (gap-with-my-special-subset) we should not have in our mappings. In case of doubt, in the process of adding entries/aliases to the mappings, we should be pinging the upstreams for their input.

---

libjpeg is an interesting situation. It's kinda like an interface, the libjpeg wikipedia page describes the situation well - there's an original library and multiple forks, the most prominent one being libjpeg-turbo (website, repo). libjpeg-turbo implements two APIs, a libjpeg-compatible one and its own one.

When adding these mappings I can imagine multiple outcomes:

1. We add generic/libjpeg (meaning the original) and generic/libjpeg-turbo (or github/libjpeg-turbo/libjpeg-turbo).
2. We add interface/libjpeg, to make more explicit that it's effectively an interface with distros plugging in another fork for the generic package name. And again also add generic/libjpeg-turbo since it has an extra API which is unique so packages may want that specifically.
3. We do both (1) and (2), so that if you really want the original you use generic/libjpeg and if you want whatever implements that API you use interface/libjpeg.

There will be many cases like this, so the important thing is that we have a very clear process for decision making.

[jaimergp]

I like the "interface" solution for ABI-compatible alternatives, but I'm not yet sure on what to do about the situation where a package has different names (within the same ecosystem) depending on the context. I am thinking of the m2- and m2w64- packages (vs the native unix ones) in conda-forge. I guess that's a limitation of conda-forge and they should be really using different build-strings if anything.

Do you think there's a need for some kind of "environment marker" metadata? I don't really want to complicate the schema just for the edge cases we may find in multi-platform package managers (since most of them are single-platform).

[rgommers]

I'm not yet sure on what to do about the situation where a package has different names (within the same ecosystem) depending on the context. I am thinking of the m2- and m2w64- packages (vs the native unix ones) in conda-forge.

That's similar to using different build options for the same package. I don't think there is any reasonable way we can capture that.

From a quick read-up on the situation at https://conda-forge.org/docs/maintainer/infrastructure/#compilers-and-runtimes: it seems like m2w64 is on the decline and will hopefully go away completely. Either way, that isn't something that should be relied on directly as far as I can tell, but for the resolver to manage.

I guess that's a limitation of conda-forge and they should be really using different build-strings if anything.

Yes looks like it.

Do you think there's a need for some kind of "environment marker" metadata?

No I don't think so, that seems pretty clearly out of scope.

[rgommers]

pkg-config is another example of this stuff: is it an interface with two implementations, or is it a concrete package pkg-config with an independent reimplementation of the same API (pkgconf). This was brought up at https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml/31888/83 by @mgorny.

It could go either way there as well. The pkgconf author has an explicit opinion there, from the README: "If you want pkgconf to be used when you invoke pkg-config, you should install a symlink for this. We do not do this for you, as we believe it is better for vendors to make this determination themselves."

I think I'd agree that package authors almost always want dep:interface/pkg-config, because the differences between the concrete packages are not relevant for 99% of use cases (although the same README says "no bug-for-bug compatibility"). So I'd do this:

• Add interface/pkg-config, and use that in the examples
• If and only if someone puts in a request for adding these to the central list, because they do rely on the differences between the two packages: add generic/pkg-config and/or generic/pkgconf.

[mgorny]

Also note that some distributions also make a similar distinction — e.g. Gentoo nowadays has dev-util/pkgconf and virtual/pkgconfig. Nowadays, the latter simply pulls in the former, but historically it supported multiple implementations. So if you depended on dev-util/pkgconf, you'd be saying "I want pkgconf specifically", and virtual/pkgconfig would be "I want any implementation with pkg-config executable". Technically there could be also a case such "I want pkgconf with a specific version, with a pkg-config wrapper" (that used to be optional before)).

In general, I dare say that keeping packages "more atomic" in the spec reduces the risk of ambiguous mapping to distributions. That said, we also need to be concerned about the difficulty of use. If developers can't easily figure out which packages precisely they need, they're going to make mistakes and/or get frustrated.

[rgommers]

Thanks for the extra context Michal, that is helpful.

Technically there could be also a case such "I want pkgconf with a specific version, with a pkg-config wrapper"

I'm not sure we can do that one as a single dep:, however we can by expressing it via use of both interface/pkg-config and generic/pkgconf.

In general, I dare say that keeping packages "more atomic" in the spec reduces the risk of ambiguous mapping to distributions.

I'd say yes agreed, but only if there is enough precedent in the upstream project or across distros. There are too many distros who do really weird stuff that is specific to their distro and that upstream either doesn't want to know about or is explicitly unhappy with. Those things I don't think are supportable. So boost in many parts sure, but pkg-with-custom-subset no.

There's always going to be a grey zone I think. The packages that matter are usually pretty actively maintained, so I'd put in the process there to ping upstream for helping decide on the harder cases.