Make all Coherent System projects have a minimal `pyproject.toml`

[johnslavik]

This issue aims to point out problems resulting from not adhering to PEP 517/518, that is storing a minimal pyproject.toml file in every Coherent project:

[build-system]
requires = ['coherent.build']
build-backend = 'coherent.build'

At the moment, this file is only supplied temporarily in certain contexts and coherent.build has full control over its lifespan. I am thinking this isn't the most optimal approach, because:

1. Coherent projects aren't universally identifiable—there is no official way of identifying what projects use the essential layout. Since that layout is inseparably tied to coherent.build, it would be easy to figure that out by looking into build-system.build-backend. I think that assuming the presence of __init__.py or __main__.py is slightly short-sighted: this definitely isn't the essential layout.
2. Coherent projects don't play well with the existing installers—coh install makes assumptions about the preferred way of installing a project. I personally like to develop using virtual environments and that is one of the most popular practices, enabling inline hints and more. It should be possible to simply install Coherent projects using whatever method preferred: for instance, I want to develop 3 coherent projects in my super directory using editable installs.
3. Coherent projects aren't detected by third-party software as Python packages—while it could often be easily done just by seeing .py files in the root directory, this is often short-sighted too and might not be enough to prove a directory constituing a standalone package. For that reason Starship looks for specific files, with pyproject.toml being the common part of two enablers (https://starship.rs/config/#package-version, https://starship.rs/config/#python).

While the idea of temporarily supplying pyproject.toml allows to save some boilerplate, I still do think it comes with much more caveats than benefits. I'd like to suggest every Coherent project has a pyproject.toml so that it is just easier to work with.

Now that poses some questions, what about (meta)/. If we statically keep pyproject.toml for every Coherent project, shouldn't we rather let others customize ancillary details inside pyproject.toml and then, optionally, in (meta)/?
Personally, I think that it would mean even easier migration to the Coherent System from flat/src layouts.

---

• Update docs to reflect initialization steps.
• (optional) Create a routine to initialize a repo with the boilerplate.
• Apply changes to existing coherent projects (coherent.*, tempora@experiment/coherent).

[jaraco]

This issue has a lot of overlap with #8.

I do agree, there are some serious downsides to the current experience.

I think you're right that as a practical matter, we should, at least for now, require that every coherent project include the boilerplate file.

How should we go about implementing the change? I can think of at least a couple places that need to be updated - in the getting started guide, in the builder readme.

Probably now we also need a command to write the boilerplate, no? Something like coh init (🤮) (which would also set up things like the GitHub actions). Or should coherent system projects have a skeleton to manage these common concerns? My instinct is no, but if these boilerplate files start changing at the system level, it's going to quickly become untenable to manage them through scripts and manual operations.

Now that poses some questions, what about (meta)/. If we statically keep pyproject.toml for every Coherent project, shouldn't we rather let others customize ancillary details inside pyproject.toml and then, optionally, in (meta)/?

Yes, possibly. We can consider those concerns later.

Are you willing to work on the effort? I'll add some subtasks in the original description.

[jaraco]

What I'm considering out-of-scope for now:

• Removing the logic that provides pyproject.toml on-demand. That logic will give precedence to an extant pyproject.toml, so let's rely on that for now.

[johnslavik]

Yes. It's been tempting for a while now. Always happy to remove code.

[johnslavik]

Are you willing to work on the effort? I'll add some subtasks in the original description.

Thanks, sure!