Rethink binary distribution for faster bug-fix releases

[shinaoka]

We should adopt a clearer short-term release strategy for SparseIR.jl so that small validated bug fixes can be released immediately, without waiting for a manual Yggdrasil binary update.

Recommended direction

The recommended setup is:

• use a manual release workflow on the sparse-ir-rs side
• that workflow runs cargo publish first
• only if cargo publish succeeds, continue with binary release steps
• build platform binaries with BinaryBuilder.jl
• publish artifact tarballs only, not JLLs, for downstream consumption
• let SparseIR.jl consume those published artifacts directly

In other words, the recommended path is not to keep doing manual Yggdrasil updates for every bug fix, and not to make SparseIR.jl build Rust on every user machine as the main strategy. The main recommendation is a GitHub-driven release workflow on the Rust side, gated by successful cargo publish, with BinaryBuilder.jl used to produce distributable binary artifacts.

Motivation

Right now, small fixes in the Rust/C API layer are harder to ship quickly than they should be. Even when a fix is localized and validated, downstream binary distribution still adds too much latency.

A concrete example is the threaded Matsubara fit fix in #120. That kind of fix should be releasable quickly once the Rust side is known to be correct.

How BinaryBuilder.jl fits into the recommendation

The intended role of BinaryBuilder.jl here is narrow and specific: use it as the multi-platform build backend, but do not make JLL generation the main downstream interface.

The important conclusion is that the output we actually want downstream is the set of platform-specific artifact tarballs. In other words:

• BinaryBuilder.jl is used to build release binaries reproducibly across platforms
• the release workflow should publish the resulting binary tarballs as artifacts
• SparseIR.jl should consume those published artifacts directly
• generating a full JLL/Yggdrasil-style wrapper is not the recommended path for urgent bug-fix releases

This keeps the useful part of BinaryBuilder.jl (reproducible cross-platform binary builds) while avoiding the operational cost of a full JLL/Yggdrasil workflow for each small fix.

Why this is the recommended option

This recommendation has a few advantages:

• crates.io remains the release truth, because the workflow only continues after successful cargo publish
• the published binary artifacts correspond to the exact published Rust source
• release latency for urgent bug fixes stays low
• SparseIR.jl can stay on an artifact-based consumption model
• BinaryBuilder.jl can still be used for multi-platform builds, without requiring a full JLL/Yggdrasil workflow for each fix

Important technical point

This is not only about automation. It is also about dependency control. The release pipeline should make the runtime dependency policy explicit, audit produced binaries in CI, and fail if unexpected shared-library dependencies appear.

Scope of this issue

This issue is about the downstream SparseIR.jl release/distribution strategy and how it should consume Rust-side release artifacts. The upstream Rust-side workflow itself is discussed separately in the linked upstream issue.

Related:

• Concurrent fit calls panic with RefCell already borrowed #120
• upstream: Investigate BinaryBuilder.jl and alternative GitHub-native release workflows sparse-ir-rs#172

[shinaoka]

Related upstream discussion on the Rust side: SpM-lab/sparse-ir-rs#172

[mwallerb]

You (maybe with some AI help? ;), write: "... downstream binary distribution still adds too much latency."

What is "too much"? How long is that release latency, really? It cannot be more than a few hours, surely ...? I am having difficulty envisioning an issue with sparse-ir that is so severe and urgent that we cannot wait for downstream to rebuild.

The solution here seems jury-rigged and poorly motivated.

[shinaoka]

A new build must be approved by a human maintainer of Yggdrasil.

Histogram of time from PR creation to merge (excluding new packages, n=963):

  0- 15m ██████████████████████████████████████████ 123
 15- 30m ██████████████████████████               77
 30- 45m ████████████████████                     59
 45m- 1h ██████████                               31
  1-  2h ████████████████████████████             83
  2-  3h ██████████████████                       55
  3-  6h █████████████████████████████            88
  6- 12h ██████████████████████████████           92
 12- 24h ████████████████████████████████         95
  1-  2d ██████████████████████████               78
  2-  3d ███████████                              34
  3-  5d ██████████████████                       54
  5-  7d ███████                                  21
  1-  2w ███████████                              34
  2w+    █████████████                            39

[shinaoka]

(I hope that they introduce automated approvals for minor version updates by AI bots... Not yet happening )