Development of tensor4all-rs and Tensor4all.jl

[shinaoka]

@rittermarc @vondelft @grothesque @SamuelBadr @terasakisatoshi @Waintal @yurielnf @pwulles @Ken-physics @MFrankenbach @GiggleLiu @mwallerb @Ryo-wtnb11

I am wondering how to organize the new package Tensor4all.jl and the rust backend tensor4all-rs (and a Python wrapper?).
https://github.com/tensor4all/tensor4all-rs
https://github.com/tensor4all/Tensor4all.jl

QuanticsGrids.jl, QuanticsTCI.jl, and TensorCrossInterpolation are well organized and kind of stable.
I am asking myself if we should replace them by Julia wrappers, which may cause technical troubles in the early stages of the development. It is also very challenging to keep the backward compatibility of the API.

A more preferable option is that the new package only provides functionalities for tree tensor networks.
Then, we do not have to strictly keep the backward compatibility.

We have a Julia prototype based on TCI2:
https://github.com/tensor4all/TreeTCI.jl

The Grenoble team has a C++ implementation of tree TCI.
We could somehow merge them into a single rust crate, and provide its Julia and Python wrapper.

What do you think?

[shinaoka]

[@tensor4all/developers Now, we have a developer team.]

[shinaoka]

A Rust port:

https://github.com/tensor4all/tensor4all-rs/tree/main/crates/tensor4all-treetci

[GiggleLiu]

Will you consider prioritize CLI? Given a CLI, agents can use and test the package much easier. I am a bit afraid that porting Julia adds another layer of maintainence complexity.

[shinaoka]

A very interesting idea. Do you mean something like this?

tensor4all einsum "ij,jk->ik" A.npy B.npy -o C.npy
tensor4all tci --func "sin(x+y)" --dims 64 64 --tol 1e-10

Another main reason for the Julia and Python bindings is to stress-test the C-API with real usage. A CLI can only cover a small part of the API, so it won’t be enough for that. Once the C-API is stable, Fortran and C++ users can build on it reliably.

[mwallerb]

@shinaoka Agreed. I do not think we can expect users to port all of their code to Rust. I also do not think we can expect users to prototype and do exploration in Rust. From that perspective, to be frank, we are now in the common but very dangerous space where we have an aging but stable Julia ecosystem and an new and shiny but incomplete Rust ecosystem, in short, two unfinished products.

This is why, in my view, interoperability of the two ecosystems should be the top priority - at the very minimum, this means an extremely well-tested Julia shim and easy-to-use conversion routines between current and new code. Then one could mix and match, port piece-by-piece etc.

Edited to add: @GiggleLiu Oh, I think I misunderstood: you didn't mean the Julia wrapper as a whole, just the port of TCI?

[GiggleLiu]

I knew the quantum chemistry people prefers cli tools. But not many from tensor network community favor cli. cli is mainly for exposing higher level APIs, like a TCI application, or DMRG application. e.g.

dmrg heisenberg J=1.2 N=10

@shinaoka no need to be such a lower level API. I mean, implement the application level with rust alone may be beneficial to stabilize the code. @mwallerb right, not the whole set of api.

[shinaoka]

The opinions of both of you make sense. The tensor type in tensor4all-rs is already interconvertible with ITensor's type (without symmetries).

• Each high-level Rust application code should provide a CI. Example, Bethe-Salpeter equation solver.
• News features implemented in Rust (like tree TCI) should be exposed to Julia via C-AFI, so that end users can integrate the new features into their Julia ecosystem.

[shinaoka]

Does this sound reasonable? @mwallerb @SamuelBadr @rittermarc

tensor4all/Tensor4all.jl#21

[shinaoka]

This is a continuation of our discussions in the tensor4all.jl room in the tensor4all space on Matrix.

Here is my current understanding of the situation.
We should keep interoperability as a goal, but we should first solidify the Rust foundation before deciding its precise form.

Facts

• Core development on the Asia side has already shifted to Rust.
• tensor4all-rs has already gained important capabilities that are still missing in the Julia ecosystem like Tree TCI, tree tensor networks. This will acquire soon higher-order automatic differentiation via tenferro-rs. This will likely be critical for solving ill-conditioned equations.

Current impression

After spending a few weeks developing Tensor4all.jl, my current impression is as follows:

• Maintaining the C API is becoming increasingly challenging . I think we can be confident about its stability only after the underlying foundation has been sufficiently stabilized. We already have more than 200 functions in tensor4all-capi, and in addition we also need to maintain the Julia bindings.
• Some parts of tensor4all-rs are already well tested and benchmarked, but other parts have not yet been exercised sufficiently in real use cases. We should validate each component carefully before putting an FFI layer on top of it.

Suggestions

• We should first test and solidify tensor4all-rs before adding further FFI layers. Otherwise, we risk increasing development complexity unnecessarily.
• We should encourage our collaborators to test the Rust library with AI coding agents, and observe how easily they can use it with reasonably inexpensive models, for example models in the ~$20/month range.
• To make this possible, we need to improve the documentation, especially by adding more example code. Otherwise, AI agents will have to inspect the Rust codebase too deeply just to figure out how to use the library.
• We should decide what to expose to other languages, and how to expose it, only after the Rust foundation has been solidified and we have assessed the AI-agentic coding experience of our collaborators.
• For now, we should simply keep the current Julia ecosystem alive, including TensorCrossInterpolation.jl, without expanding its scope too aggressively.

Next steps

• Port BubbleTeaCI.jl to Rust, in order to determine whether the current Rust crates already provide all the necessary functionality and to further solidify the foundation layer. We will add necessary features to tensor4all-rs.
• Ask younger collaborators to reproduce simple Julia examples using tensor4all-rs, with inexpensive models such as GPT-5.4 mini or OSS models, and observe where they struggle.

[shinaoka]

The following reflects my personal view on the longer-term ecosystem. It is not the main point I want to argue here.

• In the long run, Python + Rust may be a better combination. Python does not require precompilation in the same way, and its community is much larger than Julia’s.
• I am not very optimistic about Julia’s long-term future. Julia is reasonably fast and relatively easy for humans to write, but that advantage matters less than before.
• Writing high-performance parallel code in Julia can also be challenging because of GC-related issues.
• For core development, I currently see limited value in using Julia. At this point, Rust seems to offer a much faster path for serious development.
• I am testing OSS models like GLM-5.1. I want to see if they are enough for end users.