Concurrent fit calls panic with RefCell already borrowed

[SamuelBadr]

Summary

Concurrent calls to SparseIR.fit from multiple Julia threads panic with RefCell borrow errors.

Minimal Reproducer

using SparseIR
using Random
using Base.Threads: @threads, nthreads

Random.seed!(1)
basis = SparseIR.FiniteTempBasis{Fermionic}(0.7, 8.0, 1e-2)
sampling = SparseIR.MatsubaraSampling(basis)
n = length(SparseIR.sampling_points(sampling))
values = ComplexF64.(randn(n, 8), randn(n, 8))

@threads for _ in 1:nthreads()
    fit(sampling, values)
end

How To Run

Threaded (reproduces panic):

julia -t auto /path/to/reproducer.jl

Single-thread control (no panic):

julia -t 1 /path/to/reproducer.jl

Observed Output

thread '<unnamed>' panicked at sparse-ir/src/fitters/complex.rs:634:23:
RefCell already borrowed

Environment

• Julia 1.12.5
• SparseIR.jl 2.1.1
• macOS arm64

I originally hit this in a larger solver stack, then reduced it to the snippet above.

[shinaoka]

Root cause was a thread-unsafe lazy SVD cache in the Rust Matsubara fitters. The full-range path used ComplexMatrixFitter and the positive-only path used ComplexToRealFitter, both backed by RefCell<Option<...>>, so concurrent calls from multiple Julia threads could collide during first-use initialization.

I opened two PRs:

• Rust fix: fix: make Matsubara fit cache thread-safe sparse-ir-rs#171
• Julia docs/cache notes: docs: clarify local Rust rebuild cache behavior #121

What I verified locally:

• original Julia reproducer panicked before the fix
• cargo test -p sparse-ir-capi test_concurrent_matsubara_fit -- --nocapture now passes for both full-range and positive-only Matsubara fit paths
• after rebuilding SparseIR.jl, the original threaded Julia reproducer runs without panic

One extra wrinkle on the Julia side: after Pkg.build(), Julia precompile cache can still hold on to the artifact-provided libsparse_ir_capi path. I documented the workaround in the Julia PR: remove ~/.julia/compiled/.../SparseIR and start a fresh Julia process if that happens.

[shinaoka]

I opened a separate follow-up issue for the release/distribution strategy discussion: #122. The motivation is to make small bug-fix releases much faster, without requiring a manual Yggdrasil binary update each time.

[shinaoka]

Using RustToolChain.jl makes sense in the short term.
@terasakisatoshi What do you think?

[terasakisatoshi]

Using RustToolChain.jl makes sense in the short term. @terasakisatoshi What do you think?

I agree. We should replace cargo with using RustToolChain: cargo; cargo() in the line

[terasakisatoshi]

The #123 is introduced to resolve #120 (comment)

[shinaoka]

Please try out the main branch!　@SamuelBadr

[SamuelBadr]

Oh sorry, I saw this only just now. I tried to upgrade to the main branch in a project also pulling in OvercompleteIR.jl (a branch adapted for SparseIR.jl version 2) and got precompilation failures like this one:

julia> err
PkgPrecompileError: The following 4 direct dependencies failed to precompile:

OvercompleteIR 

Failed to precompile OvercompleteIR [0a76eecd-8221-469d-947c-036d2b9a0785] to "/home/samuel/.julia/compiled/v1.12/OvercompleteIR/jl_cl3uO7".
ERROR: LoadError: InitError: could not load library "/home/samuel/.julia/packages/SparseIR/pIm81/deps/libsparse_ir_capi.so"
/home/samuel/.julia/packages/SparseIR/pIm81/deps/libsparse_ir_capi.so: undefined symbol: dgemm_
Stacktrace:
  [1] _init_sparseir_blas_backend
    @ ~/.julia/packages/SparseIR/pIm81/src/SparseIR.jl:66 [inlined]
  [2] __init__()
    @ SparseIR ~/.julia/packages/SparseIR/pIm81/src/SparseIR.jl:88
  [3] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1443
  [4] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1431
  [5] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}; register::Bool)
    @ Base ./loading.jl:1319
  [6] _include_from_serialized
    @ ./loading.jl:1274 [inlined]
  [7] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
    @ Base ./loading.jl:2115
  [8] _require_search_from_serialized
    @ ./loading.jl:2009 [inlined]
  [9] __require_prelocked(pkg::Base.PkgId, env::String)
    @ Base ./loading.jl:2627
 [10] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2493
 [11] macro expansion
    @ ./loading.jl:2421 [inlined]
 [12] macro expansion
    @ ./lock.jl:376 [inlined]
 [13] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2386
 [14] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2362
 [15] top-level scope
    @ ~/.julia/packages/OvercompleteIR/H3ytb/src/OvercompleteIR.jl:3
 [16] include(mod::Module, _path::String)
    @ Base ./Base.jl:306
 [17] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::Nothing)
    @ Base ./loading.jl:3024
 [18] top-level scope
    @ stdin:5
 [19] eval(m::Module, e::Any)
    @ Core ./boot.jl:489
 [20] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
    @ Base ./loading.jl:2870
 [21] include_string
    @ ./loading.jl:2880 [inlined]
 [22] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:315
 [23] _start()
    @ Base ./client.jl:550
during initialization of module SparseIR
in expression starting at /home/samuel/.julia/packages/OvercompleteIR/H3ytb/src/OvercompleteIR.jl:1
in expression starting at stdin:5