Panic: Re-org detected, should be non-empty in this scope

[Tomas-M]

I get this error while running zingo-cli in non-interactive mode (eg. calling with zingo-cli --waitsync messages)
Using current 'stable' branch compiled from sources.

If that helps, I can provide the zingo-wallet.dat file (I have copies of all zingo-wallet.dat files before and after I run zingo-cli, including debug logs, and I can confirm that previous run of zingo-cli successfully ended with standart output - Save task shutdown successfully. Zingo CLI quit successfully.) and the subsequent run always causes this panic. So you can test with the wallet.dat file if you need to debug this.

2026-02-19T17:25:06.606467Z  INFO pepper_sync::sync: Re-org detected.

thread 'tokio-runtime-worker' panicked at /usr/src/zingolib-current/zingolib/pepper-sync/src/sync.rs:1284:10:
should be non-empty in this scope
stack backtrace:
   0:     0x555b33edcac2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hdcfcb6d4c8489523
   1:     0x555b33f069e3 - core::fmt::write::h8a494366950f23bb
   2:     0x555b33ed7fd3 - std::io::Write::write_fmt::h6556609fca33d0b1
   3:     0x555b33edc912 - std::sys::backtrace::BacktraceLock::print::hb2a626a81e06b2dc
   4:     0x555b33ede283 - std::panicking::default_hook::{{closure}}::h4f78485264f12d10
   5:     0x555b33ede065 - std::panicking::default_hook::h2c66fc99e962531d
   6:     0x555b33edec95 - std::panicking::rust_panic_with_hook::h33ac55f64bbd807d
   7:     0x555b33edea2a - std::panicking::begin_panic_handler::{{closure}}::h30e7cb89678a57fe
   8:     0x555b33edcfc9 - std::sys::backtrace::__rust_end_short_backtrace::hed60f27456c16ced
   9:     0x555b33ede6bd - __rustc[de2ca18b4c54d5b8]::rust_begin_unwind
  10:     0x555b33f03c20 - core::panicking::panic_fmt::h62f63d096dd276af
  11:     0x555b33f03afb - core::option::expect_failed::h091923fb77e757a1
  12:     0x555b32fe2741 - pepper_sync::sync::truncate_wallet_data::hf2226a2ac1fbc89d
  13:     0x555b32fdc12e - pepper_sync::sync::clear_wallet_data::h3ec28a5e9547513f
  14:     0x555b32fe2279 - pepper_sync::sync::truncate_wallet_data::hf2226a2ac1fbc89d
  15:     0x555b32fdd946 - pepper_sync::sync::process_scan_results::{{closure}}::hefc88dec32c0118f
  16:     0x555b32fe7ad2 - pepper_sync::sync::sync::{{closure}}::hc4d35c81067b895c
  17:     0x555b33020d5a - tokio::runtime::task::core::Core<T,S>::poll::h70eda8f5a4817a95
  18:     0x555b32da4f43 - tokio::runtime::task::raw::poll::h5c3ab6105f3834bc
  19:     0x555b33a98544 - tokio::runtime::scheduler::multi_thread::worker::Context::run_task::h66572328ec30f808
  20:     0x555b33a97a90 - tokio::runtime::scheduler::multi_thread::worker::Context::run::h1fe9ac17058796dc
  21:     0x555b33aaa8a6 - tokio::runtime::context::scoped::Scoped<T>::set::ha9a0bf55efccc784
  22:     0x555b33aaa7de - tokio::runtime::context::runtime::enter_runtime::h1d0e5d625db17f32
  23:     0x555b33a9731d - tokio::runtime::scheduler::multi_thread::worker::run::h35d52f4501a9e956
  24:     0x555b33aa01ad - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::haf94fb902865efbe
  25:     0x555b33a9f647 - tokio::runtime::task::core::Core<T,S>::poll::hd685ab8d4e6910c6
  26:     0x555b33a8faf1 - tokio::runtime::task::harness::Harness<T,S>::poll::h978d9813c0198524
  27:     0x555b33a94099 - tokio::runtime::blocking::pool::Inner::run::h292e80295c88a510
  28:     0x555b33aab280 - std::sys::backtrace::__rust_begin_short_backtrace::h9ed294b7c8ed8b5b
  29:     0x555b33aa38c2 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h7a4c144967e490d3
  30:     0x555b33ee1f4f - std::sys::pal::unix::thread::Thread::new::thread_start::hb6e99e73da4d28f8
  31:     0x7fd54c8a31f5 - <unknown>
  32:     0x7fd54c92389c - <unknown>
  33:                0x0 - <unknown>

thread '<unnamed>' panicked at zingolib/src/lightclient/sync.rs:114:47:
task panicked: JoinError::Panic(Id(17), "should be non-empty in this scope", ...)
stack backtrace:
   0:     0x555b33edcac2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hdcfcb6d4c8489523
   1:     0x555b33f069e3 - core::fmt::write::h8a494366950f23bb
   2:     0x555b33ed7fd3 - std::io::Write::write_fmt::h6556609fca33d0b1
   3:     0x555b33edc912 - std::sys::backtrace::BacktraceLock::print::hb2a626a81e06b2dc
   4:     0x555b33ede283 - std::panicking::default_hook::{{closure}}::h4f78485264f12d10
   5:     0x555b33ede065 - std::panicking::default_hook::h2c66fc99e962531d
   6:     0x555b33edec95 - std::panicking::rust_panic_with_hook::h33ac55f64bbd807d
   7:     0x555b33edea2a - std::panicking::begin_panic_handler::{{closure}}::h30e7cb89678a57fe
   8:     0x555b33edcfc9 - std::sys::backtrace::__rust_end_short_backtrace::hed60f27456c16ced
   9:     0x555b33ede6bd - __rustc[de2ca18b4c54d5b8]::rust_begin_unwind
  10:     0x555b33f03c20 - core::panicking::panic_fmt::h62f63d096dd276af
  11:     0x555b33f04046 - core::result::unwrap_failed::h95bc3f5a607b2c95
  12:     0x555b333c2ee7 - zingolib::lightclient::sync::<impl zingolib::lightclient::LightClient>::poll_sync::he892ce37462a427b
  13:     0x555b33037968 - <zingo_cli::commands::SyncCommand as zingo_cli::commands::Command>::exec::h234de0e96aaa772e
  14:     0x555b3303cb51 - zingo_cli::commands::do_user_command::h4ea2536e038665db
  15:     0x555b32ddcfb2 - std::sys::backtrace::__rust_begin_short_backtrace::h83b07d504c46a2b9
  16:     0x555b332112e8 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h76ce0ba564601457
  17:     0x555b33ee1f4f - std::sys::pal::unix::thread::Thread::new::thread_start::hb6e99e73da4d28f8
  18:     0x7fd54c8a31f5 - <unknown>
  19:     0x7fd54c92389c - <unknown>
  20:                0x0 - <unknown>

thread 'main' panicked at zingo-cli/src/lib.rs:563:14:
called `Result::unwrap()` on an `Err` value: SendError { .. }
stack backtrace:
   0:     0x555b33edcac2 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hdcfcb6d4c8489523
   1:     0x555b33f069e3 - core::fmt::write::h8a494366950f23bb
   2:     0x555b33ed7fd3 - std::io::Write::write_fmt::h6556609fca33d0b1
   3:     0x555b33edc912 - std::sys::backtrace::BacktraceLock::print::hb2a626a81e06b2dc
   4:     0x555b33ede283 - std::panicking::default_hook::{{closure}}::h4f78485264f12d10
   5:     0x555b33ede065 - std::panicking::default_hook::h2c66fc99e962531d
   6:     0x555b33edec95 - std::panicking::rust_panic_with_hook::h33ac55f64bbd807d
   7:     0x555b33edea2a - std::panicking::begin_panic_handler::{{closure}}::h30e7cb89678a57fe
   8:     0x555b33edcfc9 - std::sys::backtrace::__rust_end_short_backtrace::hed60f27456c16ced
   9:     0x555b33ede6bd - __rustc[de2ca18b4c54d5b8]::rust_begin_unwind
  10:     0x555b33f03c20 - core::panicking::panic_fmt::h62f63d096dd276af
  11:     0x555b33f04046 - core::result::unwrap_failed::h95bc3f5a607b2c95
  12:     0x555b32d5e93f - zingo_cli::dispatch_command_or_start_interactive::h1059789e0fb1d7ab
  13:     0x555b32d60082 - zingo_cli::run_cli::hb541fcd3a4fe6572
  14:     0x555b32cf1ae5 - zingo_cli::main::h5fe045482b0d2537
  15:     0x555b32cf33e3 - std::sys::backtrace::__rust_begin_short_backtrace::h25058f0d22ca2063
  16:     0x555b32cf0f59 - std::rt::lang_start::{{closure}}::h0c0b0b980401f14f
  17:     0x555b33eceac0 - std::rt::lang_start_internal::h34f9328d113fd60a
  18:     0x555b32cf1b85 - main
  19:     0x7fd54c84124a - <unknown>
  20:     0x7fd54c841305 - __libc_start_main
  21:     0x555b32ce9ba1 - _start
  22:                0x0 - <unknown>

[juanky201271]

Thanks for reporting, We are going to try to reproduce it first, and @Oscar-Pepper will take a look ASAP.

[Oscar-Pepper]

hi @Tomas-M i am taking a closer look now. the wallet.dat would be very helpful! thanks for raising this issue

[Tomas-M]

Wallet file can be downloaded here:
https://zec-pay.com/zingo-wallet.dat
It is the wallet file after successful zingo-cli run, and subsequent run causes the described panic

[Tomas-M]

The panic in particular in the reported wallet file happened during this command:

zingo-cli --waitsync sync status

Here is full output which preceeded the panic:

Launching sync task...
Launching save task...
2026-02-19T17:25:06.370053Z  INFO pepper_sync::sync: Starting sync...
2026-02-19T17:25:06.370102Z DEBUG pepper_sync::client: Fetching mempool stream
2026-02-19T17:25:06.370114Z DEBUG pepper_sync::client::fetch: Fetching chain tip.
2026-02-19T17:25:06.449619Z DEBUG pepper_sync::client::fetch: Fetching raw transactions in block range BlockHeight(3245913)..BlockHeight(3246023) for address "t1bF72SzLzKT66zzsVhhfRtfDfRPwDwBivg"
2026-02-19T17:25:06.464520Z DEBUG pepper_sync::client::fetch: Fetching raw transactions in block range BlockHeight(3245913)..BlockHeight(3246023) for address "t1doq3dEN143VdXBgsxSHzHE3YHPxB8sa8B"
2026-02-19T17:25:06.480727Z DEBUG pepper_sync::client::fetch: Fetching raw transactions in block range BlockHeight(3245913)..BlockHeight(3246023) for address "t1TtD4MefkeXwegLM8NegQRA9sELJkSHDAp"
2026-02-19T17:25:06.495224Z DEBUG pepper_sync::client::fetch: Fetching subtree roots. start index: 1128. shielded protocol: 0
2026-02-19T17:25:06.509457Z DEBUG pepper_sync::client::fetch: Fetching subtree roots. start index: 760. shielded protocol: 1
2026-02-19T17:25:06.524732Z DEBUG pepper_sync::client::fetch: Fetching tree state. BlockHeight(3246022)
2026-02-19T17:25:06.555173Z DEBUG pepper_sync::scan::task: Spawning batcher
2026-02-19T17:25:06.555201Z DEBUG pepper_sync::scan::task: Spawning worker 0
2026-02-19T17:25:06.555237Z DEBUG pepper_sync::scan::task: Spawning worker 1
2026-02-19T17:25:06.555373Z DEBUG pepper_sync::sync: mempool received txid 1bfde08c8d7a7ad8ff2912a209b2dc956a95845d9471d577d4332a73ea048eee at height 3246023
2026-02-19T17:25:06.557736Z DEBUG pepper_sync::sync: mempool received txid c5a3c51679938193e89ba0f23acef528e2b9f6a6e219825bffb369c05a97e8ba at height 3246023
2026-02-19T17:25:06.559702Z DEBUG pepper_sync::client::fetch: Fetching compact blocks. BlockHeight(3246013)..BlockHeight(3246023)
2026-02-19T17:25:06.606467Z  INFO pepper_sync::sync: Re-org detected.
:
:
and then the backtrace already reported in the intial issue

[Oscar-Pepper]

Wallet file can be downloaded here: https://zec-pay.com/zingo-wallet.dat It is the wallet file after successful zingo-cli run, and subsequent run causes the described panic

thank you. I am working on this now!

[Oscar-Pepper]

ok I have debugged and fixed this issue. there was a combination of scan ranges not truncating correctly and the wallet save flag not being set after a wallet clear. this fix is included in #2194 and will be included in the new zingolib release very soon with other fixes and improvements that should improve your experience. @Tomas-M please could you test #2146 again with the changes from #2194 merged to verify there is still an issue? please test our code rather than your fork as we do not hit the unused variable warnings etc.

[Oscar-Pepper]

its important note is that there is still an underyling issue that the sapling checkpoint was missing to cause the wallet clear in the first place. i will raise an issue if one doesnt exist

[Tomas-M]

Thank you. Should I test on the default dev brach, or stable branch?

[Tomas-M]

I tested it with "stable" branch. It looks fine so far. My wallets get synced from cron each 5 minutes, so I will keep it running and report back if there is any problem. It can take few hours or days for the problem to appear.

[zancas]

Since we're still not seeing any reproduction, I will close this for now.