[Bug]: crashed at MemIndexPage::Unpin on close eloqstore during running jepsen test on standby

[lzxddz]

What happened?

Steps to Reproduce

Environment

• OS:
• Version:
• Any other relevant details:
• Any related issues:

[lzxddz]

core dump

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=136241779496512) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=136241779496512) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=136241779496512, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007be950042476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007be9500287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007be953cd2712 in __sanitizer::Abort () at ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cpp:151
#6  0x00007be953cde2cc in __sanitizer::Die () at ../../../../src/libsanitizer/sanitizer_common/sanitizer_termination.cpp:58
#7  0x00007be953cbd77c in __asan::ScopedInErrorReport::~ScopedInErrorReport (this=0x7be8b6328aa6, __in_chrg=<optimized out>)
    at ../../../../src/libsanitizer/asan/asan_report.cpp:190
#8  0x00007be953cbd015 in __asan::ReportGenericError (pc=101690834341551, bp=bp@entry=136239419397920, sp=sp@entry=136239419397904,
    addr=88441983193956, is_write=is_write@entry=false, access_size=access_size@entry=4, exp=0, fatal=true)
    at ../../../../src/libsanitizer/asan/asan_report.cpp:478
#9  0x00007be953cbdd3b in __asan::__asan_report_load4 (addr=<optimized out>) at ../../../../src/libsanitizer/asan/asan_rtl.cpp:120
#10 0x00005c7cbe0eceaf in eloqstore::MemIndexPage::Unpin (this=0x507000fdd360)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/include/storage/mem_index_page.h:120
#11 0x00005c7cbe0eccd2 in eloqstore::MemIndexPage::Handle::Reset (this=0x7be8b6329970, page=0x0)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/include/storage/mem_index_page.h:65
#12 0x00005c7cbe13e5ab in eloqstore::MemIndexPage::Handle::~Handle (this=0x7be8b6329970, __in_chrg=<optimized out>)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/include/storage/mem_index_page.h:58
#13 0x00005c7cbe2bd8ab in eloqstore::IndexPageManager::FindPage (this=0x51d00011df40, mapping=0x50f0003113b0, page_id=1)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/src/storage/index_page_manager.cpp:472
#14 0x00005c7cbe1961aa in eloqstore::ScanIterator::BuildIndexStack (this=0x7be8b6329f10, key=...)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/src/tasks/scan_task.cpp:93
#15 0x00005c7cbe195c23 in eloqstore::ScanIterator::Seek (this=0x7be8b6329f10, key=..., ttl=false)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/src/tasks/scan_task.cpp:57
#16 0x00005c7cbe197bbb in eloqstore::ScanTask::Scan (this=0x7be8d004d9c8)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/src/tasks/scan_task.cpp:299
#17 0x00005c7cbe106ccc in operator() (__closure=0x7be8b632af38)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/src/storage/shard.cpp:451
#18 0x00005c7cbe10c227 in operator() (__closure=0x7be8b632af38, sink=...)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/include/storage/shard.h:130
#19 0x00005c7cbe11e98c in std::__invoke_impl<boost::context::continuation, eloqstore::Shard::StartTask<eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()> >(eloqstore::KvTask*, eloqstore::KvRequest*, eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()>)::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(std::__invoke_other, struct {...} &) (__f=...)
    at /usr/include/c++/11/bits/invoke.h:61
#20 0x00005c7cbe11e2cb in std::__invoke<eloqstore::Shard::StartTask<eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()> >(eloqstore::KvTask*, eloqstore::KvRequest*, eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()>)::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(struct {...} &) (__fn=...) at /usr/include/c++/11/bits/invoke.h:97
#21 0x00005c7cbe11dc50 in std::invoke<eloqstore::Shard::StartTask<eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()> >(eloqstore::KvTask*, eloqstore::KvRequest*, eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()>)::<lambda(boost::context::continuation&&)>&, boost::context::continuation>(struct {...} &) (__fn=...) at /usr/include/c++/11/functional:98
#22 0x00005c7cbe11b397 in boost::context::detail::capture_record<boost::context::continuation, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>&, eloqstore::Shard::StartTask<eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()> >(eloqstore::KvTask*, eloqstore::KvRequest*, eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()>)::<lambda(boost::context::continuation&&)> >::run(void) (this=0x7be8b632ab00) at /usr/include/boost/context/continuation_ucontext.hpp:267
#23 0x00005c7cbe11903e in boost::context::detail::entry_func<boost::context::detail::capture_record<boost::context::continuation, boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>&, eloqstore::Shard::StartTask<eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()> >(eloqstore::KvTask*, eloqstore::KvRequest*, eloqstore::Shard::ProcessReq(eloqstore::KvRequest*)::<lambda()>)::<lambda(boost::context::continuation&&)> > >(void *) (data=0x7be8b632ab00)
    at /usr/include/boost/context/continuation_ucontext.hpp:68
#24 0x00007be95005a130 in ?? () at ../sysdeps/unix/sysv/linux/x86_64/__start_context.S:90 from /lib/x86_64-linux-gnu/libc.so.6
#25 0x0000000000000000 in ?? ()
(gdb) f 10
#10 0x00005c7cbe0eceaf in eloqstore::MemIndexPage::Unpin (this=0x507000fdd360)
    at /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/include/storage/mem_index_page.h:120
120     /home/eloq/eloqkv/data_substrate/store_handler/eloq_data_store_service/eloqstore/include/storage/mem_index_page.h: No such file or directory.
(gdb) p ref_cnt_
$1 = 20