[Bug]: LocalCcShards::UpdateSlices assert(!slice_split_keys.empty()) failed

[xiexiaoy]

Core was generated by `./bin/eloqdoc --config=./etc/mongod_coro.conf --data_substrate_config=./etc/dat'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
Download failed: 无效的参数.  Continuing without source file ./nptl/./nptl/pthread_kill.c.                                                                                                                                                                                                            
44	./nptl/pthread_kill.c: 没有那个文件或目录.
[Current thread is 1 (Thread 0x7f20a51bb6c0 (LWP 859208))]
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007f2119a3c3b6 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007f2119a2287c in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007f2119a2279b in __assert_fail_base (fmt=0x7f2119bb9f68 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7f2109f8b3b9 "!slice_split_keys.empty()", 
    file=file@entry=0x7f2109f88fe8 "/home/xiexy/eloqdata-mongo/src/mongo/db/modules/eloq/data_substrate/tx_service/src/cc/local_cc_shards.cpp", line=line@entry=5291, 
    function=function@entry=0x7f2109f8b2b8 "void txservice::LocalCcShards::UpdateSlices(const txservice::TableName&, const txservice::TableSchema*, txservice::StoreRange*, bool, const std::vector<txservice::FlushRecord>&, UpdateSliceStatus&)") at ./assert/assert.c:92
#6  0x00007f2119a33b66 in __assert_fail (assertion=0x7f2109f8b3b9 "!slice_split_keys.empty()", file=0x7f2109f88fe8 "/home/xiexy/eloqdata-mongo/src/mongo/db/modules/eloq/data_substrate/tx_service/src/cc/local_cc_shards.cpp", line=5291, 
    function=0x7f2109f8b2b8 "void txservice::LocalCcShards::UpdateSlices(const txservice::TableName&, const txservice::TableSchema*, txservice::StoreRange*, bool, const std::vector<txservice::FlushRecord>&, UpdateSliceStatus&)") at ./assert/assert.c:101
#7  0x00007f21099d46c5 in txservice::LocalCcShards::UpdateSlices (this=0x221b2798400, table_name=..., schema=0x221dcad0300, store_range=0x221d65d0780, all_data_exported=false, data_sync_vec=std::vector of length 6143, capacity 6210 = {...}, status=...)
    at /home/xiexy/eloqdata-mongo/src/mongo/db/modules/eloq/data_substrate/tx_service/src/cc/local_cc_shards.cpp:5291
#8  0x00007f21099ce549 in txservice::LocalCcShards::DataSyncForRangePartition (this=0x221b2798400, data_sync_task=std::shared_ptr<txservice::DataSyncTask> (use count 2, weak count 0) = {...}, worker_idx=0)
    at /home/xiexy/eloqdata-mongo/src/mongo/db/modules/eloq/data_substrate/tx_service/src/cc/local_cc_shards.cpp:4172
#9  0x00007f21099c88e8 in txservice::LocalCcShards::DataSyncWorker (this=0x221b2798400, worker_idx=0) at /home/xiexy/eloqdata-mongo/src/mongo/db/modules/eloq/data_substrate/tx_service/src/cc/local_cc_shards.cpp:2945
#10 0x00007f21099bb4a6 in operator() (__closure=0x221b3da8ec8) at /home/xiexy/eloqdata-mongo/src/mongo/db/modules/eloq/data_substrate/tx_service/src/cc/local_cc_shards.cpp:286
#11 0x00007f21099ed08f in std::__invoke_impl<void, txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> >(std::__invoke_other, struct {...} &&) (__f=...) at /usr/include/c++/12/bits/invoke.h:61
#12 0x00007f21099ecdbf in std::__invoke<txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> >(struct {...} &&) (__fn=...) at /usr/include/c++/12/bits/invoke.h:96
#13 0x00007f21099ecb44 in std::thread::_Invoker<std::tuple<txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x221b3da8ec8) at /usr/include/c++/12/bits/std_thread.h:279
#14 0x00007f21099ec9b2 in std::thread::_Invoker<std::tuple<txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> > >::operator()(void) (this=0x221b3da8ec8) at /usr/include/c++/12/bits/std_thread.h:286
#15 0x00007f21099ec7ac in std::thread::_State_impl<std::thread::_Invoker<std::tuple<txservice::LocalCcShards::StartBackgroudWorkers()::<lambda()> > > >::_M_run(void) (this=0x221b3da8ec0) at /usr/include/c++/12/bits/std_thread.h:231
#16 0x00007f2119ee6363 in std::execute_native_thread_routine (__p=0x221b3da8ec0) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:104
#17 0x00007f2119a8f6ba in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:444
#18 0x00007f2119b1e120 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) frame 7
#7  0x00007f21099d46c5 in txservice::LocalCcShards::UpdateSlices (this=0x221b2798400, table_name=..., schema=0x221dcad0300, store_range=0x221d65d0780, all_data_exported=false, data_sync_vec=std::vector of length 6143, capacity 6210 = {...}, status=...)
    at /home/xiexy/eloqdata-mongo/src/mongo/db/modules/eloq/data_substrate/tx_service/src/cc/local_cc_shards.cpp:5291
5291	            assert(!slice_split_keys.empty());
(gdb) frame 7
#7  0x00007f21099d46c5 in txservice::LocalCcShards::UpdateSlices (this=0x221b2798400, table_name=..., schema=0x221dcad0300, store_range=0x221d65d0780, all_data_exported=false, data_sync_vec=std::vector of length 6143, capacity 6210 = {...}, status=...)
    at /home/xiexy/eloqdata-mongo/src/mongo/db/modules/eloq/data_substrate/tx_service/src/cc/local_cc_shards.cpp:5291
5291	            assert(!slice_split_keys.empty());
(gdb) list
5286	        uint32_t subslice_post_ckpt_size = 0;
5287	        std::vector<SliceChangeInfo> slice_split_keys =
5288	            std::move(status.paused_split_keys_);
5289	        if (status.paused_slice_)
5290	        {
5291	            assert(!slice_split_keys.empty());
5292	            uint32_t paused_subslice_post_ckpt_size =
5293	                slice_split_keys.back().post_update_size_;
5294	            subslice_post_ckpt_size =
5295	                paused_subslice_post_ckpt_size < avg_subslice_size

PR eloqdata/eloqdoc#361 with test case tests/jstests/eloq_basic/ttl_basic_test.js.

      -DCMAKE_BUILD_TYPE=Debug \
      -DRANGE_PARTITION_ENABLED=ON \
      -DEXT_TX_PROC_ENABLED=ON \
      -DSTATISTICS=ON \
      -DUSE_ASAN=OFF \
      -DELOQ_MODULE_ENABLED=ON \
      -DWITH_LOG_STATE=ROCKSDB \
      -DWITH_DATA_STORE=ELOQDSS_ROCKSDB \
      -DFORK_HM_PROCESS=ON \
      -DOPEN_LOG_SERVICE=OFF \

$ cat ../etc/data_substrate.conf 
[local]
core_number=2
checkpoint_interval=10
node_memory_limit_mb=2048
enable_io_uring=true
raft_log_async_fsync=true
enable_data_store=true
enable_wal=true
eloq_data_path=/home/xiexy/edisk/service/eloqdata-mongodb/data/data
event_dispatcher_num=1
#auto_redirect=true
enable_mvcc=true
# txlog_rocksdb_cloud_bucket_prefix=eloqdoc-
# txlog_rocksdb_cloud_bucket_name=test
# txlog_rocksdb_cloud_object_path=txlog
# txlog_rocksdb_cloud_s3_endpoint_url=http://127.0.0.1:9900

[cluster]
ip_port_list=127.0.0.1:16379
#txlog_service_list=127.0.0.1:9000
#txlog_group_replica_num=1

[store]
# rocksdb_cloud_bucket_prefix=eloqdoc-
# rocksdb_cloud_bucket_name=test
# rocksdb_cloud_object_path=datastore
# rocksdb_cloud_s3_endpoint_url=http://127.0.0.1:9900
# aws_access_key_id=minioadmin
# aws_secret_key=minioadmin

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

_{You can configure auto-planning by selecting labels in the issue_enrichment configuration.}

🔗 Similar Issues

Possible Duplicates

• [Bug]: LocalCcShards::UpdateSlices assert(!slice_split_keys.empty()); #95

Related Issues

• [Bug]: ScanSliceDeltaSizeCcForRangePartition - Invalid access at address: 0x18b67b010 #197
• [Bug]: buffer commands on leader node eloqkv#195
• [Bug]: EloqRecordStoreCursor::seekExact store_record set to nullptr by concurrent EloqIndexCursor::seekExact eloqdoc#374
• [Bug]: Crashed at FindSliceKeys on test eloqdoc cluster (3 nodes). #171
• [Bug]: Crash during run eloqkv jepsen single test (ASAN enabled) eloqstore#201

🔗 Related PRs

#139 - Improve hash-partition data sync scheduling and memory control. [merged]
#172 - Fix FindSliceKeys crashed for data_sync_vec empty [merged]
eloqdata/eloqkv#265 - Support multi dss shard [merged]
eloqdata/eloqsql#158 - Support multi dss shards [closed]
#250 - fix bug and refactor init process [merged]

👤 Suggested Assignees

• lzxddz
• xiexiaoy
• MrGuin
• yi-xmu
• thweetkomputer

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[xiexiaoy]

Similar issue #95

[yi-xmu]

This assert will be triggered in the following situation: a slice needs to be split, but the first few keys in the slice are all in a DELETED state.