[Bug] doris-be:3.1.3 crashing on startup

[plastikman]

Search before asking

• I had searched in the issues and found no similar issues.

Version

3.1.3

What's Wrong?

When starting Doris only one of the 3 be pods starts, the other 2 crash with the attached error

Here is the final output, but this is the crash

I20251204 21:08:29.351032 154 data_dir.cpp:882] gc data dir path: /opt/apache-doris/be/storage finished
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# 0x00007F2AFC218520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6
 3# raise in /lib/x86_64-linux-gnu/libc.so.6
 4# abort in /lib/x86_64-linux-gnu/libc.so.6
 5# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
 6# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
 7# 0x000055E4EC5EC121 in /opt/apache-doris/be/lib/doris_be
 8# 0x000055E4EC5EC274 in /opt/apache-doris/be/lib/doris_be
 9# std::__throw_system_error(int) at ../../../../../libstdc++-v3/src/c++11/system_error.cc:338
10# 0x000055E4EC6BB1ED in /opt/apache-doris/be/lib/doris_be
11# std::thread* doris::ThreadGroup::create_thread<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)> >(std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread_group.h:65
12# doris::WorkThreadPool<false>::WorkThreadPool(unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /home/zcp/repo_center/doris_release/doris/be/src/util/work_thread_pool.hpp:61
13# doris::PInternalService::PInternalService(doris::ExecEnv*) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:218
14# doris::PInternalServiceImpl::PInternalServiceImpl(doris::StorageEngine&, doris::ExecEnv*) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:259
15# doris::BRpcService::start(int, int) in /opt/apache-doris/be/lib/doris_be
16# main at /home/zcp/repo_center/doris_release/doris/be/src/service/doris_main.cpp:570
17# 0x00007F2AFC1FFD90 in /lib/x86_64-linux-gnu/libc.so.6
18# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
19# _start in /opt/apache-doris/be/lib/doris_be
/opt/apache-doris/be/bin/start_be.sh: line 450:   154 Aborted                 (core dumped) ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" 2>> "${LOG_DIR}/be.out" < /dev/null
[Thu Dec  4 21:08:49 UTC 2025] run post_exit

Here is the full log

[Thu Dec  4 21:08:27 UTC 2025] [info] the host machine support avx2 instruction set.
[Thu Dec  4 21:08:27 UTC 2025] [info] Process conf file be.conf ...
[Thu Dec  4 21:08:27 UTC 2025] [info] use root no password show backends result 1764879474792	doris-cluster-be-0.doris-cluster-be-internal.doris.svc.cluster.local	9050	9060	8040	8060	-1	2025-12-04 20:25:42	2025-12-04 20:26:37	false	false	0	0.000 	0.000 	1006.754 GB	1006.854 GB	0.01 %	0.01 %	0.000 	{"location" : "default"}	java.net.ConnectException: Connection refused		{"lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false,"isActive":true,"isShutdown":true,"currentFragmentNum":0,"lastFragmentUpdateTime":0}	41		32	0.00 	0
1764879474793	doris-cluster-be-2.doris-cluster-be-internal.doris.svc.cluster.local	9050	9060	8040	8060	-1	2025-12-04 21:02:07	2025-12-04 21:08:23	true	false	0	0.000 	0.000 	1006.751 GB	1006.854 GB	0.01 %	0.01 %	0.000 	{"location" : "default"}		doris-3.1.3-rc01-be1b1227a8	{"lastSuccessReportTabletsTime":"2025-12-04 21:07:32","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false,"isActive":true,"isShutdown":false,"currentFragmentNum":0,"lastFragmentUpdateTime":1764882423360}	0	mix	32	16.00 GB	0
1764879474920	doris-cluster-be-1.doris-cluster-be-internal.doris.svc.cluster.local	9050	9060	8040	8060	-1	2025-12-04 20:27:45	2025-12-04 21:01:42	false	false	22	33.809 KB	0.000 	1006.754 GB	1006.854 GB	0.01 %	0.01 %	0.000 	{"location" : "default"}	java.net.ConnectException: Connection refused	doris-3.1.3-rc01-be1b1227a8	{"lastSuccessReportTabletsTime":"2025-12-04 21:01:47","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false,"isActive":true,"isShutdown":true,"currentFragmentNum":0,"lastFragmentUpdateTime":1764882043363}	40		32	32.00 GB	0 .
[Thu Dec  4 21:08:27 UTC 2025] [info] Check myself (doris-cluster-be-1.doris-cluster-be-internal.doris.svc.cluster.local:9050) exist in FE, start be directly ...
the annotations value:
the value not equal!  debug apache.com.doris/runmode
[Thu Dec  4 21:08:27 UTC 2025] run start_be.sh
StdoutLogger 2025-12-04 21:08:27,833 Start time: Thu Dec  4 21:08:27 UTC 2025
I20251204 21:08:27.925618 154 doris_main.cpp:388]  version doris-3.1.3-rc01(AVX2) RELEASE features: -TDE,-HDFS_STORAGE_VAULT,+UI,+AZURE_BLOB,+AZURE_STORAGE_VAULT,-HIVE_UDF,+BE_JAVA_EXTENSIONS (build git://vm-8@be1b1227a89253ba301e5f3c7b41c82a7adf46f6)
Built on Wed, 19 Nov 2025 22:51:16 4Z by vm-8
I20251204 21:08:27.951983 154 jni-util.cpp:107] set final LIBHDFS_OPTS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -Xlog:gc*:/opt/apache-doris/be/log/be.gc.log.:time,uptime:filecount=10,filesize=50M -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED -Djava.security.krb5.conf=/etc/krb5.conf  -Djdk.lang.processReaperUseDefaultStackSize=true
I20251204 21:08:28.973666 154 doris_main.cpp:496] Doris backend JNI is initialized.
I20251204 21:08:28.974579 154 mem_info.cpp:384] Physical Memory: 135065067520, BE Available Physical Memory(consider cgroup): 17179869184, Mem Limit: 14.40 GB, origin config value: 90%, System Mem Available Min Reserve: 819.20 MB, Vm Min Free KBytes: 66.00 MB, Vm Overcommit Memory: 1
I20251204 21:08:28.974611 154 doris_main.cpp:514] Cpu Info:
  Model: AMD EPYC 9V74 80-Core Processor
  Cores: 32
  Max Possible Cores: 32
  L1 Cache: 32.00 KB (Line: 64.00 B)
  L2 Cache: 1.00 MB (Line: 64.00 B)
  L3 Cache: 320.00 MB (Line: 64.00 B)
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    avx
    avx2
  Numa Nodes: 1
  Numa Nodes of Cores: 0->0 | 1->0 | 2->0 | 3->0 | 4->0 | 5->0 | 6->0 | 7->0 | 8->0 | 9->0 | 10->0 | 11->0 | 12->0 | 13->0 | 14->0 | 15->0 | 16->0 | 17->0 | 18->0 | 19->0 | 20->0 | 21->0 | 22->0 | 23->0 | 24->0 | 25->0 | 26->0 | 27->0 | 28->0 | 29->0 | 30->0 | 31->0 |
I20251204 21:08:28.974634 154 doris_main.cpp:515] Disk Info:
  Num disks 3: sr, nvme0n, nvme0n1p
I20251204 21:08:28.974648 154 doris_main.cpp:516] Physical Memory: 17179869184
Memory Limt: 15461882265
CGroup Info: Process CGroup Memory Info (cgroups path: /sys/fs/cgroup, cgroup version: v2): memory limit: 17179869184, memory usage: 496256568
I20251204 21:08:28.975189 154 backend_options.cpp:62] local host ip=172.24.17.238
W20251204 21:08:28.982931 154 timezone_utils.cpp:98] Meet illegal tzdata file: leap-seconds.list. skipped
W20251204 21:08:28.985920 154 timezone_utils.cpp:98] Meet illegal tzdata file: zone1970.tab. skipped
W20251204 21:08:28.986508 154 timezone_utils.cpp:98] Meet illegal tzdata file: leapseconds. skipped
W20251204 21:08:29.001443 154 timezone_utils.cpp:98] Meet illegal tzdata file: zone.tab. skipped
W20251204 21:08:29.030421 154 timezone_utils.cpp:98] Meet illegal tzdata file: tzdata.zi. skipped
W20251204 21:08:29.031971 154 timezone_utils.cpp:98] Meet illegal tzdata file: iso3166.tab. skipped
W20251204 21:08:29.034533 154 timezone_utils.cpp:98] Meet illegal tzdata file: zonenow.tab. skipped
I20251204 21:08:29.037967 154 timezone_utils.cpp:115] Preloaded658 timezones.
E20251204 21:08:29.045794 154 variable.cpp:179] Already exposed `doris_cache_data_page_cache' whose value is `0'
E20251204 21:08:29.045833 154 variable.cpp:179] Already exposed `doris_cache_data_page_cache_persecond' whose value is `0'
I20251204 21:08:29.045851 154 cache_manager.h:47] Register Cache DataPageCache
E20251204 21:08:29.045946 154 variable.cpp:179] Already exposed `doris_cache_index_page_cache' whose value is `0'
E20251204 21:08:29.045964 154 variable.cpp:179] Already exposed `doris_cache_index_page_cache_persecond' whose value is `0'
I20251204 21:08:29.045980 154 cache_manager.h:47] Register Cache IndexPageCache
E20251204 21:08:29.046115 154 variable.cpp:179] Already exposed `doris_cache_pkindex_page_cache' whose value is `0'
E20251204 21:08:29.046139 154 variable.cpp:179] Already exposed `doris_cache_pkindex_page_cache_persecond' whose value is `0'
I20251204 21:08:29.046155 154 cache_manager.h:47] Register Cache PKIndexPageCache
I20251204 21:08:29.046166 154 exec_env_init.cpp:524] Storage page cache memory limit: 2.88 GB, origin config value: 20%
E20251204 21:08:29.046231 154 variable.cpp:179] Already exposed `doris_cache_point_query_row_cache' whose value is `0'
E20251204 21:08:29.046248 154 variable.cpp:179] Already exposed `doris_cache_point_query_row_cache_persecond' whose value is `0'
I20251204 21:08:29.046262 154 cache_manager.h:47] Register Cache PointQueryRowCache
I20251204 21:08:29.046271 154 exec_env_init.cpp:537] Row cache memory limit: 2.88 GB, origin config value: 20%
E20251204 21:08:29.046314 154 variable.cpp:179] Already exposed `doris_cache_segment_cache' whose value is `0'
E20251204 21:08:29.046331 154 variable.cpp:179] Already exposed `doris_cache_segment_cache_persecond' whose value is `0'
I20251204 21:08:29.046345 154 cache_manager.h:47] Register Cache SegmentCache
I20251204 21:08:29.046356 154 exec_env_init.cpp:571] segment_cache_capacity <= fd_number * 1 / 5, fd_number: 1048576 segment_cache_capacity: 209700 min_segment_cache_mem_limit 773094110
E20251204 21:08:29.046383 154 variable.cpp:179] Already exposed `doris_cache_schema_cache' whose value is `0'
E20251204 21:08:29.046401 154 variable.cpp:179] Already exposed `doris_cache_schema_cache_persecond' whose value is `0'
I20251204 21:08:29.046415 154 cache_manager.h:47] Register Cache SchemaCache
I20251204 21:08:29.046428 154 exec_env_init.cpp:579] max file reader cache size is: 349525, resource hard limit is: 1048576, config file_cache_max_file_reader_cache_size is: 1000000
E20251204 21:08:29.046458 154 variable.cpp:179] Already exposed `doris_cache_common_obj_lrucache' whose value is `0'
E20251204 21:08:29.046474 154 variable.cpp:179] Already exposed `doris_cache_common_obj_lrucache_persecond' whose value is `0'
I20251204 21:08:29.046490 154 cache_manager.h:47] Register Cache CommonObjLRUCache
E20251204 21:08:29.046525 154 variable.cpp:179] Already exposed `doris_cache_point_query_lookup_connection_cache' whose value is `0'
E20251204 21:08:29.046542 154 variable.cpp:179] Already exposed `doris_cache_point_query_lookup_connection_cache_persecond' whose value is `0'
I20251204 21:08:29.046559 154 cache_manager.h:47] Register Cache PointQueryLookupConnectionCache
I20251204 21:08:29.046572 154 inverted_index_cache.cpp:59] fd_number: 1048576, inverted index open searcher limit: 209715
E20251204 21:08:29.046658 154 variable.cpp:179] Already exposed `doris_cache_inverted_index_searcher_cache' whose value is `0'
E20251204 21:08:29.046675 154 variable.cpp:179] Already exposed `doris_cache_inverted_index_searcher_cache_persecond' whose value is `0'
I20251204 21:08:29.046691 154 cache_manager.h:47] Register Cache InvertedIndexSearcherCache
I20251204 21:08:29.046702 154 exec_env_init.cpp:600] Inverted index searcher cache memory limit: 1.44 GB, origin config value: 10%
E20251204 21:08:29.046797 154 variable.cpp:179] Already exposed `doris_cache_inverted_index_query_cache' whose value is `0'
E20251204 21:08:29.046814 154 variable.cpp:179] Already exposed `doris_cache_inverted_index_query_cache_persecond' whose value is `0'
I20251204 21:08:29.046828 154 cache_manager.h:47] Register Cache InvertedIndexQueryCache
I20251204 21:08:29.046839 154 exec_env_init.cpp:614] Inverted index query match cache memory limit: 1.44 GB, origin config value: 10%
E20251204 21:08:29.046875 154 variable.cpp:179] Already exposed `doris_cache_query_cache' whose value is `0'
E20251204 21:08:29.046890 154 variable.cpp:179] Already exposed `doris_cache_query_cache_persecond' whose value is `0'
I20251204 21:08:29.046904 154 cache_manager.h:47] Register Cache QueryCache
I20251204 21:08:29.046913 154 exec_env_init.cpp:623] query cache memory limit: 512MB
E20251204 21:08:29.047004 154 variable.cpp:179] Already exposed `doris_cache_mow_delete_bitmap_agg_cache' whose value is `0'
E20251204 21:08:29.047021 154 variable.cpp:179] Already exposed `doris_cache_mow_delete_bitmap_agg_cache_persecond' whose value is `0'
I20251204 21:08:29.047040 154 cache_manager.h:47] Register Cache MowDeleteBitmapAggCache
I20251204 21:08:29.047051 154 cgroup_cpu_ctl.cpp:68] [cgroup_init_path]doris cgroup home path is not specify, if you not use workload group, you can ignore this log.
I20251204 21:08:29.047065 154 exec_env_init.cpp:410] pipeline executors_size set |size=32
I20251204 21:08:29.048720 154 task_scheduler.cpp:61] TaskScheduler set cores|size=32
I20251204 21:08:29.049173 154 fragment_mgr.cpp:906] FragmentMgr cancel worker start working.
I20251204 21:08:29.195452 154 load_path_mgr.cpp:67] Load path configured to []
I20251204 21:08:29.195498 154 result_buffer_mgr.cpp:192] result buffer manager cancel thread begin.
E20251204 21:08:29.202203 154 variable.cpp:179] Already exposed `doris_cache_last_success_channel_cache' whose value is `0'
E20251204 21:08:29.202243 154 variable.cpp:179] Already exposed `doris_cache_last_success_channel_cache_persecond' whose value is `0'
I20251204 21:08:29.202261 154 cache_manager.h:47] Register Cache LastSuccessChannelCache
I20251204 21:08:29.202314 154 wal_manager.cpp:116] wal_dir:/opt/apache-doris/be/storage/wal, tmp_dir:/opt/apache-doris/be/storage/wal/tmp
E20251204 21:08:29.202518 154 variable.cpp:179] Already exposed `doris_cache_tablet_schema_cache' whose value is `0'
E20251204 21:08:29.202543 154 variable.cpp:179] Already exposed `doris_cache_tablet_schema_cache_persecond' whose value is `0'
I20251204 21:08:29.202565 154 cache_manager.h:47] Register Cache TabletSchemaCache
E20251204 21:08:29.202596 154 variable.cpp:179] Already exposed `doris_cache_tablet_column_object_pool' whose value is `0'
E20251204 21:08:29.202613 154 variable.cpp:179] Already exposed `doris_cache_tablet_column_object_pool_persecond' whose value is `0'
I20251204 21:08:29.202630 154 cache_manager.h:47] Register Cache TabletColumnObjectPool
I20251204 21:08:29.202664 154 exec_env_init.cpp:695] The current deployment mode is local.
start BE in local mode
E20251204 21:08:29.202853 154 variable.cpp:179] Already exposed `doris_cache_mow_tablet_version_cache' whose value is `0'
E20251204 21:08:29.202873 154 variable.cpp:179] Already exposed `doris_cache_mow_tablet_version_cache_persecond' whose value is `0'
I20251204 21:08:29.202888 154 cache_manager.h:47] Register Cache MowTabletVersionCache
E20251204 21:08:29.202908 154 variable.cpp:179] Already exposed `doris_cache_create_tablet_rridx_cache' whose value is `0'
E20251204 21:08:29.202924 154 variable.cpp:179] Already exposed `doris_cache_create_tablet_rridx_cache_persecond' whose value is `0'
I20251204 21:08:29.202939 154 cache_manager.h:47] Register Cache CreateTabletRRIdxCache
I20251204 21:08:29.202951 154 storage_engine.cpp:209] starting backend using uid:b548cd5a4d5f68ef-4b4fe5d33c1713b4
I20251204 21:08:29.203083 154 data_dir.cpp:890] path: /opt/apache-doris/be/storage total capacity: 1081101176832, available capacity: 1080993832960, usage: 9.92912e-05, in_use: 0
I20251204 21:08:29.206951 154 data_dir.cpp:201] path: /opt/apache-doris/be/storage, hash: 18393237053768532980
I20251204 21:08:29.207983 154 olap_meta.cpp:73] [Rocksdb] RocksDB version: 5.14.2
I20251204 21:08:29.208002 154 olap_meta.cpp:73] [Rocksdb] Git sha rocksdb_build_git_sha:d82a5c6372e051eadd0db9bfa2d5047730d663a1
I20251204 21:08:29.208012 154 olap_meta.cpp:73] [Rocksdb] Compile date Nov 12 2025
I20251204 21:08:29.208026 154 olap_meta.cpp:73] [Rocksdb] DB SUMMARY
I20251204 21:08:29.208077 154 olap_meta.cpp:73] [Rocksdb] CURRENT file:  CURRENT
I20251204 21:08:29.208091 154 olap_meta.cpp:73] [Rocksdb] IDENTITY file:  IDENTITY
I20251204 21:08:29.208105 154 olap_meta.cpp:73] [Rocksdb] MANIFEST file:  MANIFEST-000043 size: 557 Bytes
I20251204 21:08:29.208117 154 olap_meta.cpp:73] [Rocksdb] SST files in /opt/apache-doris/be/storage/meta dir, Total Num: 3, files: 000034.sst 000040.sst 000044.sst
I20251204 21:08:29.208127 154 olap_meta.cpp:73] [Rocksdb] Write Ahead Log file in /opt/apache-doris/be/storage/meta: 000045.log size: 43379 ;
I20251204 21:08:29.208137 154 olap_meta.cpp:73] [Rocksdb]                         Options.error_if_exists: 0
I20251204 21:08:29.208149 154 olap_meta.cpp:73] [Rocksdb]                       Options.create_if_missing: 1
I20251204 21:08:29.208158 154 olap_meta.cpp:73] [Rocksdb]                         Options.paranoid_checks: 1
I20251204 21:08:29.208169 154 olap_meta.cpp:73] [Rocksdb]                                     Options.env: 0x55e4ee558f60
I20251204 21:08:29.208179 154 olap_meta.cpp:73] [Rocksdb]                                Options.info_log: 0x7f2a578218e0
I20251204 21:08:29.208189 154 olap_meta.cpp:73] [Rocksdb]                Options.max_file_opening_threads: 16
I20251204 21:08:29.208199 154 olap_meta.cpp:73] [Rocksdb]                              Options.statistics: (nil)
I20251204 21:08:29.208210 154 olap_meta.cpp:73] [Rocksdb]                               Options.use_fsync: 0
I20251204 21:08:29.208220 154 olap_meta.cpp:73] [Rocksdb]                       Options.max_log_file_size: 0
I20251204 21:08:29.208231 154 olap_meta.cpp:73] [Rocksdb]                  Options.max_manifest_file_size: 18446744073709551615
I20251204 21:08:29.208240 154 olap_meta.cpp:73] [Rocksdb]                   Options.log_file_time_to_roll: 0
I20251204 21:08:29.208252 154 olap_meta.cpp:73] [Rocksdb]                       Options.keep_log_file_num: 1000
I20251204 21:08:29.208261 154 olap_meta.cpp:73] [Rocksdb]                    Options.recycle_log_file_num: 0
I20251204 21:08:29.208272 154 olap_meta.cpp:73] [Rocksdb]                         Options.allow_fallocate: 1
I20251204 21:08:29.208282 154 olap_meta.cpp:73] [Rocksdb]                        Options.allow_mmap_reads: 0
I20251204 21:08:29.208293 154 olap_meta.cpp:73] [Rocksdb]                       Options.allow_mmap_writes: 0
I20251204 21:08:29.208302 154 olap_meta.cpp:73] [Rocksdb]                        Options.use_direct_reads: 0
I20251204 21:08:29.208312 154 olap_meta.cpp:73] [Rocksdb]                        Options.use_direct_io_for_flush_and_compaction: 0
I20251204 21:08:29.208323 154 olap_meta.cpp:73] [Rocksdb]          Options.create_missing_column_families: 1
I20251204 21:08:29.208333 154 olap_meta.cpp:73] [Rocksdb]                              Options.db_log_dir:
I20251204 21:08:29.208344 154 olap_meta.cpp:73] [Rocksdb]                                 Options.wal_dir: /opt/apache-doris/be/storage/meta
I20251204 21:08:29.208353 154 olap_meta.cpp:73] [Rocksdb]                Options.table_cache_numshardbits: 6
I20251204 21:08:29.208377 154 olap_meta.cpp:73] [Rocksdb]                      Options.max_subcompactions: 1
I20251204 21:08:29.208386 154 olap_meta.cpp:73] [Rocksdb]                  Options.max_background_flushes: -1
I20251204 21:08:29.208397 154 olap_meta.cpp:73] [Rocksdb]                         Options.WAL_ttl_seconds: 0
I20251204 21:08:29.208406 154 olap_meta.cpp:73] [Rocksdb]                       Options.WAL_size_limit_MB: 0
I20251204 21:08:29.208416 154 olap_meta.cpp:73] [Rocksdb]             Options.manifest_preallocation_size: 4194304
I20251204 21:08:29.208426 154 olap_meta.cpp:73] [Rocksdb]                     Options.is_fd_close_on_exec: 1
I20251204 21:08:29.208436 154 olap_meta.cpp:73] [Rocksdb]                   Options.advise_random_on_open: 1
I20251204 21:08:29.208444 154 olap_meta.cpp:73] [Rocksdb]                    Options.db_write_buffer_size: 0
I20251204 21:08:29.208453 154 olap_meta.cpp:73] [Rocksdb]                    Options.write_buffer_manager: 0x7f2a57821900
I20251204 21:08:29.208464 154 olap_meta.cpp:73] [Rocksdb]         Options.access_hint_on_compaction_start: 1
I20251204 21:08:29.208473 154 olap_meta.cpp:73] [Rocksdb]  Options.new_table_reader_for_compaction_inputs: 0
I20251204 21:08:29.208484 154 olap_meta.cpp:73] [Rocksdb]           Options.random_access_max_buffer_size: 1048576
I20251204 21:08:29.208493 154 olap_meta.cpp:73] [Rocksdb]                      Options.use_adaptive_mutex: 0
I20251204 21:08:29.208503 154 olap_meta.cpp:73] [Rocksdb]                            Options.rate_limiter: (nil)
I20251204 21:08:29.208511 154 olap_meta.cpp:73] [Rocksdb]     Options.sst_file_manager.rate_bytes_per_sec: 0
I20251204 21:08:29.208523 154 olap_meta.cpp:73] [Rocksdb]                       Options.wal_recovery_mode: 2
I20251204 21:08:29.208531 154 olap_meta.cpp:73] [Rocksdb]                  Options.enable_thread_tracking: 0
I20251204 21:08:29.208542 154 olap_meta.cpp:73] [Rocksdb]                  Options.enable_pipelined_write: 0
I20251204 21:08:29.208552 154 olap_meta.cpp:73] [Rocksdb]         Options.allow_concurrent_memtable_write: 1
I20251204 21:08:29.208563 154 olap_meta.cpp:73] [Rocksdb]      Options.enable_write_thread_adaptive_yield: 1
I20251204 21:08:29.208572 154 olap_meta.cpp:73] [Rocksdb]             Options.write_thread_max_yield_usec: 100
I20251204 21:08:29.208583 154 olap_meta.cpp:73] [Rocksdb]            Options.write_thread_slow_yield_usec: 3
I20251204 21:08:29.208593 154 olap_meta.cpp:73] [Rocksdb]                               Options.row_cache: None
I20251204 21:08:29.208604 154 olap_meta.cpp:73] [Rocksdb]                              Options.wal_filter: None
I20251204 21:08:29.208614 154 olap_meta.cpp:73] [Rocksdb]             Options.avoid_flush_during_recovery: 0
I20251204 21:08:29.208626 154 olap_meta.cpp:73] [Rocksdb]             Options.allow_ingest_behind: 0
I20251204 21:08:29.208634 154 olap_meta.cpp:73] [Rocksdb]             Options.preserve_deletes: 0
I20251204 21:08:29.208645 154 olap_meta.cpp:73] [Rocksdb]             Options.two_write_queues: 0
I20251204 21:08:29.208654 154 olap_meta.cpp:73] [Rocksdb]             Options.manual_wal_flush: 0
I20251204 21:08:29.208665 154 olap_meta.cpp:73] [Rocksdb]             Options.max_background_jobs: 16
I20251204 21:08:29.208676 154 olap_meta.cpp:73] [Rocksdb]             Options.max_background_compactions: -1
I20251204 21:08:29.208688 154 olap_meta.cpp:73] [Rocksdb]             Options.avoid_flush_during_shutdown: 0
I20251204 21:08:29.208696 154 olap_meta.cpp:73] [Rocksdb]           Options.writable_file_max_buffer_size: 1048576
I20251204 21:08:29.208715 154 olap_meta.cpp:73] [Rocksdb]             Options.delayed_write_rate : 16777216
I20251204 21:08:29.208725 154 olap_meta.cpp:73] [Rocksdb]             Options.max_total_wal_size: 0
I20251204 21:08:29.208736 154 olap_meta.cpp:73] [Rocksdb]             Options.delete_obsolete_files_period_micros: 21600000000
I20251204 21:08:29.208745 154 olap_meta.cpp:73] [Rocksdb]                   Options.stats_dump_period_sec: 600
I20251204 21:08:29.208755 154 olap_meta.cpp:73] [Rocksdb]                          Options.max_open_files: -1
I20251204 21:08:29.208765 154 olap_meta.cpp:73] [Rocksdb]                          Options.bytes_per_sync: 0
I20251204 21:08:29.208774 154 olap_meta.cpp:73] [Rocksdb]                      Options.wal_bytes_per_sync: 0
I20251204 21:08:29.208786 154 olap_meta.cpp:73] [Rocksdb]       Options.compaction_readahead_size: 0
I20251204 21:08:29.208794 154 olap_meta.cpp:73] [Rocksdb] Compression algorithms supported:
I20251204 21:08:29.208811 154 olap_meta.cpp:73] [Rocksdb] 	kZSTD supported: 1
I20251204 21:08:29.208822 154 olap_meta.cpp:73] [Rocksdb] 	kXpressCompression supported: 0
I20251204 21:08:29.208832 154 olap_meta.cpp:73] [Rocksdb] 	kBZip2Compression supported: 1
I20251204 21:08:29.208842 154 olap_meta.cpp:73] [Rocksdb] 	kZSTDNotFinalCompression supported: 1
I20251204 21:08:29.208853 154 olap_meta.cpp:73] [Rocksdb] 	kLZ4Compression supported: 1
I20251204 21:08:29.208861 154 olap_meta.cpp:73] [Rocksdb] 	kZlibCompression supported: 1
I20251204 21:08:29.208873 154 olap_meta.cpp:73] [Rocksdb] 	kLZ4HCCompression supported: 1
I20251204 21:08:29.208881 154 olap_meta.cpp:73] [Rocksdb] 	kSnappyCompression supported: 1
I20251204 21:08:29.208894 154 olap_meta.cpp:73] [Rocksdb] Fast CRC32 supported: Not supported on x86
I20251204 21:08:29.209136 154 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3084] Recovering from manifest file: MANIFEST-000043
I20251204 21:08:29.209228 154 olap_meta.cpp:73] [Rocksdb] [db/column_family.cc:473] --------------- Options for column family [default]:
I20251204 21:08:29.209240 154 olap_meta.cpp:73] [Rocksdb]               Options.comparator: leveldb.BytewiseComparator
I20251204 21:08:29.209249 154 olap_meta.cpp:73] [Rocksdb]           Options.merge_operator: None
I20251204 21:08:29.209260 154 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter: None
I20251204 21:08:29.209270 154 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter_factory: None
I20251204 21:08:29.209278 154 olap_meta.cpp:73] [Rocksdb]         Options.memtable_factory: SkipListFactory
I20251204 21:08:29.209290 154 olap_meta.cpp:73] [Rocksdb]            Options.table_factory: BlockBasedTable
I20251204 21:08:29.209314 154 olap_meta.cpp:73] [Rocksdb]            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f2a5784e008)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7f2a57827f10
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8388608
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
I20251204 21:08:29.209336 154 olap_meta.cpp:73] [Rocksdb]        Options.write_buffer_size: 67108864
I20251204 21:08:29.209347 154 olap_meta.cpp:73] [Rocksdb]  Options.max_write_buffer_number: 2
I20251204 21:08:29.209357 154 olap_meta.cpp:73] [Rocksdb]          Options.compression: Snappy
I20251204 21:08:29.209367 154 olap_meta.cpp:73] [Rocksdb]                  Options.bottommost_compression: Disabled
I20251204 21:08:29.209379 154 olap_meta.cpp:73] [Rocksdb]       Options.prefix_extractor: nullptr
I20251204 21:08:29.209390 154 olap_meta.cpp:73] [Rocksdb]   Options.memtable_insert_with_hint_prefix_extractor: nullptr
I20251204 21:08:29.209399 154 olap_meta.cpp:73] [Rocksdb]             Options.num_levels: 7
I20251204 21:08:29.209410 154 olap_meta.cpp:73] [Rocksdb]        Options.min_write_buffer_number_to_merge: 1
I20251204 21:08:29.209419 154 olap_meta.cpp:73] [Rocksdb]     Options.max_write_buffer_number_to_maintain: 0
I20251204 21:08:29.209430 154 olap_meta.cpp:73] [Rocksdb]            Options.compression_opts.window_bits: -14
I20251204 21:08:29.209440 154 olap_meta.cpp:73] [Rocksdb]                  Options.compression_opts.level: -1
I20251204 21:08:29.209451 154 olap_meta.cpp:73] [Rocksdb]               Options.compression_opts.strategy: 0
I20251204 21:08:29.209460 154 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.max_dict_bytes: 0
I20251204 21:08:29.209471 154 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.zstd_max_train_bytes: 0
I20251204 21:08:29.209481 154 olap_meta.cpp:73] [Rocksdb]      Options.level0_file_num_compaction_trigger: 4
I20251204 21:08:29.209494 154 olap_meta.cpp:73] [Rocksdb]          Options.level0_slowdown_writes_trigger: 20
I20251204 21:08:29.209506 154 olap_meta.cpp:73] [Rocksdb]              Options.level0_stop_writes_trigger: 36
I20251204 21:08:29.209517 154 olap_meta.cpp:73] [Rocksdb]                   Options.target_file_size_base: 67108864
I20251204 21:08:29.209527 154 olap_meta.cpp:73] [Rocksdb]             Options.target_file_size_multiplier: 1
I20251204 21:08:29.209537 154 olap_meta.cpp:73] [Rocksdb]                Options.max_bytes_for_level_base: 268435456
I20251204 21:08:29.209545 154 olap_meta.cpp:73] [Rocksdb] Options.level_compaction_dynamic_level_bytes: 0
I20251204 21:08:29.209558 154 olap_meta.cpp:73] [Rocksdb]          Options.max_bytes_for_level_multiplier: 10.000000
I20251204 21:08:29.209566 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[0]: 1
I20251204 21:08:29.209578 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[1]: 1
I20251204 21:08:29.209586 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[2]: 1
I20251204 21:08:29.209597 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[3]: 1
I20251204 21:08:29.209607 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[4]: 1
I20251204 21:08:29.209618 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[5]: 1
I20251204 21:08:29.209627 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[6]: 1
I20251204 21:08:29.209636 154 olap_meta.cpp:73] [Rocksdb]       Options.max_sequential_skip_in_iterations: 8
I20251204 21:08:29.209647 154 olap_meta.cpp:73] [Rocksdb]                    Options.max_compaction_bytes: 1677721600
I20251204 21:08:29.209656 154 olap_meta.cpp:73] [Rocksdb]                        Options.arena_block_size: 8388608
I20251204 21:08:29.209676 154 olap_meta.cpp:73] [Rocksdb]   Options.soft_pending_compaction_bytes_limit: 68719476736
I20251204 21:08:29.209685 154 olap_meta.cpp:73] [Rocksdb]   Options.hard_pending_compaction_bytes_limit: 274877906944
I20251204 21:08:29.209697 154 olap_meta.cpp:73] [Rocksdb]       Options.rate_limit_delay_max_milliseconds: 100
I20251204 21:08:29.209705 154 olap_meta.cpp:73] [Rocksdb]                Options.disable_auto_compactions: 0
I20251204 21:08:29.209715 154 olap_meta.cpp:73] [Rocksdb]                        Options.compaction_style: kCompactionStyleLevel
I20251204 21:08:29.209725 154 olap_meta.cpp:73] [Rocksdb]                          Options.compaction_pri: kByCompensatedSize
I20251204 21:08:29.209736 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.size_ratio: 1
I20251204 21:08:29.209745 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.min_merge_width: 2
I20251204 21:08:29.209756 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_merge_width: 4294967295
I20251204 21:08:29.209765 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_size_amplification_percent: 200
I20251204 21:08:29.209776 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.compression_size_percent: -1
I20251204 21:08:29.209786 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
I20251204 21:08:29.209797 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.max_table_files_size: 1073741824
I20251204 21:08:29.209806 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.allow_compaction: 0
I20251204 21:08:29.209817 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.ttl: 0
I20251204 21:08:29.209826 154 olap_meta.cpp:73] [Rocksdb]                   Options.table_properties_collectors:
I20251204 21:08:29.209837 154 olap_meta.cpp:73] [Rocksdb]                   Options.inplace_update_support: 0
I20251204 21:08:29.209846 154 olap_meta.cpp:73] [Rocksdb]                 Options.inplace_update_num_locks: 10000
I20251204 21:08:29.209858 154 olap_meta.cpp:73] [Rocksdb]               Options.memtable_prefix_bloom_size_ratio: 0.000000
I20251204 21:08:29.209867 154 olap_meta.cpp:73] [Rocksdb]   Options.memtable_huge_page_size: 0
I20251204 21:08:29.209877 154 olap_meta.cpp:73] [Rocksdb]                           Options.bloom_locality: 0
I20251204 21:08:29.209885 154 olap_meta.cpp:73] [Rocksdb]                    Options.max_successive_merges: 0
I20251204 21:08:29.209894 154 olap_meta.cpp:73] [Rocksdb]                Options.optimize_filters_for_hits: 0
I20251204 21:08:29.209905 154 olap_meta.cpp:73] [Rocksdb]                Options.paranoid_file_checks: 0
I20251204 21:08:29.209914 154 olap_meta.cpp:73] [Rocksdb]                Options.force_consistency_checks: 0
I20251204 21:08:29.209925 154 olap_meta.cpp:73] [Rocksdb]                Options.report_bg_io_stats: 0
I20251204 21:08:29.209935 154 olap_meta.cpp:73] [Rocksdb]                               Options.ttl: 0
I20251204 21:08:29.210018 154 olap_meta.cpp:73] [Rocksdb] [db/column_family.cc:473] --------------- Options for column family [doris]:
I20251204 21:08:29.210031 154 olap_meta.cpp:73] [Rocksdb]               Options.comparator: leveldb.BytewiseComparator
I20251204 21:08:29.210074 154 olap_meta.cpp:73] [Rocksdb]           Options.merge_operator: None
I20251204 21:08:29.210091 154 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter: None
I20251204 21:08:29.210106 154 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter_factory: None
I20251204 21:08:29.210135 154 olap_meta.cpp:73] [Rocksdb]         Options.memtable_factory: SkipListFactory
I20251204 21:08:29.210146 154 olap_meta.cpp:73] [Rocksdb]            Options.table_factory: BlockBasedTable
I20251204 21:08:29.210165 154 olap_meta.cpp:73] [Rocksdb]            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f2a5784e010)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7f2a57827f80
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8388608
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
I20251204 21:08:29.210186 154 olap_meta.cpp:73] [Rocksdb]        Options.write_buffer_size: 67108864
I20251204 21:08:29.210196 154 olap_meta.cpp:73] [Rocksdb]  Options.max_write_buffer_number: 2
I20251204 21:08:29.210207 154 olap_meta.cpp:73] [Rocksdb]          Options.compression: Snappy
I20251204 21:08:29.210216 154 olap_meta.cpp:73] [Rocksdb]                  Options.bottommost_compression: Disabled
I20251204 21:08:29.210227 154 olap_meta.cpp:73] [Rocksdb]       Options.prefix_extractor: nullptr
I20251204 21:08:29.210237 154 olap_meta.cpp:73] [Rocksdb]   Options.memtable_insert_with_hint_prefix_extractor: nullptr
I20251204 21:08:29.210246 154 olap_meta.cpp:73] [Rocksdb]             Options.num_levels: 7
I20251204 21:08:29.210255 154 olap_meta.cpp:73] [Rocksdb]        Options.min_write_buffer_number_to_merge: 1
I20251204 21:08:29.210264 154 olap_meta.cpp:73] [Rocksdb]     Options.max_write_buffer_number_to_maintain: 0
I20251204 21:08:29.210273 154 olap_meta.cpp:73] [Rocksdb]            Options.compression_opts.window_bits: -14
I20251204 21:08:29.210282 154 olap_meta.cpp:73] [Rocksdb]                  Options.compression_opts.level: -1
I20251204 21:08:29.210294 154 olap_meta.cpp:73] [Rocksdb]               Options.compression_opts.strategy: 0
I20251204 21:08:29.210302 154 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.max_dict_bytes: 0
I20251204 21:08:29.210312 154 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.zstd_max_train_bytes: 0
I20251204 21:08:29.210321 154 olap_meta.cpp:73] [Rocksdb]      Options.level0_file_num_compaction_trigger: 4
I20251204 21:08:29.210331 154 olap_meta.cpp:73] [Rocksdb]          Options.level0_slowdown_writes_trigger: 20
I20251204 21:08:29.210340 154 olap_meta.cpp:73] [Rocksdb]              Options.level0_stop_writes_trigger: 36
I20251204 21:08:29.210351 154 olap_meta.cpp:73] [Rocksdb]                   Options.target_file_size_base: 67108864
I20251204 21:08:29.210361 154 olap_meta.cpp:73] [Rocksdb]             Options.target_file_size_multiplier: 1
I20251204 21:08:29.210369 154 olap_meta.cpp:73] [Rocksdb]                Options.max_bytes_for_level_base: 268435456
I20251204 21:08:29.210378 154 olap_meta.cpp:73] [Rocksdb] Options.level_compaction_dynamic_level_bytes: 0
I20251204 21:08:29.210388 154 olap_meta.cpp:73] [Rocksdb]          Options.max_bytes_for_level_multiplier: 10.000000
I20251204 21:08:29.210399 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[0]: 1
I20251204 21:08:29.210417 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[1]: 1
I20251204 21:08:29.210429 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[2]: 1
I20251204 21:08:29.210438 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[3]: 1
I20251204 21:08:29.210449 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[4]: 1
I20251204 21:08:29.210458 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[5]: 1
I20251204 21:08:29.210469 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[6]: 1
I20251204 21:08:29.210479 154 olap_meta.cpp:73] [Rocksdb]       Options.max_sequential_skip_in_iterations: 8
I20251204 21:08:29.210487 154 olap_meta.cpp:73] [Rocksdb]                    Options.max_compaction_bytes: 1677721600
I20251204 21:08:29.210496 154 olap_meta.cpp:73] [Rocksdb]                        Options.arena_block_size: 8388608
I20251204 21:08:29.210505 154 olap_meta.cpp:73] [Rocksdb]   Options.soft_pending_compaction_bytes_limit: 68719476736
I20251204 21:08:29.210515 154 olap_meta.cpp:73] [Rocksdb]   Options.hard_pending_compaction_bytes_limit: 274877906944
I20251204 21:08:29.210525 154 olap_meta.cpp:73] [Rocksdb]       Options.rate_limit_delay_max_milliseconds: 100
I20251204 21:08:29.210536 154 olap_meta.cpp:73] [Rocksdb]                Options.disable_auto_compactions: 0
I20251204 21:08:29.210546 154 olap_meta.cpp:73] [Rocksdb]                        Options.compaction_style: kCompactionStyleLevel
I20251204 21:08:29.210557 154 olap_meta.cpp:73] [Rocksdb]                          Options.compaction_pri: kByCompensatedSize
I20251204 21:08:29.210566 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.size_ratio: 1
I20251204 21:08:29.210575 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.min_merge_width: 2
I20251204 21:08:29.210584 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_merge_width: 4294967295
I20251204 21:08:29.210595 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_size_amplification_percent: 200
I20251204 21:08:29.210604 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.compression_size_percent: -1
I20251204 21:08:29.210616 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
I20251204 21:08:29.210625 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.max_table_files_size: 1073741824
I20251204 21:08:29.210635 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.allow_compaction: 0
I20251204 21:08:29.210644 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.ttl: 0
I20251204 21:08:29.210655 154 olap_meta.cpp:73] [Rocksdb]                   Options.table_properties_collectors:
I20251204 21:08:29.210665 154 olap_meta.cpp:73] [Rocksdb]                   Options.inplace_update_support: 0
I20251204 21:08:29.210675 154 olap_meta.cpp:73] [Rocksdb]                 Options.inplace_update_num_locks: 10000
I20251204 21:08:29.210686 154 olap_meta.cpp:73] [Rocksdb]               Options.memtable_prefix_bloom_size_ratio: 0.000000
I20251204 21:08:29.210695 154 olap_meta.cpp:73] [Rocksdb]   Options.memtable_huge_page_size: 0
I20251204 21:08:29.210706 154 olap_meta.cpp:73] [Rocksdb]                           Options.bloom_locality: 0
I20251204 21:08:29.210716 154 olap_meta.cpp:73] [Rocksdb]                    Options.max_successive_merges: 0
I20251204 21:08:29.210733 154 olap_meta.cpp:73] [Rocksdb]                Options.optimize_filters_for_hits: 0
I20251204 21:08:29.210743 154 olap_meta.cpp:73] [Rocksdb]                Options.paranoid_file_checks: 0
I20251204 21:08:29.210752 154 olap_meta.cpp:73] [Rocksdb]                Options.force_consistency_checks: 0
I20251204 21:08:29.210763 154 olap_meta.cpp:73] [Rocksdb]                Options.report_bg_io_stats: 0
I20251204 21:08:29.210772 154 olap_meta.cpp:73] [Rocksdb]                               Options.ttl: 0
I20251204 21:08:29.210815 154 olap_meta.cpp:73] [Rocksdb] [db/column_family.cc:473] --------------- Options for column family [meta]:
I20251204 21:08:29.210827 154 olap_meta.cpp:73] [Rocksdb]               Options.comparator: leveldb.BytewiseComparator
I20251204 21:08:29.210837 154 olap_meta.cpp:73] [Rocksdb]           Options.merge_operator: None
I20251204 21:08:29.210846 154 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter: None
I20251204 21:08:29.210855 154 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter_factory: None
I20251204 21:08:29.210865 154 olap_meta.cpp:73] [Rocksdb]         Options.memtable_factory: SkipListFactory
I20251204 21:08:29.210873 154 olap_meta.cpp:73] [Rocksdb]            Options.table_factory: BlockBasedTable
I20251204 21:08:29.210896 154 olap_meta.cpp:73] [Rocksdb]            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7f2a5784e018)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7f2a57827ff0
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8388608
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
I20251204 21:08:29.210906 154 olap_meta.cpp:73] [Rocksdb]        Options.write_buffer_size: 67108864
I20251204 21:08:29.210916 154 olap_meta.cpp:73] [Rocksdb]  Options.max_write_buffer_number: 5
I20251204 21:08:29.210927 154 olap_meta.cpp:73] [Rocksdb]          Options.compression: Snappy
I20251204 21:08:29.210937 154 olap_meta.cpp:73] [Rocksdb]                  Options.bottommost_compression: Disabled
I20251204 21:08:29.210947 154 olap_meta.cpp:73] [Rocksdb]       Options.prefix_extractor: rocksdb.FixedPrefix.4
I20251204 21:08:29.210958 154 olap_meta.cpp:73] [Rocksdb]   Options.memtable_insert_with_hint_prefix_extractor: nullptr
I20251204 21:08:29.210968 154 olap_meta.cpp:73] [Rocksdb]             Options.num_levels: 7
I20251204 21:08:29.210978 154 olap_meta.cpp:73] [Rocksdb]        Options.min_write_buffer_number_to_merge: 1
I20251204 21:08:29.210987 154 olap_meta.cpp:73] [Rocksdb]     Options.max_write_buffer_number_to_maintain: 0
I20251204 21:08:29.210996 154 olap_meta.cpp:73] [Rocksdb]            Options.compression_opts.window_bits: -14
I20251204 21:08:29.211005 154 olap_meta.cpp:73] [Rocksdb]                  Options.compression_opts.level: -1
I20251204 21:08:29.211014 154 olap_meta.cpp:73] [Rocksdb]               Options.compression_opts.strategy: 0
I20251204 21:08:29.211023 154 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.max_dict_bytes: 0
I20251204 21:08:29.211042 154 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.zstd_max_train_bytes: 0
I20251204 21:08:29.211051 154 olap_meta.cpp:73] [Rocksdb]      Options.level0_file_num_compaction_trigger: 4
I20251204 21:08:29.211061 154 olap_meta.cpp:73] [Rocksdb]          Options.level0_slowdown_writes_trigger: 20
I20251204 21:08:29.211069 154 olap_meta.cpp:73] [Rocksdb]              Options.level0_stop_writes_trigger: 36
I20251204 21:08:29.211081 154 olap_meta.cpp:73] [Rocksdb]                   Options.target_file_size_base: 67108864
I20251204 21:08:29.211089 154 olap_meta.cpp:73] [Rocksdb]             Options.target_file_size_multiplier: 1
I20251204 21:08:29.211098 154 olap_meta.cpp:73] [Rocksdb]                Options.max_bytes_for_level_base: 268435456
I20251204 21:08:29.211107 154 olap_meta.cpp:73] [Rocksdb] Options.level_compaction_dynamic_level_bytes: 0
I20251204 21:08:29.211118 154 olap_meta.cpp:73] [Rocksdb]          Options.max_bytes_for_level_multiplier: 10.000000
I20251204 21:08:29.211127 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[0]: 1
I20251204 21:08:29.211138 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[1]: 1
I20251204 21:08:29.211148 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[2]: 1
I20251204 21:08:29.211158 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[3]: 1
I20251204 21:08:29.211166 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[4]: 1
I20251204 21:08:29.211177 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[5]: 1
I20251204 21:08:29.211186 154 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[6]: 1
I20251204 21:08:29.211195 154 olap_meta.cpp:73] [Rocksdb]       Options.max_sequential_skip_in_iterations: 8
I20251204 21:08:29.211206 154 olap_meta.cpp:73] [Rocksdb]                    Options.max_compaction_bytes: 1677721600
I20251204 21:08:29.211215 154 olap_meta.cpp:73] [Rocksdb]                        Options.arena_block_size: 8388608
I20251204 21:08:29.211231 154 olap_meta.cpp:73] [Rocksdb]   Options.soft_pending_compaction_bytes_limit: 68719476736
I20251204 21:08:29.211244 154 olap_meta.cpp:73] [Rocksdb]   Options.hard_pending_compaction_bytes_limit: 274877906944
I20251204 21:08:29.211256 154 olap_meta.cpp:73] [Rocksdb]       Options.rate_limit_delay_max_milliseconds: 100
I20251204 21:08:29.211265 154 olap_meta.cpp:73] [Rocksdb]                Options.disable_auto_compactions: 0
I20251204 21:08:29.211274 154 olap_meta.cpp:73] [Rocksdb]                        Options.compaction_style: kCompactionStyleLevel
I20251204 21:08:29.211285 154 olap_meta.cpp:73] [Rocksdb]                          Options.compaction_pri: kByCompensatedSize
I20251204 21:08:29.211294 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.size_ratio: 1
I20251204 21:08:29.211305 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.min_merge_width: 2
I20251204 21:08:29.211314 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_merge_width: 4294967295
I20251204 21:08:29.211325 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_size_amplification_percent: 200
I20251204 21:08:29.211334 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.compression_size_percent: -1
I20251204 21:08:29.211344 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
I20251204 21:08:29.211352 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.max_table_files_size: 1073741824
I20251204 21:08:29.211373 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.allow_compaction: 0
I20251204 21:08:29.211383 154 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.ttl: 0
I20251204 21:08:29.211393 154 olap_meta.cpp:73] [Rocksdb]                   Options.table_properties_collectors:
I20251204 21:08:29.211402 154 olap_meta.cpp:73] [Rocksdb]                   Options.inplace_update_support: 0
I20251204 21:08:29.211412 154 olap_meta.cpp:73] [Rocksdb]                 Options.inplace_update_num_locks: 10000
I20251204 21:08:29.211421 154 olap_meta.cpp:73] [Rocksdb]               Options.memtable_prefix_bloom_size_ratio: 0.000000
I20251204 21:08:29.211431 154 olap_meta.cpp:73] [Rocksdb]   Options.memtable_huge_page_size: 0
I20251204 21:08:29.211443 154 olap_meta.cpp:73] [Rocksdb]                           Options.bloom_locality: 0
I20251204 21:08:29.211452 154 olap_meta.cpp:73] [Rocksdb]                    Options.max_successive_merges: 0
I20251204 21:08:29.211462 154 olap_meta.cpp:73] [Rocksdb]                Options.optimize_filters_for_hits: 0
I20251204 21:08:29.211472 154 olap_meta.cpp:73] [Rocksdb]                Options.paranoid_file_checks: 0
I20251204 21:08:29.211481 154 olap_meta.cpp:73] [Rocksdb]                Options.force_consistency_checks: 0
I20251204 21:08:29.211490 154 olap_meta.cpp:73] [Rocksdb]                Options.report_bg_io_stats: 0
I20251204 21:08:29.211499 154 olap_meta.cpp:73] [Rocksdb]                               Options.ttl: 0
I20251204 21:08:29.215567 154 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3350] Recovered from manifest file:/opt/apache-doris/be/storage/meta/MANIFEST-000043 succeeded,manifest_file_number is 43, next_file_number is 46, last_sequence is 339, log_number is 0,prev_log_number is 0,max_column_family is 2,min_log_number_to_keep is 0
I20251204 21:08:29.215595 154 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3367] Column family [default] (ID 0), log number is 42
I20251204 21:08:29.215606 154 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3367] Column family [doris] (ID 1), log number is 42
I20251204 21:08:29.215615 154 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3367] Column family [meta] (ID 2), log number is 42
I20251204 21:08:29.215706 154 olap_meta.cpp:73] [Rocksdb] EVENT_LOG_v1 {"time_micros": 1764882509215704, "job": 1, "event": "recovery_started", "log_files": [45]}
I20251204 21:08:29.215720 154 olap_meta.cpp:73] [Rocksdb] [db/db_impl_open.cc:549] Recovering log #45 mode 2
I20251204 21:08:29.215922 154 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:2846] Creating manifest 47
I20251204 21:08:29.257565 154 olap_meta.cpp:73] [Rocksdb] EVENT_LOG_v1 {"time_micros": 1764882509257558, "cf_name": "meta", "job": 1, "event": "table_file_creation", "file_number": 48, "file_size": 4955, "table_properties": {"data_size": 4015, "index_size": 28, "filter_size": 0, "raw_key_size": 43, "raw_average_key_size": 43, "raw_value_size": 43293, "raw_average_value_size": 43293, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "", "kDeletedKeys": "0", "kMergeOperands": "0"}}
I20251204 21:08:29.267045 154 olap_meta.cpp:73] [Rocksdb] EVENT_LOG_v1 {"time_micros": 1764882509267043, "job": 1, "event": "recovery_finished"}
I20251204 21:08:29.286283 154 olap_meta.cpp:73] [Rocksdb] [db/db_impl_open.cc:1219] DB pointer 0x7f2a57878000
I20251204 21:08:29.286401 154 storage_engine.cpp:143] stream load record path: /opt/apache-doris/be/storage
I20251204 21:08:29.326341 154 storage_engine.cpp:480] will check 'ulimit' value.
I20251204 21:08:29.326598 154 data_dir.cpp:346] start to load tablets from /opt/apache-doris/be/storage
I20251204 21:08:29.326881 154 data_dir.cpp:339] successfully check incompatible old format meta /opt/apache-doris/be/storage
I20251204 21:08:29.326902 154 data_dir.cpp:357] begin loading rowset from meta
I20251204 21:08:29.327240 154 data_dir.cpp:413] load rowset from meta finished, cost: 0 ms, data dir: /opt/apache-doris/be/storage
I20251204 21:08:29.327257 154 data_dir.cpp:419] begin loading tablet from meta
I20251204 21:08:29.328748 154 data_dir.cpp:468] load tablet from meta finished, loaded tablet: 22, error tablet: 0, cost: 1 ms, path: /opt/apache-doris/be/storage
I20251204 21:08:29.328784 154 data_dir.cpp:501] load pending publish task from meta finished, cost: 0 ms, data dir: /opt/apache-doris/be/storage
I20251204 21:08:29.328804 154 data_dir.cpp:655] load delete bitmap from meta finished, cost: 0 ms, data dir: /opt/apache-doris/be/storage
I20251204 21:08:29.328814 154 data_dir.cpp:661] finish to load tablets from /opt/apache-doris/be/storage, total rowset meta: 6, invalid rowset num: 0, visible/stale rowsets' delete bitmap count: 0, invalid rowsets' delete bitmap count: 0
I20251204 21:08:29.329721 154 storage_engine.cpp:211] success to init storage engine.
I20251204 21:08:29.329799 154 olap_server.cpp:223] unused rowset monitor thread started
I20251204 21:08:29.329860 154 olap_server.cpp:229] evict quering thread started
I20251204 21:08:29.329865 154 storage_engine.cpp:1214] start to delete unused rowset, size: 0, unused delete bitmap size: 0
I20251204 21:08:29.329890 154 storage_engine.cpp:1279] collected 0 unused rowsets to remove, skipped 0 rowsets due to use count > 1, skipped 0 rowsets due to don't need to delete file, skipped 0 rowsets due to delayed expired timestamp. left 0 unused delete bitmap.
I20251204 21:08:29.329910 154 storage_engine.cpp:1304] removed all collected unused rowsets
I20251204 21:08:29.329917 154 olap_server.cpp:235] garbage sweeper thread started
I20251204 21:08:29.329979 154 olap_server.cpp:241] disk stat monitor thread started
I20251204 21:08:29.330024 154 storage_engine.cpp:816] start trash and snapshot sweep. is_clean=0
I20251204 21:08:29.330130 154 storage_engine.cpp:440] get root path info cost: 0 ms. tablet counter: 22
I20251204 21:08:29.330166 154 storage_engine.cpp:840] Start to sweep path /opt/apache-doris/be/storage
I20251204 21:08:29.330478 154 storage_engine.cpp:980] remove 0 invalid rowset meta from dir: /opt/apache-doris/be/storage
I20251204 21:08:29.330503 154 storage_engine.cpp:1014] remove 0 invalid binlog meta from dir: /opt/apache-doris/be/storage
I20251204 21:08:29.330518 154 storage_engine.cpp:1041] removed invalid delete bitmap from dir: /opt/apache-doris/be/storage, deleted tablets size: 0
I20251204 21:08:29.330533 154 storage_engine.cpp:1066] removed invalid pending publish info from dir: /opt/apache-doris/be/storage, deleted pending publish info size: 0
I20251204 21:08:29.330623 154 data_dir.cpp:906] path: /opt/apache-doris/be/storage trash capacity: 0
I20251204 21:08:29.330989 154 olap_server.cpp:280] compaction tasks producer thread started
I20251204 21:08:29.331060 154 olap_server.cpp:285] tablet replicas info update thread started
I20251204 21:08:29.331063 154 olap_server.cpp:651] try to start compaction producer process!
I20251204 21:08:29.331138 154 olap_server.cpp:769] start to update replica infos!
W20251204 21:08:29.331182 154 olap_server.cpp:786] Have not get FE Master heartbeat yet
I20251204 21:08:29.331610 154 olap_server.cpp:303] tablet checkpoint tasks producer thread started
I20251204 21:08:29.331669 154 olap_server.cpp:499] begin to produce tablet meta checkpoint tasks, data_dir=/opt/apache-doris/be/storage
I20251204 21:08:29.331704 154 olap_server.cpp:308] tablet path check thread started
I20251204 21:08:29.331763 154 olap_server.cpp:320] path gc threads started. number:1
I20251204 21:08:29.331816 154 tablet_manager.cpp:1512] finish to do meta checkpoint on dir: /opt/apache-doris/be/storage, number: 0, cost(ms): 0
I20251204 21:08:29.331861 154 olap_server.cpp:466] try to start path gc thread!
I20251204 21:08:29.331880 154 olap_server.cpp:486] try to perform path gc! disk remain [0.999901] internal [86400]
I20251204 21:08:29.331894 154 data_dir.cpp:814] start to gc data dir /opt/apache-doris/be/storage
I20251204 21:08:29.332013 154 olap_server.cpp:327] cooldown thread pool started
I20251204 21:08:29.332067 154 olap_server.cpp:333] cooldown tasks producer thread started
I20251204 21:08:29.332127 154 olap_server.cpp:339] remove unused remote files thread started
I20251204 21:08:29.332156 154 olap_server.cpp:1326] cooldown producer get tablet num: 0
I20251204 21:08:29.332188 154 olap_server.cpp:345] cold data compaction producer thread started
I20251204 21:08:29.333788 154 olap_server.cpp:356] async publish thread started
I20251204 21:08:29.333863 154 olap_server.cpp:362] check tablet delete bitmap score thread started
I20251204 21:08:29.333876 154 olap_server.cpp:364] all storage engine's background threads are started.
I20251204 21:08:29.333935 154 olap_server.cpp:1769] try to start check tablet delete bitmap score!
I20251204 21:08:29.333941 154 workload_sched_policy_mgr.cpp:35] start workload scheduler
I20251204 21:08:29.333976 154 spill_stream_manager.cpp:51] init spill stream manager
I20251204 21:08:29.334069 154 spill_stream_manager.cpp:323] spill storage path: /opt/apache-doris/be/storage, capacity: 1006.85 GB, limit: 181.23 GB, available: 1006.75 GB
I20251204 21:08:29.337246 154 spill_stream_manager.cpp:86] spill gc thread started
I20251204 21:08:29.337481 154 exec_env_init.cpp:201] Using SimdJSON Haswell implementation
I20251204 21:08:29.342844 154 agent_server.cpp:60] Register workload group listener
I20251204 21:08:29.342868 154 agent_server.cpp:66] Register workload scheduler policy listener
I20251204 21:08:29.346339 154 backend_service.cpp:662] Doris BackendService listening on 9060
I20251204 21:08:29.346364 154 backend_service.cpp:666] ingest binlog thread pool size is -1, so we will in sync mode
I20251204 21:08:29.346558 154 thrift_server.cpp:423] ThriftServer 'backend' started on port: 9060
I20251204 21:08:29.349546 154 data_dir.cpp:771] [path gc] delete garbage path: /opt/apache-doris/be/storage/data/14/1764879474977/1277111528/0200000000000048134fccc6c1d84fbc625b413c3bd23298_0.dat
I20251204 21:08:29.349601 154 data_dir.cpp:771] [path gc] delete garbage path: /opt/apache-doris/be/storage/data/14/1764879474977/1277111528/0200000000000055134fccc6c1d84fbc625b413c3bd23298_0.dat
I20251204 21:08:29.349642 154 data_dir.cpp:771] [path gc] delete garbage path: /opt/apache-doris/be/storage/data/14/1764879474977/1277111528/020000000000005b134fccc6c1d84fbc625b413c3bd23298_0.dat
I20251204 21:08:29.349679 154 data_dir.cpp:771] [path gc] delete garbage path: /opt/apache-doris/be/storage/data/14/1764879474977/1277111528/0200000000000054134fccc6c1d84fbc625b413c3bd23298_0.dat
I20251204 21:08:29.351032 154 data_dir.cpp:882] gc data dir path: /opt/apache-doris/be/storage finished
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# 0x00007F2AFC218520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6
 3# raise in /lib/x86_64-linux-gnu/libc.so.6
 4# abort in /lib/x86_64-linux-gnu/libc.so.6
 5# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
 6# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
 7# 0x000055E4EC5EC121 in /opt/apache-doris/be/lib/doris_be
 8# 0x000055E4EC5EC274 in /opt/apache-doris/be/lib/doris_be
 9# std::__throw_system_error(int) at ../../../../../libstdc++-v3/src/c++11/system_error.cc:338
10# 0x000055E4EC6BB1ED in /opt/apache-doris/be/lib/doris_be
11# std::thread* doris::ThreadGroup::create_thread<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)> >(std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread_group.h:65
12# doris::WorkThreadPool<false>::WorkThreadPool(unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /home/zcp/repo_center/doris_release/doris/be/src/util/work_thread_pool.hpp:61
13# doris::PInternalService::PInternalService(doris::ExecEnv*) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:218
14# doris::PInternalServiceImpl::PInternalServiceImpl(doris::StorageEngine&, doris::ExecEnv*) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:259
15# doris::BRpcService::start(int, int) in /opt/apache-doris/be/lib/doris_be
16# main at /home/zcp/repo_center/doris_release/doris/be/src/service/doris_main.cpp:570
17# 0x00007F2AFC1FFD90 in /lib/x86_64-linux-gnu/libc.so.6
18# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
19# _start in /opt/apache-doris/be/lib/doris_be
/opt/apache-doris/be/bin/start_be.sh: line 450:   154 Aborted                 (core dumped) ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" 2>> "${LOG_DIR}/be.out" < /dev/null
[Thu Dec  4 21:08:49 UTC 2025] run post_exit

What You Expected?

I had hoped that all 3 of the BE's would start

How to Reproduce?

I have deployed the 1.6.1 helm operator and the fe/be of 3.1.3 its reproducible each time i redeploy.

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[plastikman]

Here is my be-configmap.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: be-configmap
  namespace: doris
data:
  be.conf: |
    # Enable file cache
    # enable_file_cache = true

    # JVM options for JDK 17 with module access permissions
    JAVA_OPTS_FOR_JDK_17 = "-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -Xlog:gc*:$LOG_DIR/be.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED"

These were taken from the example, crash happens without anything set or with these set.

Here is my doris-cluster.yaml

---
apiVersion: doris.selectdb.com/v1
kind: DorisCluster
metadata:
  name: doris-cluster
  namespace: doris
spec:
  feSpec:
    replicas: 3
    image: apache/doris:fe-3.1.3
    configMapInfo:
      configMapName: fe-configmap
      resolveKey: fe.conf
    startTimeout: 600
    limits:
      cpu: 6
      memory: 12Gi
    requests:
      cpu: 6
      memory: 12Gi
    nodeSelector:
      groupname: core
    tolerations:
    - key: "kubernetes.azure.com/scalesetpriority"
      operator: Equal
      value: spot
      effect: NoSchedule
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: node/type
              operator: In
              values:
              - spot
              - ondemand
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 99
          preference:
            matchExpressions:
            - key: node/type
              operator: In
              values:
              - spot
        - weight: 1
          preference:
            matchExpressions:
            - key: node/type
              operator: In
              values:
              - ondemand
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/component: fe
          topologyKey: kubernetes.io/hostname
    topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app.kubernetes.io/component: fe
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app.kubernetes.io/component: fe
    persistentVolumes:
    - mountPath: /opt/apache-doris/fe/doris-meta
      name: meta
      persistentVolumeClaimSpec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 200Gi
        storageClassName: premium-disk
  beSpec:
    replicas: 3
    image: apache/doris:be-3.1.3
    configMapInfo:
      configMapName: be-configmap
      resolveKey: be.conf
    limits:
      cpu: 8
      memory: 16Gi
    requests:
      cpu: 8
      memory: 16Gi
    nodeSelector:
      groupname: core
    tolerations:
    - key: "kubernetes.azure.com/scalesetpriority"
      operator: Equal
      value: spot
      effect: NoSchedule
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: node/type
              operator: In
              values:
              - spot
              - ondemand
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 99
          preference:
            matchExpressions:
            - key: node/type
              operator: In
              values:
              - spot
        - weight: 1
          preference:
            matchExpressions:
            - key: node/type
              operator: In
              values:
              - ondemand
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/component: be
          topologyKey: kubernetes.io/hostname
    topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: ScheduleAnyway
      labelSelector:
        matchLabels:
          app.kubernetes.io/component: be
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app.kubernetes.io/component: be
    persistentVolumes:
    - logNotNeed: true
      mountPath: /opt/apache-doris/be/storage
      name: be-storage
      persistentVolumeClaimSpec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Ti
        storageClassName: premium-disk

Here is my kustomize

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: doris

resources:
  - resources/namespace.yaml
  - resources/fe-configmap.yaml
  - resources/be-configmap.yaml
  - resources/doris-cluster.yaml
  - resources/service-fe.yaml
  - resources/service-be.yaml

patches:
- target:
    kind: Service
    name: doris-fe-service
  patch: |-
    - op: replace
      path: /metadata/annotations
      value:
          service.beta.kubernetes.io/azure-load-balancer-internal: "true"
          service.beta.kubernetes.io/azure-load-balancer-tcp-idle-timeout: "1800"
          service.beta.kubernetes.io/azure-load-balancer-tcp-reset: "true"
          service.beta.kubernetes.io/load-balancer-source-ranges: 10.0.0.0/8

service-be.yaml

---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: doris
    cluster: doris-cluster
    component: be
  name: doris-be-service
  namespace: doris
spec:
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: webserver
      port: 8040
      protocol: TCP
      targetPort: 8040
    - name: heartbeat
      port: 9050
      protocol: TCP
      targetPort: 9050
    - name: be-rpc
      port: 9060
      protocol: TCP
      targetPort: 9060
    - name: brpc
      port: 8060
      protocol: TCP
      targetPort: 8060
  selector:
    app.kubernetes.io/component: be
  sessionAffinity: None
  type: ClusterIP

Here is the config for my doris-operator

values.yaml

nameOverride: ""
fullnameOverride: ""

nodeSelector:
  groupname: core

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: "kubernetes.azure.com/scalesetpriority"
          operator: In
          values:
          - "spot"

tolerations:
  - key: "kubernetes.azure.com/scalesetpriority"
    operator: Equal
    value: spot
    effect: NoSchedule

# Security context for the operator
securityContext:
  runAsNonRoot: true
  runAsUser: 65532
  fsGroup: 65532

# Operator container configuration
operator:
  resources:
    limits:
      cpu: 500m
      memory: 1Gi
    requests:
      cpu: 100m
      memory: 256Mi

  image:
    pullPolicy: IfNotPresent


  # Log level: debug, info, warn, error
  logLevel: info

# Install CRDs with Helm
installCRDs: true

serviceAccount:
  create: true
  name: ""

kustomize

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: doris-operator


helmCharts:
- name: doris-operator
  releaseName: doris-operator
  repo: https://charts.selectdb.com
  version: 1.6.1
  namespace: doris-operator
  includeCRDs: true
  valuesMerge: override
  valuesFile: resources/values.yaml

[plastikman]

I have extracted the core dump that i can share as well. I think its probably too large to share in github though.

[yiguolei]

@plastikman It seems it crashed when create thread. Please Check if the file handle limit is large enough.

[plastikman]

root@doris-cluster-be-0:/etc/doris# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 515197
max locked memory           (kbytes, -l) 64
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) unlimited
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited
root@doris-cluster-be-0:/etc/doris# 

[plastikman]

Note that the first pod that starts wins, so if i kill all 3 pods the first one to come up will start, so i can kill be-0, be-1, then be-2 and be-0 will start with no issues, i can then kill be-2, then be-1, then be-0 and then be-2 will start without any issue.

So this is always with the second and third node that comes up.

[yiguolei]

Could please attach the content in be.out file and also the be.conf file content?

[plastikman]

hello @yiguolei the be.out is the log in the first post, and the be.conf is in the second post.

This is running in AKS (Azure Kubernetes Service) running on Kubernetes 1.32

I am running on DAS32_v6 nodes as well

Here is the latest be.out

[Fri Dec  5 14:52:52 UTC 2025] [info] the host machine support avx2 instruction set.
[Fri Dec  5 14:52:52 UTC 2025] [info] Process conf file be.conf ...
[Fri Dec  5 14:52:52 UTC 2025] [info] use root no password show backends result 1764887436965	doris-cluster-be-2.doris-cluster-be-internal.doris.svc.cluster.local	9050	9060	8040	8060	-1	2025-12-05 02:03:48	2025-12-05 08:41:44	false	false	0	0.000 	0.000 	1006.748 GB	1006.854 GB	0.01 %	0.01 %	0.000 	{"location" : "default"}	java.net.ConnectException: Connection refused	doris-3.1.3-rc01-be1b1227a8	{"lastSuccessReportTabletsTime":"2025-12-05 08:41:03","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false,"isActive":true,"isShutdown":true,"currentFragmentNum":0,"lastFragmentUpdateTime":1764924005645}	2209		32	16.00 GB	0
1764887437038	doris-cluster-be-0.doris-cluster-be-internal.doris.svc.cluster.local	9050	-1	-1	-1	-1	NULL	NULL	false	false	0	0.000 	0.000 	1.000 B	0.000 	0.00 %	0.00 %	0.000 	{"location" : "default"}	java.net.ConnectException: Connection refused		{"lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false,"isActive":true,"isShutdown":false,"currentFragmentNum":0,"lastFragmentUpdateTime":0}	5672		1	0.00 	0
1764887437057	doris-cluster-be-1.doris-cluster-be-internal.doris.svc.cluster.local	9050	9060	8040	8060	-1	2025-12-05 14:11:56	2025-12-05 14:52:50	true	false	24	129.800 KB	0.000 	1006.745 GB	1006.854 GB	0.01 %	0.01 %	0.000 	{"location" : "default"}		doris-3.1.3-rc01-be1b1227a8	{"lastSuccessReportTabletsTime":"2025-12-05 14:51:52","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false,"isActive":true,"isShutdown":false,"currentFragmentNum":0,"lastFragmentUpdateTime":1764946355832}	0	mix	32	16.00 GB	0 .
[Fri Dec  5 14:52:52 UTC 2025] [info] Check myself (doris-cluster-be-0.doris-cluster-be-internal.doris.svc.cluster.local:9050) exist in FE, start be directly ...
the annotations value:
the value not equal!  debug apache.com.doris/runmode
[Fri Dec  5 14:52:52 UTC 2025] run start_be.sh
StdoutLogger 2025-12-05 14:52:52,420 Start time: Fri Dec  5 14:52:52 UTC 2025
I20251205 14:52:52.533481 389 doris_main.cpp:388]  version doris-3.1.3-rc01(AVX2) RELEASE features: -TDE,-HDFS_STORAGE_VAULT,+UI,+AZURE_BLOB,+AZURE_STORAGE_VAULT,-HIVE_UDF,+BE_JAVA_EXTENSIONS (build git://vm-8@be1b1227a89253ba301e5f3c7b41c82a7adf46f6)
Built on Wed, 19 Nov 2025 22:51:16 4Z by vm-8
I20251205 14:52:52.558771 389 jni-util.cpp:107] set final LIBHDFS_OPTS: -XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -Xlog:gc*:/opt/apache-doris/be/log/be.gc.log.:time,uptime:filecount=10,filesize=50M -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED -Djava.security.krb5.conf=/etc/krb5.conf  -Djdk.lang.processReaperUseDefaultStackSize=true
I20251205 14:52:53.582561 389 doris_main.cpp:496] Doris backend JNI is initialized.
I20251205 14:52:53.583421 389 mem_info.cpp:384] Physical Memory: 135065071616, BE Available Physical Memory(consider cgroup): 17179869184, Mem Limit: 14.40 GB, origin config value: 90%, System Mem Available Min Reserve: 819.20 MB, Vm Min Free KBytes: 66.00 MB, Vm Overcommit Memory: 1
I20251205 14:52:53.583456 389 doris_main.cpp:514] Cpu Info:
  Model: AMD EPYC 9V74 80-Core Processor
  Cores: 32
  Max Possible Cores: 32
  L1 Cache: 32.00 KB (Line: 64.00 B)
  L2 Cache: 1.00 MB (Line: 64.00 B)
  L3 Cache: 320.00 MB (Line: 64.00 B)
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
    avx
    avx2
  Numa Nodes: 1
  Numa Nodes of Cores: 0->0 | 1->0 | 2->0 | 3->0 | 4->0 | 5->0 | 6->0 | 7->0 | 8->0 | 9->0 | 10->0 | 11->0 | 12->0 | 13->0 | 14->0 | 15->0 | 16->0 | 17->0 | 18->0 | 19->0 | 20->0 | 21->0 | 22->0 | 23->0 | 24->0 | 25->0 | 26->0 | 27->0 | 28->0 | 29->0 | 30->0 | 31->0 |
I20251205 14:52:53.583484 389 doris_main.cpp:515] Disk Info:
  Num disks 3: sr, nvme0n, nvme0n1p
I20251205 14:52:53.583498 389 doris_main.cpp:516] Physical Memory: 17179869184
Memory Limt: 15461882265
CGroup Info: Process CGroup Memory Info (cgroups path: /sys/fs/cgroup, cgroup version: v2): memory limit: 17179869184, memory usage: 467863256
I20251205 14:52:53.584076 389 backend_options.cpp:62] local host ip=172.24.31.190
W20251205 14:52:53.591830 389 timezone_utils.cpp:98] Meet illegal tzdata file: leap-seconds.list. skipped
W20251205 14:52:53.594645 389 timezone_utils.cpp:98] Meet illegal tzdata file: zone1970.tab. skipped
W20251205 14:52:53.595238 389 timezone_utils.cpp:98] Meet illegal tzdata file: leapseconds. skipped
W20251205 14:52:53.610193 389 timezone_utils.cpp:98] Meet illegal tzdata file: zone.tab. skipped
W20251205 14:52:53.639490 389 timezone_utils.cpp:98] Meet illegal tzdata file: tzdata.zi. skipped
W20251205 14:52:53.641067 389 timezone_utils.cpp:98] Meet illegal tzdata file: iso3166.tab. skipped
W20251205 14:52:53.643607 389 timezone_utils.cpp:98] Meet illegal tzdata file: zonenow.tab. skipped
I20251205 14:52:53.647033 389 timezone_utils.cpp:115] Preloaded658 timezones.
E20251205 14:52:53.655022 389 variable.cpp:179] Already exposed `doris_cache_data_page_cache' whose value is `0'
E20251205 14:52:53.655057 389 variable.cpp:179] Already exposed `doris_cache_data_page_cache_persecond' whose value is `0'
I20251205 14:52:53.655084 389 cache_manager.h:47] Register Cache DataPageCache
E20251205 14:52:53.655189 389 variable.cpp:179] Already exposed `doris_cache_index_page_cache' whose value is `0'
E20251205 14:52:53.655206 389 variable.cpp:179] Already exposed `doris_cache_index_page_cache_persecond' whose value is `0'
I20251205 14:52:53.655220 389 cache_manager.h:47] Register Cache IndexPageCache
E20251205 14:52:53.655313 389 variable.cpp:179] Already exposed `doris_cache_pkindex_page_cache' whose value is `0'
E20251205 14:52:53.655330 389 variable.cpp:179] Already exposed `doris_cache_pkindex_page_cache_persecond' whose value is `0'
I20251205 14:52:53.655346 389 cache_manager.h:47] Register Cache PKIndexPageCache
I20251205 14:52:53.655361 389 exec_env_init.cpp:524] Storage page cache memory limit: 2.88 GB, origin config value: 20%
E20251205 14:52:53.655434 389 variable.cpp:179] Already exposed `doris_cache_point_query_row_cache' whose value is `0'
E20251205 14:52:53.655450 389 variable.cpp:179] Already exposed `doris_cache_point_query_row_cache_persecond' whose value is `0'
I20251205 14:52:53.655468 389 cache_manager.h:47] Register Cache PointQueryRowCache
I20251205 14:52:53.655483 389 exec_env_init.cpp:537] Row cache memory limit: 2.88 GB, origin config value: 20%
E20251205 14:52:53.655537 389 variable.cpp:179] Already exposed `doris_cache_segment_cache' whose value is `0'
E20251205 14:52:53.655567 389 variable.cpp:179] Already exposed `doris_cache_segment_cache_persecond' whose value is `0'
I20251205 14:52:53.655582 389 cache_manager.h:47] Register Cache SegmentCache
I20251205 14:52:53.655597 389 exec_env_init.cpp:571] segment_cache_capacity <= fd_number * 1 / 5, fd_number: 1048576 segment_cache_capacity: 209700 min_segment_cache_mem_limit 773094110
E20251205 14:52:53.655630 389 variable.cpp:179] Already exposed `doris_cache_schema_cache' whose value is `0'
E20251205 14:52:53.655646 389 variable.cpp:179] Already exposed `doris_cache_schema_cache_persecond' whose value is `0'
I20251205 14:52:53.655663 389 cache_manager.h:47] Register Cache SchemaCache
I20251205 14:52:53.655675 389 exec_env_init.cpp:579] max file reader cache size is: 349525, resource hard limit is: 1048576, config file_cache_max_file_reader_cache_size is: 1000000
E20251205 14:52:53.655710 389 variable.cpp:179] Already exposed `doris_cache_common_obj_lrucache' whose value is `0'
E20251205 14:52:53.655726 389 variable.cpp:179] Already exposed `doris_cache_common_obj_lrucache_persecond' whose value is `0'
I20251205 14:52:53.655741 389 cache_manager.h:47] Register Cache CommonObjLRUCache
E20251205 14:52:53.655771 389 variable.cpp:179] Already exposed `doris_cache_point_query_lookup_connection_cache' whose value is `0'
E20251205 14:52:53.655787 389 variable.cpp:179] Already exposed `doris_cache_point_query_lookup_connection_cache_persecond' whose value is `0'
I20251205 14:52:53.655802 389 cache_manager.h:47] Register Cache PointQueryLookupConnectionCache
I20251205 14:52:53.655817 389 inverted_index_cache.cpp:59] fd_number: 1048576, inverted index open searcher limit: 209715
E20251205 14:52:53.655923 389 variable.cpp:179] Already exposed `doris_cache_inverted_index_searcher_cache' whose value is `0'
E20251205 14:52:53.655942 389 variable.cpp:179] Already exposed `doris_cache_inverted_index_searcher_cache_persecond' whose value is `0'
I20251205 14:52:53.655956 389 cache_manager.h:47] Register Cache InvertedIndexSearcherCache
I20251205 14:52:53.655972 389 exec_env_init.cpp:600] Inverted index searcher cache memory limit: 1.44 GB, origin config value: 10%
E20251205 14:52:53.656086 389 variable.cpp:179] Already exposed `doris_cache_inverted_index_query_cache' whose value is `0'
E20251205 14:52:53.656111 389 variable.cpp:179] Already exposed `doris_cache_inverted_index_query_cache_persecond' whose value is `0'
I20251205 14:52:53.656126 389 cache_manager.h:47] Register Cache InvertedIndexQueryCache
I20251205 14:52:53.656147 389 exec_env_init.cpp:614] Inverted index query match cache memory limit: 1.44 GB, origin config value: 10%
E20251205 14:52:53.656188 389 variable.cpp:179] Already exposed `doris_cache_query_cache' whose value is `0'
E20251205 14:52:53.656208 389 variable.cpp:179] Already exposed `doris_cache_query_cache_persecond' whose value is `0'
I20251205 14:52:53.656221 389 cache_manager.h:47] Register Cache QueryCache
I20251205 14:52:53.656235 389 exec_env_init.cpp:623] query cache memory limit: 512MB
E20251205 14:52:53.656363 389 variable.cpp:179] Already exposed `doris_cache_mow_delete_bitmap_agg_cache' whose value is `0'
E20251205 14:52:53.656384 389 variable.cpp:179] Already exposed `doris_cache_mow_delete_bitmap_agg_cache_persecond' whose value is `0'
I20251205 14:52:53.656399 389 cache_manager.h:47] Register Cache MowDeleteBitmapAggCache
I20251205 14:52:53.656414 389 cgroup_cpu_ctl.cpp:68] [cgroup_init_path]doris cgroup home path is not specify, if you not use workload group, you can ignore this log.
I20251205 14:52:53.656435 389 exec_env_init.cpp:410] pipeline executors_size set |size=32
I20251205 14:52:53.659003 389 task_scheduler.cpp:61] TaskScheduler set cores|size=32
I20251205 14:52:53.659441 389 fragment_mgr.cpp:906] FragmentMgr cancel worker start working.
I20251205 14:52:53.803375 389 load_path_mgr.cpp:67] Load path configured to []
I20251205 14:52:53.803422 389 result_buffer_mgr.cpp:192] result buffer manager cancel thread begin.
E20251205 14:52:53.810350 389 variable.cpp:179] Already exposed `doris_cache_last_success_channel_cache' whose value is `0'
E20251205 14:52:53.810382 389 variable.cpp:179] Already exposed `doris_cache_last_success_channel_cache_persecond' whose value is `0'
I20251205 14:52:53.810400 389 cache_manager.h:47] Register Cache LastSuccessChannelCache
I20251205 14:52:53.810472 389 wal_manager.cpp:116] wal_dir:/opt/apache-doris/be/storage/wal, tmp_dir:/opt/apache-doris/be/storage/wal/tmp
E20251205 14:52:53.810674 389 variable.cpp:179] Already exposed `doris_cache_tablet_schema_cache' whose value is `0'
E20251205 14:52:53.810699 389 variable.cpp:179] Already exposed `doris_cache_tablet_schema_cache_persecond' whose value is `0'
I20251205 14:52:53.810717 389 cache_manager.h:47] Register Cache TabletSchemaCache
E20251205 14:52:53.810747 389 variable.cpp:179] Already exposed `doris_cache_tablet_column_object_pool' whose value is `0'
E20251205 14:52:53.810763 389 variable.cpp:179] Already exposed `doris_cache_tablet_column_object_pool_persecond' whose value is `0'
I20251205 14:52:53.810777 389 cache_manager.h:47] Register Cache TabletColumnObjectPool
I20251205 14:52:53.810817 389 exec_env_init.cpp:695] The current deployment mode is local.
start BE in local mode
E20251205 14:52:53.811003 389 variable.cpp:179] Already exposed `doris_cache_mow_tablet_version_cache' whose value is `0'
E20251205 14:52:53.811022 389 variable.cpp:179] Already exposed `doris_cache_mow_tablet_version_cache_persecond' whose value is `0'
I20251205 14:52:53.811036 389 cache_manager.h:47] Register Cache MowTabletVersionCache
E20251205 14:52:53.811064 389 variable.cpp:179] Already exposed `doris_cache_create_tablet_rridx_cache' whose value is `0'
E20251205 14:52:53.811079 389 variable.cpp:179] Already exposed `doris_cache_create_tablet_rridx_cache_persecond' whose value is `0'
I20251205 14:52:53.811093 389 cache_manager.h:47] Register Cache CreateTabletRRIdxCache
I20251205 14:52:53.811111 389 storage_engine.cpp:209] starting backend using uid:3246ae2be90b96fe-39b5ecf2aa3fd282
I20251205 14:52:53.811216 389 data_dir.cpp:890] path: /opt/apache-doris/be/storage total capacity: 1081101176832, available capacity: 1080977223680, usage: 0.000114655, in_use: 0
I20251205 14:52:53.815014 389 data_dir.cpp:201] path: /opt/apache-doris/be/storage, hash: 5192008450149472686
I20251205 14:52:53.816128 389 olap_meta.cpp:73] [Rocksdb] RocksDB version: 5.14.2
I20251205 14:52:53.816152 389 olap_meta.cpp:73] [Rocksdb] Git sha rocksdb_build_git_sha:d82a5c6372e051eadd0db9bfa2d5047730d663a1
I20251205 14:52:53.816172 389 olap_meta.cpp:73] [Rocksdb] Compile date Nov 12 2025
I20251205 14:52:53.816185 389 olap_meta.cpp:73] [Rocksdb] DB SUMMARY
I20251205 14:52:53.816236 389 olap_meta.cpp:73] [Rocksdb] CURRENT file:  CURRENT
I20251205 14:52:53.816251 389 olap_meta.cpp:73] [Rocksdb] IDENTITY file:  IDENTITY
I20251205 14:52:53.816268 389 olap_meta.cpp:73] [Rocksdb] MANIFEST file:  MANIFEST-000347 size: 222 Bytes
I20251205 14:52:53.816283 389 olap_meta.cpp:73] [Rocksdb] SST files in /opt/apache-doris/be/storage/meta dir, Total Num: 0, files:
I20251205 14:52:53.816298 389 olap_meta.cpp:73] [Rocksdb] Write Ahead Log file in /opt/apache-doris/be/storage/meta: 000348.log size: 19 ;
I20251205 14:52:53.816311 389 olap_meta.cpp:73] [Rocksdb]                         Options.error_if_exists: 0
I20251205 14:52:53.816326 389 olap_meta.cpp:73] [Rocksdb]                       Options.create_if_missing: 1
I20251205 14:52:53.816341 389 olap_meta.cpp:73] [Rocksdb]                         Options.paranoid_checks: 1
I20251205 14:52:53.816354 389 olap_meta.cpp:73] [Rocksdb]                                     Options.env: 0x557694ec2f60
I20251205 14:52:53.816368 389 olap_meta.cpp:73] [Rocksdb]                                Options.info_log: 0x7fe76e868f40
I20251205 14:52:53.816382 389 olap_meta.cpp:73] [Rocksdb]                Options.max_file_opening_threads: 16
I20251205 14:52:53.816396 389 olap_meta.cpp:73] [Rocksdb]                              Options.statistics: (nil)
I20251205 14:52:53.816409 389 olap_meta.cpp:73] [Rocksdb]                               Options.use_fsync: 0
I20251205 14:52:53.816426 389 olap_meta.cpp:73] [Rocksdb]                       Options.max_log_file_size: 0
I20251205 14:52:53.816437 389 olap_meta.cpp:73] [Rocksdb]                  Options.max_manifest_file_size: 18446744073709551615
I20251205 14:52:53.816452 389 olap_meta.cpp:73] [Rocksdb]                   Options.log_file_time_to_roll: 0
I20251205 14:52:53.816465 389 olap_meta.cpp:73] [Rocksdb]                       Options.keep_log_file_num: 1000
I20251205 14:52:53.816480 389 olap_meta.cpp:73] [Rocksdb]                    Options.recycle_log_file_num: 0
I20251205 14:52:53.816494 389 olap_meta.cpp:73] [Rocksdb]                         Options.allow_fallocate: 1
I20251205 14:52:53.816509 389 olap_meta.cpp:73] [Rocksdb]                        Options.allow_mmap_reads: 0
I20251205 14:52:53.816522 389 olap_meta.cpp:73] [Rocksdb]                       Options.allow_mmap_writes: 0
I20251205 14:52:53.816536 389 olap_meta.cpp:73] [Rocksdb]                        Options.use_direct_reads: 0
I20251205 14:52:53.816550 389 olap_meta.cpp:73] [Rocksdb]                        Options.use_direct_io_for_flush_and_compaction: 0
I20251205 14:52:53.816565 389 olap_meta.cpp:73] [Rocksdb]          Options.create_missing_column_families: 1
I20251205 14:52:53.816576 389 olap_meta.cpp:73] [Rocksdb]                              Options.db_log_dir:
I20251205 14:52:53.816591 389 olap_meta.cpp:73] [Rocksdb]                                 Options.wal_dir: /opt/apache-doris/be/storage/meta
I20251205 14:52:53.816604 389 olap_meta.cpp:73] [Rocksdb]                Options.table_cache_numshardbits: 6
I20251205 14:52:53.816618 389 olap_meta.cpp:73] [Rocksdb]                      Options.max_subcompactions: 1
I20251205 14:52:53.816630 389 olap_meta.cpp:73] [Rocksdb]                  Options.max_background_flushes: -1
I20251205 14:52:53.816645 389 olap_meta.cpp:73] [Rocksdb]                         Options.WAL_ttl_seconds: 0
I20251205 14:52:53.816658 389 olap_meta.cpp:73] [Rocksdb]                       Options.WAL_size_limit_MB: 0
I20251205 14:52:53.816671 389 olap_meta.cpp:73] [Rocksdb]             Options.manifest_preallocation_size: 4194304
I20251205 14:52:53.816684 389 olap_meta.cpp:73] [Rocksdb]                     Options.is_fd_close_on_exec: 1
I20251205 14:52:53.816699 389 olap_meta.cpp:73] [Rocksdb]                   Options.advise_random_on_open: 1
I20251205 14:52:53.816712 389 olap_meta.cpp:73] [Rocksdb]                    Options.db_write_buffer_size: 0
I20251205 14:52:53.816730 389 olap_meta.cpp:73] [Rocksdb]                    Options.write_buffer_manager: 0x7fe76e868fc0
I20251205 14:52:53.816743 389 olap_meta.cpp:73] [Rocksdb]         Options.access_hint_on_compaction_start: 1
I20251205 14:52:53.816757 389 olap_meta.cpp:73] [Rocksdb]  Options.new_table_reader_for_compaction_inputs: 0
I20251205 14:52:53.816771 389 olap_meta.cpp:73] [Rocksdb]           Options.random_access_max_buffer_size: 1048576
I20251205 14:52:53.816784 389 olap_meta.cpp:73] [Rocksdb]                      Options.use_adaptive_mutex: 0
I20251205 14:52:53.816798 389 olap_meta.cpp:73] [Rocksdb]                            Options.rate_limiter: (nil)
I20251205 14:52:53.816812 389 olap_meta.cpp:73] [Rocksdb]     Options.sst_file_manager.rate_bytes_per_sec: 0
I20251205 14:52:53.816826 389 olap_meta.cpp:73] [Rocksdb]                       Options.wal_recovery_mode: 2
I20251205 14:52:53.816840 389 olap_meta.cpp:73] [Rocksdb]                  Options.enable_thread_tracking: 0
I20251205 14:52:53.816855 389 olap_meta.cpp:73] [Rocksdb]                  Options.enable_pipelined_write: 0
I20251205 14:52:53.816869 389 olap_meta.cpp:73] [Rocksdb]         Options.allow_concurrent_memtable_write: 1
I20251205 14:52:53.816884 389 olap_meta.cpp:73] [Rocksdb]      Options.enable_write_thread_adaptive_yield: 1
I20251205 14:52:53.816897 389 olap_meta.cpp:73] [Rocksdb]             Options.write_thread_max_yield_usec: 100
I20251205 14:52:53.816911 389 olap_meta.cpp:73] [Rocksdb]            Options.write_thread_slow_yield_usec: 3
I20251205 14:52:53.816924 389 olap_meta.cpp:73] [Rocksdb]                               Options.row_cache: None
I20251205 14:52:53.816938 389 olap_meta.cpp:73] [Rocksdb]                              Options.wal_filter: None
I20251205 14:52:53.816951 389 olap_meta.cpp:73] [Rocksdb]             Options.avoid_flush_during_recovery: 0
I20251205 14:52:53.816965 389 olap_meta.cpp:73] [Rocksdb]             Options.allow_ingest_behind: 0
I20251205 14:52:53.816978 389 olap_meta.cpp:73] [Rocksdb]             Options.preserve_deletes: 0
I20251205 14:52:53.816992 389 olap_meta.cpp:73] [Rocksdb]             Options.two_write_queues: 0
I20251205 14:52:53.817005 389 olap_meta.cpp:73] [Rocksdb]             Options.manual_wal_flush: 0
I20251205 14:52:53.817019 389 olap_meta.cpp:73] [Rocksdb]             Options.max_background_jobs: 16
I20251205 14:52:53.817032 389 olap_meta.cpp:73] [Rocksdb]             Options.max_background_compactions: -1
I20251205 14:52:53.817046 389 olap_meta.cpp:73] [Rocksdb]             Options.avoid_flush_during_shutdown: 0
I20251205 14:52:53.817059 389 olap_meta.cpp:73] [Rocksdb]           Options.writable_file_max_buffer_size: 1048576
I20251205 14:52:53.817073 389 olap_meta.cpp:73] [Rocksdb]             Options.delayed_write_rate : 16777216
I20251205 14:52:53.817086 389 olap_meta.cpp:73] [Rocksdb]             Options.max_total_wal_size: 0
I20251205 14:52:53.817099 389 olap_meta.cpp:73] [Rocksdb]             Options.delete_obsolete_files_period_micros: 21600000000
I20251205 14:52:53.817113 389 olap_meta.cpp:73] [Rocksdb]                   Options.stats_dump_period_sec: 600
I20251205 14:52:53.817126 389 olap_meta.cpp:73] [Rocksdb]                          Options.max_open_files: -1
I20251205 14:52:53.817140 389 olap_meta.cpp:73] [Rocksdb]                          Options.bytes_per_sync: 0
I20251205 14:52:53.817154 389 olap_meta.cpp:73] [Rocksdb]                      Options.wal_bytes_per_sync: 0
I20251205 14:52:53.817169 389 olap_meta.cpp:73] [Rocksdb]       Options.compaction_readahead_size: 0
I20251205 14:52:53.817183 389 olap_meta.cpp:73] [Rocksdb] Compression algorithms supported:
I20251205 14:52:53.817198 389 olap_meta.cpp:73] [Rocksdb] 	kZSTD supported: 1
I20251205 14:52:53.817210 389 olap_meta.cpp:73] [Rocksdb] 	kXpressCompression supported: 0
I20251205 14:52:53.817225 389 olap_meta.cpp:73] [Rocksdb] 	kBZip2Compression supported: 1
I20251205 14:52:53.817237 389 olap_meta.cpp:73] [Rocksdb] 	kZSTDNotFinalCompression supported: 1
I20251205 14:52:53.817257 389 olap_meta.cpp:73] [Rocksdb] 	kLZ4Compression supported: 1
I20251205 14:52:53.817269 389 olap_meta.cpp:73] [Rocksdb] 	kZlibCompression supported: 1
I20251205 14:52:53.817284 389 olap_meta.cpp:73] [Rocksdb] 	kLZ4HCCompression supported: 1
I20251205 14:52:53.817297 389 olap_meta.cpp:73] [Rocksdb] 	kSnappyCompression supported: 1
I20251205 14:52:53.817312 389 olap_meta.cpp:73] [Rocksdb] Fast CRC32 supported: Not supported on x86
I20251205 14:52:53.817545 389 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3084] Recovering from manifest file: MANIFEST-000347
I20251205 14:52:53.817636 389 olap_meta.cpp:73] [Rocksdb] [db/column_family.cc:473] --------------- Options for column family [default]:
I20251205 14:52:53.817652 389 olap_meta.cpp:73] [Rocksdb]               Options.comparator: leveldb.BytewiseComparator
I20251205 14:52:53.817664 389 olap_meta.cpp:73] [Rocksdb]           Options.merge_operator: None
I20251205 14:52:53.817678 389 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter: None
I20251205 14:52:53.817691 389 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter_factory: None
I20251205 14:52:53.817705 389 olap_meta.cpp:73] [Rocksdb]         Options.memtable_factory: SkipListFactory
I20251205 14:52:53.817719 389 olap_meta.cpp:73] [Rocksdb]            Options.table_factory: BlockBasedTable
I20251205 14:52:53.817749 389 olap_meta.cpp:73] [Rocksdb]            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fe76e88c008)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7fe76e8639d0
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8388608
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
I20251205 14:52:53.817765 389 olap_meta.cpp:73] [Rocksdb]        Options.write_buffer_size: 67108864
I20251205 14:52:53.817777 389 olap_meta.cpp:73] [Rocksdb]  Options.max_write_buffer_number: 2
I20251205 14:52:53.817792 389 olap_meta.cpp:73] [Rocksdb]          Options.compression: Snappy
I20251205 14:52:53.817807 389 olap_meta.cpp:73] [Rocksdb]                  Options.bottommost_compression: Disabled
I20251205 14:52:53.817823 389 olap_meta.cpp:73] [Rocksdb]       Options.prefix_extractor: nullptr
I20251205 14:52:53.817836 389 olap_meta.cpp:73] [Rocksdb]   Options.memtable_insert_with_hint_prefix_extractor: nullptr
I20251205 14:52:53.817850 389 olap_meta.cpp:73] [Rocksdb]             Options.num_levels: 7
I20251205 14:52:53.817864 389 olap_meta.cpp:73] [Rocksdb]        Options.min_write_buffer_number_to_merge: 1
I20251205 14:52:53.817878 389 olap_meta.cpp:73] [Rocksdb]     Options.max_write_buffer_number_to_maintain: 0
I20251205 14:52:53.817891 389 olap_meta.cpp:73] [Rocksdb]            Options.compression_opts.window_bits: -14
I20251205 14:52:53.817904 389 olap_meta.cpp:73] [Rocksdb]                  Options.compression_opts.level: -1
I20251205 14:52:53.817919 389 olap_meta.cpp:73] [Rocksdb]               Options.compression_opts.strategy: 0
I20251205 14:52:53.817931 389 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.max_dict_bytes: 0
I20251205 14:52:53.817945 389 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.zstd_max_train_bytes: 0
I20251205 14:52:53.817957 389 olap_meta.cpp:73] [Rocksdb]      Options.level0_file_num_compaction_trigger: 4
I20251205 14:52:53.817971 389 olap_meta.cpp:73] [Rocksdb]          Options.level0_slowdown_writes_trigger: 20
I20251205 14:52:53.817991 389 olap_meta.cpp:73] [Rocksdb]              Options.level0_stop_writes_trigger: 36
I20251205 14:52:53.818004 389 olap_meta.cpp:73] [Rocksdb]                   Options.target_file_size_base: 67108864
I20251205 14:52:53.818017 389 olap_meta.cpp:73] [Rocksdb]             Options.target_file_size_multiplier: 1
I20251205 14:52:53.818032 389 olap_meta.cpp:73] [Rocksdb]                Options.max_bytes_for_level_base: 268435456
I20251205 14:52:53.818043 389 olap_meta.cpp:73] [Rocksdb] Options.level_compaction_dynamic_level_bytes: 0
I20251205 14:52:53.818059 389 olap_meta.cpp:73] [Rocksdb]          Options.max_bytes_for_level_multiplier: 10.000000
I20251205 14:52:53.818073 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[0]: 1
I20251205 14:52:53.818085 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[1]: 1
I20251205 14:52:53.818100 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[2]: 1
I20251205 14:52:53.818112 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[3]: 1
I20251205 14:52:53.818126 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[4]: 1
I20251205 14:52:53.818140 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[5]: 1
I20251205 14:52:53.818153 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[6]: 1
I20251205 14:52:53.818166 389 olap_meta.cpp:73] [Rocksdb]       Options.max_sequential_skip_in_iterations: 8
I20251205 14:52:53.818181 389 olap_meta.cpp:73] [Rocksdb]                    Options.max_compaction_bytes: 1677721600
I20251205 14:52:53.818192 389 olap_meta.cpp:73] [Rocksdb]                        Options.arena_block_size: 8388608
I20251205 14:52:53.818207 389 olap_meta.cpp:73] [Rocksdb]   Options.soft_pending_compaction_bytes_limit: 68719476736
I20251205 14:52:53.818219 389 olap_meta.cpp:73] [Rocksdb]   Options.hard_pending_compaction_bytes_limit: 274877906944
I20251205 14:52:53.818233 389 olap_meta.cpp:73] [Rocksdb]       Options.rate_limit_delay_max_milliseconds: 100
I20251205 14:52:53.818248 389 olap_meta.cpp:73] [Rocksdb]                Options.disable_auto_compactions: 0
I20251205 14:52:53.818262 389 olap_meta.cpp:73] [Rocksdb]                        Options.compaction_style: kCompactionStyleLevel
I20251205 14:52:53.818277 389 olap_meta.cpp:73] [Rocksdb]                          Options.compaction_pri: kByCompensatedSize
I20251205 14:52:53.818291 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.size_ratio: 1
I20251205 14:52:53.818303 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.min_merge_width: 2
I20251205 14:52:53.818318 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_merge_width: 4294967295
I20251205 14:52:53.818332 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_size_amplification_percent: 200
I20251205 14:52:53.818347 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.compression_size_percent: -1
I20251205 14:52:53.818359 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
I20251205 14:52:53.818373 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.max_table_files_size: 1073741824
I20251205 14:52:53.818387 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.allow_compaction: 0
I20251205 14:52:53.818400 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.ttl: 0
I20251205 14:52:53.818414 389 olap_meta.cpp:73] [Rocksdb]                   Options.table_properties_collectors:
I20251205 14:52:53.818428 389 olap_meta.cpp:73] [Rocksdb]                   Options.inplace_update_support: 0
I20251205 14:52:53.818440 389 olap_meta.cpp:73] [Rocksdb]                 Options.inplace_update_num_locks: 10000
I20251205 14:52:53.818455 389 olap_meta.cpp:73] [Rocksdb]               Options.memtable_prefix_bloom_size_ratio: 0.000000
I20251205 14:52:53.818473 389 olap_meta.cpp:73] [Rocksdb]   Options.memtable_huge_page_size: 0
I20251205 14:52:53.818485 389 olap_meta.cpp:73] [Rocksdb]                           Options.bloom_locality: 0
I20251205 14:52:53.818500 389 olap_meta.cpp:73] [Rocksdb]                    Options.max_successive_merges: 0
I20251205 14:52:53.818514 389 olap_meta.cpp:73] [Rocksdb]                Options.optimize_filters_for_hits: 0
I20251205 14:52:53.818526 389 olap_meta.cpp:73] [Rocksdb]                Options.paranoid_file_checks: 0
I20251205 14:52:53.818542 389 olap_meta.cpp:73] [Rocksdb]                Options.force_consistency_checks: 0
I20251205 14:52:53.818553 389 olap_meta.cpp:73] [Rocksdb]                Options.report_bg_io_stats: 0
I20251205 14:52:53.818568 389 olap_meta.cpp:73] [Rocksdb]                               Options.ttl: 0
I20251205 14:52:53.818645 389 olap_meta.cpp:73] [Rocksdb] [db/column_family.cc:473] --------------- Options for column family [doris]:
I20251205 14:52:53.818662 389 olap_meta.cpp:73] [Rocksdb]               Options.comparator: leveldb.BytewiseComparator
I20251205 14:52:53.818674 389 olap_meta.cpp:73] [Rocksdb]           Options.merge_operator: None
I20251205 14:52:53.818689 389 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter: None
I20251205 14:52:53.818703 389 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter_factory: None
I20251205 14:52:53.818717 389 olap_meta.cpp:73] [Rocksdb]         Options.memtable_factory: SkipListFactory
I20251205 14:52:53.818732 389 olap_meta.cpp:73] [Rocksdb]            Options.table_factory: BlockBasedTable
I20251205 14:52:53.818750 389 olap_meta.cpp:73] [Rocksdb]            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fe76e88c010)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7fe76e863a40
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8388608
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
I20251205 14:52:53.818765 389 olap_meta.cpp:73] [Rocksdb]        Options.write_buffer_size: 67108864
I20251205 14:52:53.818780 389 olap_meta.cpp:73] [Rocksdb]  Options.max_write_buffer_number: 2
I20251205 14:52:53.818795 389 olap_meta.cpp:73] [Rocksdb]          Options.compression: Snappy
I20251205 14:52:53.818810 389 olap_meta.cpp:73] [Rocksdb]                  Options.bottommost_compression: Disabled
I20251205 14:52:53.818825 389 olap_meta.cpp:73] [Rocksdb]       Options.prefix_extractor: nullptr
I20251205 14:52:53.818838 389 olap_meta.cpp:73] [Rocksdb]   Options.memtable_insert_with_hint_prefix_extractor: nullptr
I20251205 14:52:53.818850 389 olap_meta.cpp:73] [Rocksdb]             Options.num_levels: 7
I20251205 14:52:53.818866 389 olap_meta.cpp:73] [Rocksdb]        Options.min_write_buffer_number_to_merge: 1
I20251205 14:52:53.818877 389 olap_meta.cpp:73] [Rocksdb]     Options.max_write_buffer_number_to_maintain: 0
I20251205 14:52:53.818892 389 olap_meta.cpp:73] [Rocksdb]            Options.compression_opts.window_bits: -14
I20251205 14:52:53.818903 389 olap_meta.cpp:73] [Rocksdb]                  Options.compression_opts.level: -1
I20251205 14:52:53.818917 389 olap_meta.cpp:73] [Rocksdb]               Options.compression_opts.strategy: 0
I20251205 14:52:53.818929 389 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.max_dict_bytes: 0
I20251205 14:52:53.818948 389 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.zstd_max_train_bytes: 0
I20251205 14:52:53.818960 389 olap_meta.cpp:73] [Rocksdb]      Options.level0_file_num_compaction_trigger: 4
I20251205 14:52:53.818974 389 olap_meta.cpp:73] [Rocksdb]          Options.level0_slowdown_writes_trigger: 20
I20251205 14:52:53.818987 389 olap_meta.cpp:73] [Rocksdb]              Options.level0_stop_writes_trigger: 36
I20251205 14:52:53.819001 389 olap_meta.cpp:73] [Rocksdb]                   Options.target_file_size_base: 67108864
I20251205 14:52:53.819013 389 olap_meta.cpp:73] [Rocksdb]             Options.target_file_size_multiplier: 1
I20251205 14:52:53.819027 389 olap_meta.cpp:73] [Rocksdb]                Options.max_bytes_for_level_base: 268435456
I20251205 14:52:53.819041 389 olap_meta.cpp:73] [Rocksdb] Options.level_compaction_dynamic_level_bytes: 0
I20251205 14:52:53.819054 389 olap_meta.cpp:73] [Rocksdb]          Options.max_bytes_for_level_multiplier: 10.000000
I20251205 14:52:53.819067 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[0]: 1
I20251205 14:52:53.819082 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[1]: 1
I20251205 14:52:53.819093 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[2]: 1
I20251205 14:52:53.819108 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[3]: 1
I20251205 14:52:53.819120 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[4]: 1
I20251205 14:52:53.819134 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[5]: 1
I20251205 14:52:53.819146 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[6]: 1
I20251205 14:52:53.819161 389 olap_meta.cpp:73] [Rocksdb]       Options.max_sequential_skip_in_iterations: 8
I20251205 14:52:53.819175 389 olap_meta.cpp:73] [Rocksdb]                    Options.max_compaction_bytes: 1677721600
I20251205 14:52:53.819190 389 olap_meta.cpp:73] [Rocksdb]                        Options.arena_block_size: 8388608
I20251205 14:52:53.819201 389 olap_meta.cpp:73] [Rocksdb]   Options.soft_pending_compaction_bytes_limit: 68719476736
I20251205 14:52:53.819216 389 olap_meta.cpp:73] [Rocksdb]   Options.hard_pending_compaction_bytes_limit: 274877906944
I20251205 14:52:53.819231 389 olap_meta.cpp:73] [Rocksdb]       Options.rate_limit_delay_max_milliseconds: 100
I20251205 14:52:53.819244 389 olap_meta.cpp:73] [Rocksdb]                Options.disable_auto_compactions: 0
I20251205 14:52:53.819257 389 olap_meta.cpp:73] [Rocksdb]                        Options.compaction_style: kCompactionStyleLevel
I20251205 14:52:53.819271 389 olap_meta.cpp:73] [Rocksdb]                          Options.compaction_pri: kByCompensatedSize
I20251205 14:52:53.819283 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.size_ratio: 1
I20251205 14:52:53.819298 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.min_merge_width: 2
I20251205 14:52:53.819310 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_merge_width: 4294967295
I20251205 14:52:53.819324 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_size_amplification_percent: 200
I20251205 14:52:53.819337 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.compression_size_percent: -1
I20251205 14:52:53.819351 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
I20251205 14:52:53.819363 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.max_table_files_size: 1073741824
I20251205 14:52:53.819377 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.allow_compaction: 0
I20251205 14:52:53.819391 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.ttl: 0
I20251205 14:52:53.819406 389 olap_meta.cpp:73] [Rocksdb]                   Options.table_properties_collectors:
I20251205 14:52:53.819424 389 olap_meta.cpp:73] [Rocksdb]                   Options.inplace_update_support: 0
I20251205 14:52:53.819437 389 olap_meta.cpp:73] [Rocksdb]                 Options.inplace_update_num_locks: 10000
I20251205 14:52:53.819451 389 olap_meta.cpp:73] [Rocksdb]               Options.memtable_prefix_bloom_size_ratio: 0.000000
I20251205 14:52:53.819465 389 olap_meta.cpp:73] [Rocksdb]   Options.memtable_huge_page_size: 0
I20251205 14:52:53.819479 389 olap_meta.cpp:73] [Rocksdb]                           Options.bloom_locality: 0
I20251205 14:52:53.819491 389 olap_meta.cpp:73] [Rocksdb]                    Options.max_successive_merges: 0
I20251205 14:52:53.819506 389 olap_meta.cpp:73] [Rocksdb]                Options.optimize_filters_for_hits: 0
I20251205 14:52:53.819520 389 olap_meta.cpp:73] [Rocksdb]                Options.paranoid_file_checks: 0
I20251205 14:52:53.819532 389 olap_meta.cpp:73] [Rocksdb]                Options.force_consistency_checks: 0
I20251205 14:52:53.819546 389 olap_meta.cpp:73] [Rocksdb]                Options.report_bg_io_stats: 0
I20251205 14:52:53.819576 389 olap_meta.cpp:73] [Rocksdb]                               Options.ttl: 0
I20251205 14:52:53.819618 389 olap_meta.cpp:73] [Rocksdb] [db/column_family.cc:473] --------------- Options for column family [meta]:
I20251205 14:52:53.819634 389 olap_meta.cpp:73] [Rocksdb]               Options.comparator: leveldb.BytewiseComparator
I20251205 14:52:53.819648 389 olap_meta.cpp:73] [Rocksdb]           Options.merge_operator: None
I20251205 14:52:53.819661 389 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter: None
I20251205 14:52:53.819674 389 olap_meta.cpp:73] [Rocksdb]        Options.compaction_filter_factory: None
I20251205 14:52:53.819689 389 olap_meta.cpp:73] [Rocksdb]         Options.memtable_factory: SkipListFactory
I20251205 14:52:53.819702 389 olap_meta.cpp:73] [Rocksdb]            Options.table_factory: BlockBasedTable
I20251205 14:52:53.819725 389 olap_meta.cpp:73] [Rocksdb]            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x7fe76e88c018)
  cache_index_and_filter_blocks: 0
  cache_index_and_filter_blocks_with_high_priority: 0
  pin_l0_filter_and_index_blocks_in_cache: 0
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x7fe76e863ab0
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 8388608
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: nullptr
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0
I20251205 14:52:53.819739 389 olap_meta.cpp:73] [Rocksdb]        Options.write_buffer_size: 67108864
I20251205 14:52:53.819752 389 olap_meta.cpp:73] [Rocksdb]  Options.max_write_buffer_number: 5
I20251205 14:52:53.819766 389 olap_meta.cpp:73] [Rocksdb]          Options.compression: Snappy
I20251205 14:52:53.819778 389 olap_meta.cpp:73] [Rocksdb]                  Options.bottommost_compression: Disabled
I20251205 14:52:53.819797 389 olap_meta.cpp:73] [Rocksdb]       Options.prefix_extractor: rocksdb.FixedPrefix.4
I20251205 14:52:53.819808 389 olap_meta.cpp:73] [Rocksdb]   Options.memtable_insert_with_hint_prefix_extractor: nullptr
I20251205 14:52:53.819823 389 olap_meta.cpp:73] [Rocksdb]             Options.num_levels: 7
I20251205 14:52:53.819835 389 olap_meta.cpp:73] [Rocksdb]        Options.min_write_buffer_number_to_merge: 1
I20251205 14:52:53.819849 389 olap_meta.cpp:73] [Rocksdb]     Options.max_write_buffer_number_to_maintain: 0
I20251205 14:52:53.819862 389 olap_meta.cpp:73] [Rocksdb]            Options.compression_opts.window_bits: -14
I20251205 14:52:53.819876 389 olap_meta.cpp:73] [Rocksdb]                  Options.compression_opts.level: -1
I20251205 14:52:53.819895 389 olap_meta.cpp:73] [Rocksdb]               Options.compression_opts.strategy: 0
I20251205 14:52:53.819907 389 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.max_dict_bytes: 0
I20251205 14:52:53.819921 389 olap_meta.cpp:73] [Rocksdb]         Options.compression_opts.zstd_max_train_bytes: 0
I20251205 14:52:53.819934 389 olap_meta.cpp:73] [Rocksdb]      Options.level0_file_num_compaction_trigger: 4
I20251205 14:52:53.819948 389 olap_meta.cpp:73] [Rocksdb]          Options.level0_slowdown_writes_trigger: 20
I20251205 14:52:53.819960 389 olap_meta.cpp:73] [Rocksdb]              Options.level0_stop_writes_trigger: 36
I20251205 14:52:53.819974 389 olap_meta.cpp:73] [Rocksdb]                   Options.target_file_size_base: 67108864
I20251205 14:52:53.819988 389 olap_meta.cpp:73] [Rocksdb]             Options.target_file_size_multiplier: 1
I20251205 14:52:53.820003 389 olap_meta.cpp:73] [Rocksdb]                Options.max_bytes_for_level_base: 268435456
I20251205 14:52:53.820017 389 olap_meta.cpp:73] [Rocksdb] Options.level_compaction_dynamic_level_bytes: 0
I20251205 14:52:53.820032 389 olap_meta.cpp:73] [Rocksdb]          Options.max_bytes_for_level_multiplier: 10.000000
I20251205 14:52:53.820046 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[0]: 1
I20251205 14:52:53.820058 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[1]: 1
I20251205 14:52:53.820073 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[2]: 1
I20251205 14:52:53.820087 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[3]: 1
I20251205 14:52:53.820101 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[4]: 1
I20251205 14:52:53.820115 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[5]: 1
I20251205 14:52:53.820127 389 olap_meta.cpp:73] [Rocksdb] Options.max_bytes_for_level_multiplier_addtl[6]: 1
I20251205 14:52:53.820142 389 olap_meta.cpp:73] [Rocksdb]       Options.max_sequential_skip_in_iterations: 8
I20251205 14:52:53.820155 389 olap_meta.cpp:73] [Rocksdb]                    Options.max_compaction_bytes: 1677721600
I20251205 14:52:53.820169 389 olap_meta.cpp:73] [Rocksdb]                        Options.arena_block_size: 8388608
I20251205 14:52:53.820181 389 olap_meta.cpp:73] [Rocksdb]   Options.soft_pending_compaction_bytes_limit: 68719476736
I20251205 14:52:53.820196 389 olap_meta.cpp:73] [Rocksdb]   Options.hard_pending_compaction_bytes_limit: 274877906944
I20251205 14:52:53.820210 389 olap_meta.cpp:73] [Rocksdb]       Options.rate_limit_delay_max_milliseconds: 100
I20251205 14:52:53.820222 389 olap_meta.cpp:73] [Rocksdb]                Options.disable_auto_compactions: 0
I20251205 14:52:53.820237 389 olap_meta.cpp:73] [Rocksdb]                        Options.compaction_style: kCompactionStyleLevel
I20251205 14:52:53.820250 389 olap_meta.cpp:73] [Rocksdb]                          Options.compaction_pri: kByCompensatedSize
I20251205 14:52:53.820264 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.size_ratio: 1
I20251205 14:52:53.820276 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.min_merge_width: 2
I20251205 14:52:53.820290 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_merge_width: 4294967295
I20251205 14:52:53.820303 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.max_size_amplification_percent: 200
I20251205 14:52:53.820317 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.compression_size_percent: -1
I20251205 14:52:53.820329 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
I20251205 14:52:53.820344 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.max_table_files_size: 1073741824
I20251205 14:52:53.820356 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.allow_compaction: 0
I20251205 14:52:53.820374 389 olap_meta.cpp:73] [Rocksdb] Options.compaction_options_fifo.ttl: 0
I20251205 14:52:53.820386 389 olap_meta.cpp:73] [Rocksdb]                   Options.table_properties_collectors:
I20251205 14:52:53.820401 389 olap_meta.cpp:73] [Rocksdb]                   Options.inplace_update_support: 0
I20251205 14:52:53.820413 389 olap_meta.cpp:73] [Rocksdb]                 Options.inplace_update_num_locks: 10000
I20251205 14:52:53.820428 389 olap_meta.cpp:73] [Rocksdb]               Options.memtable_prefix_bloom_size_ratio: 0.000000
I20251205 14:52:53.820442 389 olap_meta.cpp:73] [Rocksdb]   Options.memtable_huge_page_size: 0
I20251205 14:52:53.820457 389 olap_meta.cpp:73] [Rocksdb]                           Options.bloom_locality: 0
I20251205 14:52:53.820470 389 olap_meta.cpp:73] [Rocksdb]                    Options.max_successive_merges: 0
I20251205 14:52:53.820484 389 olap_meta.cpp:73] [Rocksdb]                Options.optimize_filters_for_hits: 0
I20251205 14:52:53.820497 389 olap_meta.cpp:73] [Rocksdb]                Options.paranoid_file_checks: 0
I20251205 14:52:53.820511 389 olap_meta.cpp:73] [Rocksdb]                Options.force_consistency_checks: 0
I20251205 14:52:53.820523 389 olap_meta.cpp:73] [Rocksdb]                Options.report_bg_io_stats: 0
I20251205 14:52:53.820537 389 olap_meta.cpp:73] [Rocksdb]                               Options.ttl: 0
I20251205 14:52:53.822404 389 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3350] Recovered from manifest file:/opt/apache-doris/be/storage/meta/MANIFEST-000347 succeeded,manifest_file_number is 347, next_file_number is 349, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 2,min_log_number_to_keep is 0
I20251205 14:52:53.822432 389 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3367] Column family [default] (ID 0), log number is 346
I20251205 14:52:53.822448 389 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3367] Column family [doris] (ID 1), log number is 346
I20251205 14:52:53.822463 389 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:3367] Column family [meta] (ID 2), log number is 346
I20251205 14:52:53.822539 389 olap_meta.cpp:73] [Rocksdb] EVENT_LOG_v1 {"time_micros": 1764946373822538, "job": 1, "event": "recovery_started", "log_files": [348]}
I20251205 14:52:53.822557 389 olap_meta.cpp:73] [Rocksdb] [db/db_impl_open.cc:549] Recovering log #348 mode 2
I20251205 14:52:53.822649 389 olap_meta.cpp:73] [Rocksdb] [db/version_set.cc:2846] Creating manifest 350
I20251205 14:52:53.872951 389 olap_meta.cpp:73] [Rocksdb] EVENT_LOG_v1 {"time_micros": 1764946373872945, "job": 1, "event": "recovery_finished"}
I20251205 14:52:53.894551 389 olap_meta.cpp:73] [Rocksdb] [db/db_impl_open.cc:1219] DB pointer 0x7fe76e8b6000
I20251205 14:52:53.894716 389 storage_engine.cpp:143] stream load record path: /opt/apache-doris/be/storage
I20251205 14:52:53.945096 389 storage_engine.cpp:480] will check 'ulimit' value.
I20251205 14:52:53.945344 389 data_dir.cpp:346] start to load tablets from /opt/apache-doris/be/storage
I20251205 14:52:53.945457 389 data_dir.cpp:339] successfully check incompatible old format meta /opt/apache-doris/be/storage
I20251205 14:52:53.945477 389 data_dir.cpp:357] begin loading rowset from meta
I20251205 14:52:53.945490 389 data_dir.cpp:413] load rowset from meta finished, cost: 0 ms, data dir: /opt/apache-doris/be/storage
I20251205 14:52:53.945504 389 data_dir.cpp:419] begin loading tablet from meta
I20251205 14:52:53.945523 389 data_dir.cpp:468] load tablet from meta finished, loaded tablet: 0, error tablet: 0, cost: 0 ms, path: /opt/apache-doris/be/storage
I20251205 14:52:53.945539 389 data_dir.cpp:501] load pending publish task from meta finished, cost: 0 ms, data dir: /opt/apache-doris/be/storage
I20251205 14:52:53.945564 389 data_dir.cpp:655] load delete bitmap from meta finished, cost: 0 ms, data dir: /opt/apache-doris/be/storage
I20251205 14:52:53.945580 389 data_dir.cpp:661] finish to load tablets from /opt/apache-doris/be/storage, total rowset meta: 0, invalid rowset num: 0, visible/stale rowsets' delete bitmap count: 0, invalid rowsets' delete bitmap count: 0
I20251205 14:52:53.946482 389 storage_engine.cpp:211] success to init storage engine.
I20251205 14:52:53.946575 389 olap_server.cpp:223] unused rowset monitor thread started
I20251205 14:52:53.946650 389 olap_server.cpp:229] evict quering thread started
I20251205 14:52:53.946652 389 storage_engine.cpp:1214] start to delete unused rowset, size: 0, unused delete bitmap size: 0
I20251205 14:52:53.946702 389 storage_engine.cpp:1279] collected 0 unused rowsets to remove, skipped 0 rowsets due to use count > 1, skipped 0 rowsets due to don't need to delete file, skipped 0 rowsets due to delayed expired timestamp. left 0 unused delete bitmap.
I20251205 14:52:53.946722 389 storage_engine.cpp:1304] removed all collected unused rowsets
I20251205 14:52:53.946731 389 olap_server.cpp:235] garbage sweeper thread started
I20251205 14:52:53.946784 389 olap_server.cpp:241] disk stat monitor thread started
I20251205 14:52:53.946923 389 storage_engine.cpp:816] start trash and snapshot sweep. is_clean=0
I20251205 14:52:53.947008 389 storage_engine.cpp:440] get root path info cost: 0 ms. tablet counter: 0
I20251205 14:52:53.947041 389 storage_engine.cpp:840] Start to sweep path /opt/apache-doris/be/storage
I20251205 14:52:53.947171 389 storage_engine.cpp:980] remove 0 invalid rowset meta from dir: /opt/apache-doris/be/storage
I20251205 14:52:53.947201 389 storage_engine.cpp:1014] remove 0 invalid binlog meta from dir: /opt/apache-doris/be/storage
I20251205 14:52:53.947227 389 storage_engine.cpp:1041] removed invalid delete bitmap from dir: /opt/apache-doris/be/storage, deleted tablets size: 0
I20251205 14:52:53.947245 389 storage_engine.cpp:1066] removed invalid pending publish info from dir: /opt/apache-doris/be/storage, deleted pending publish info size: 0
I20251205 14:52:53.947371 389 data_dir.cpp:906] path: /opt/apache-doris/be/storage trash capacity: 0
I20251205 14:52:53.947853 389 olap_server.cpp:280] compaction tasks producer thread started
I20251205 14:52:53.947938 389 olap_server.cpp:651] try to start compaction producer process!
I20251205 14:52:53.947952 389 olap_server.cpp:285] tablet replicas info update thread started
I20251205 14:52:53.948063 389 olap_server.cpp:769] start to update replica infos!
W20251205 14:52:53.948112 389 olap_server.cpp:786] Have not get FE Master heartbeat yet
I20251205 14:52:53.948526 389 olap_server.cpp:303] tablet checkpoint tasks producer thread started
I20251205 14:52:53.948586 389 olap_server.cpp:499] begin to produce tablet meta checkpoint tasks, data_dir=/opt/apache-doris/be/storage
I20251205 14:52:53.948598 389 olap_server.cpp:308] tablet path check thread started
I20251205 14:52:53.948777 389 olap_server.cpp:320] path gc threads started. number:1
I20251205 14:52:53.948823 389 olap_server.cpp:466] try to start path gc thread!
I20251205 14:52:53.948853 389 olap_server.cpp:486] try to perform path gc! disk remain [0.999885] internal [86400]
I20251205 14:52:53.948868 389 tablet_manager.cpp:1512] finish to do meta checkpoint on dir: /opt/apache-doris/be/storage, number: 0, cost(ms): 0
I20251205 14:52:53.948873 389 data_dir.cpp:814] start to gc data dir /opt/apache-doris/be/storage
I20251205 14:52:53.949024 389 olap_server.cpp:327] cooldown thread pool started
I20251205 14:52:53.949080 389 olap_server.cpp:333] cooldown tasks producer thread started
I20251205 14:52:53.949143 389 olap_server.cpp:339] remove unused remote files thread started
I20251205 14:52:53.949162 389 olap_server.cpp:1326] cooldown producer get tablet num: 0
I20251205 14:52:53.949208 389 olap_server.cpp:345] cold data compaction producer thread started
I20251205 14:52:53.950814 389 olap_server.cpp:356] async publish thread started
I20251205 14:52:53.950896 389 olap_server.cpp:362] check tablet delete bitmap score thread started
I20251205 14:52:53.950912 389 olap_server.cpp:364] all storage engine's background threads are started.
I20251205 14:52:53.950977 389 olap_server.cpp:1769] try to start check tablet delete bitmap score!
I20251205 14:52:53.950978 389 workload_sched_policy_mgr.cpp:35] start workload scheduler
I20251205 14:52:53.951035 389 spill_stream_manager.cpp:51] init spill stream manager
I20251205 14:52:53.951175 389 spill_stream_manager.cpp:323] spill storage path: /opt/apache-doris/be/storage, capacity: 1006.85 GB, limit: 181.23 GB, available: 1006.74 GB
I20251205 14:52:53.955405 389 spill_stream_manager.cpp:86] spill gc thread started
I20251205 14:52:53.955682 389 exec_env_init.cpp:201] Using SimdJSON Haswell implementation
I20251205 14:52:53.961054 389 agent_server.cpp:60] Register workload group listener
I20251205 14:52:53.961086 389 agent_server.cpp:66] Register workload scheduler policy listener
I20251205 14:52:53.964650 389 backend_service.cpp:662] Doris BackendService listening on 9060
I20251205 14:52:53.964686 389 backend_service.cpp:666] ingest binlog thread pool size is -1, so we will in sync mode
I20251205 14:52:53.964907 389 thrift_server.cpp:423] ThriftServer 'backend' started on port: 9060
I20251205 14:52:53.966464 389 data_dir.cpp:882] gc data dir path: /opt/apache-doris/be/storage finished
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_release/doris/be/src/common/signal_handler.h:421
 1# 0x00007FE861594520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill in /lib/x86_64-linux-gnu/libc.so.6
 3# raise in /lib/x86_64-linux-gnu/libc.so.6
 4# abort in /lib/x86_64-linux-gnu/libc.so.6
 5# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
 6# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
 7# 0x0000557692F56121 in /opt/apache-doris/be/lib/doris_be
 8# 0x0000557692F56274 in /opt/apache-doris/be/lib/doris_be
 9# std::__throw_system_error(int) at ../../../../../libstdc++-v3/src/c++11/system_error.cc:338
10# 0x00005576930251ED in /opt/apache-doris/be/lib/doris_be
11# std::thread* doris::ThreadGroup::create_thread<std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)> >(std::_Bind_result<void, std::_Mem_fn<void (doris::WorkThreadPool<false>::*)(int)> (doris::WorkThreadPool<false>*, int)>) at /home/zcp/repo_center/doris_release/doris/be/src/util/thread_group.h:65
12# doris::WorkThreadPool<false>::WorkThreadPool(unsigned int, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at /home/zcp/repo_center/doris_release/doris/be/src/util/work_thread_pool.hpp:61
13# doris::PInternalService::PInternalService(doris::ExecEnv*) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:218
14# doris::PInternalServiceImpl::PInternalServiceImpl(doris::StorageEngine&, doris::ExecEnv*) at /home/zcp/repo_center/doris_release/doris/be/src/service/internal_service.cpp:259
15# doris::BRpcService::start(int, int) in /opt/apache-doris/be/lib/doris_be
16# main at /home/zcp/repo_center/doris_release/doris/be/src/service/doris_main.cpp:570
17# 0x00007FE86157BD90 in /lib/x86_64-linux-gnu/libc.so.6
18# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6
19# _start in /opt/apache-doris/be/lib/doris_be
/opt/apache-doris/be/bin/start_be.sh: line 450:   389 Aborted                 (core dumped) ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" 2>> "${LOG_DIR}/be.out" < /dev/null
[Fri Dec  5 14:53:13 UTC 2025] run post_exit

Here is the latest be.conf

    # Enable file cache
    # enable_file_cache = true

    # JVM options for JDK 17 with module access permissions

    JAVA_OPTS_FOR_JDK_17 = "-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0
    -Xlog:gc*:$LOG_DIR/be.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
    -Djavax.security.auth.useSubjectCredsOnly=false
    -Dsun.security.krb5.debug=true -Dsun.java.command=DorisBE
    -XX:-CriticalJNINatives -XX:+IgnoreUnrecognizedVMOptions
    --add-opens=java.base/java.lang=ALL-UNNAMED
    --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
    --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
    --add-opens=java.base/java.io=ALL-UNNAMED
    --add-opens=java.base/java.net=ALL-UNNAMED
    --add-opens=java.base/java.nio=ALL-UNNAMED
    --add-opens=java.base/java.util=ALL-UNNAMED
    --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
    --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
    --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
    --add-opens=java.base/sun.nio.cs=ALL-UNNAMED"


    CUR_DATE=`date +%Y%m%d-%H%M%S`
    # Log dir
    LOG_DIR="${DORIS_HOME}/log/"
    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060
    arrow_flight_sql_port = -1


    # Choose one if there are more than one ip except loopback address.
    # Note that there should at most one ip match this list.
    # If no ip match this rule, will choose one randomly.
    # use CIDR format, e.g. 10.10.10.0/24 or IP format, e.g. 10.10.10.1
    # Default value is empty.
    # priority_networks = 10.10.10.0/24;192.168.0.0/16

    # data root path, separate by ';'
    # You can specify the storage type for each root path, HDD (cold data) or
    SSD (hot data)
    # eg:
    # storage_root_path = /home/disk1/doris;/home/disk2/doris;/home/disk2/doris
    # storage_root_path = /home/disk1/doris,medium:SSD;/home/disk2/doris,medium:SSD;/home/disk2/doris,medium:HDD
    # /home/disk2/doris,medium:HDD(default)
    #
    # you also can specify the properties by setting '<property>:<value>', separate by ','
    # property 'medium' has a higher priority than the extension of path
    #

    # Default value is ${DORIS_HOME}/storage, you should create it by hand.
    storage_root_path = ${DORIS_HOME}/storage

    # Default dirs to put jdbc drivers,default value is
    ${DORIS_HOME}/jdbc_drivers
    # jdbc_drivers_dir = ${DORIS_HOME}/jdbc_drivers
    # Advanced configurations
    # INFO, WARNING, ERROR, FATAL
    sys_log_level = DEBUG
    # sys_log_roll_mode = SIZE-MB-1024
    # sys_log_roll_num = 10
    # sys_log_verbose_modules = *
    # log_buffer_level = -1

Here is the fe.conf just in case it matters

    # Enable FQDN mode for Kubernetes - required for stable pod identity
    enable_fqdn_mode = true


    # JVM options for JDK 17 with module access permissions

    JAVA_OPTS_FOR_JDK_17 = "-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0
    -Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
    -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.java.command=DorisFE
    -XX:-CriticalJNINatives -XX:+IgnoreUnrecognizedVMOptions
    --add-opens=java.base/java.lang=ALL-UNNAMED
    --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
    --add-opens=java.base/java.lang.reflect=ALL-UNNAMED
    --add-opens=java.base/java.io=ALL-UNNAMED
    --add-opens=java.base/java.net=ALL-UNNAMED
    --add-opens=java.base/java.nio=ALL-UNNAMED
    --add-opens=java.base/java.util=ALL-UNNAMED
    --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
    --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
    --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
    --add-opens=java.base/sun.nio.cs=ALL-UNNAMED"

[yiguolei]

Does the 3 pods on the same host?

[plastikman]

No the pods are spread across multiple hosts and multiple AZ's. i have rules that place them apart.

[plastikman]

This issue appears to be an issue with AMD CPU's

I have moved away from the "Standard_D32as_v6" to "Standard_D32s_v6" nodes. This seems to be an issue with the be images as i had the same exact issue with the disaggregated storage and the integrated storage.

[yiguolei]

@plastikman You can start after moving to "Standard_D32s_v6" ? Both disaggregated storage and the integrated storage are OK on the new CPU?

[plastikman]

That is correct. We had similar threading issues also on AMD processors with ambassador. I assume that there is just some thread issue between Intel CPU and AMD cpu.