-
Notifications
You must be signed in to change notification settings - Fork 5
Open
Description
DropTable fails during RocksdbCloud single longevity test. Since the first time the DropTable fails, it keeps retrying and failing for two days.
Detailed log file is on eloq@store-2:/mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/logs/node-6389/eloqkv.log.INFO.20250906-174059.2981528. The file is stored to eloq@eloq-super-server1/home/eloq/store_handler_issue_17/eloqkv.log.INFO.20250906-174059.2981528.
10838 [2025-09-06T17:54:05.118448 I 2981604] txn: 4345906833 process acquire_all_intent_op_ 10839 [2025-09-06T17:54:05.121887 I 2981604] txn: 4345906833 process prepare_log_op_
10840 [2025-09-06T17:54:05.133042 I 2981604] txn: 4345906833 process post_all_intent_op_
10841 [2025-09-06T17:54:05.135954 I 2981604] txn: 4345906833 process acquire_all_lock_op_
10842 [2025-09-06T17:54:05.136749 I 2981604] txn: 4345906833 process commit_log_op_ 10843 [2025-09-06T17:54:05.142589 I 2981604] txn: 4345906833 process upsert_kv_table_op_
10844 [2025-09-06T17:54:05.143138 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
10845 [2025-09-06T17:54:05.144135 E 2981616] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
10846 [2025-09-06T17:54:05.144574 W 2981678] DataStoreHandler: Failed to do DropKvTable.
10847 [2025-09-06T17:54:05.938580 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
10848 [2025-09-06T17:54:05.939280 E 2981618] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
10849 [2025-09-06T17:54:05.939636 W 2981678] DataStoreHandler: Failed to do DropKvTable.
10850 [2025-09-06T17:54:07.942220 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
10851 [2025-09-06T17:54:07.943130 E 2981628] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached 10852 [2025-09-06T17:54:07.943652 W 2981678] DataStoreHandler: Failed to do DropKvTable.
10853 [2025-09-06T17:54:11.949412 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
10854 [2025-09-06T17:54:11.950407 E 2981617] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
465661 [2025-09-08T12:58:31.341657 W 2981678] DataStoreHandler: Failed to do DropKvTable.
465662 [2025-09-08T12:58:33.276012 I 2981614] Total memtables size: 0 MB
465663 [2025-09-08T12:58:33.276227 I 2981614] Total SSTables size used in replay: 0 MB
465664 [2025-09-08T12:58:33.276286 I 2981614] Total replay log size: 0 MB
465665 [2025-09-08T12:58:33.353834 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
465666 [2025-09-08T12:58:33.354503 E 2981616] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
465667 [2025-09-08T12:58:33.354805 W 2981678] DataStoreHandler: Failed to do DropKvTable.
465668 [2025-09-08T12:58:33.442333 I 2981713] [Global dead lock detector]: gather lock dependancy information
465669 [2025-09-08T12:58:37.360330 I 2981678] DropTableWithRetry for table: data_table_0_2025_09_06_17_46_39_4345906721
465670 [2025-09-08T12:58:37.361240 E 2981617] Unable to drop table with error: IO error: /mnt/ssd/eloq/zhouxiao/jepsen-eloqkv-rocksdbcloud-single/EloqKV/data/port-6389/eloq_dss/rocksdb_data /ds_0/db//004112.sst-62afc6c96861c170: curlCode: 28, Timeout was reached
Reactions are currently unavailable