Data race reported by TSAN in a YugabyteDB integration test when shutting down the cluster #14
Description
Keeping a yb::CassandraSession open while restarting the yb cluster in the ClusterRestartDuringRestore causes a data race within this library. Output from tsan is here[0]. The test is here. The unprotected overwrite of Metadata::updating in Metadata::clear_and_update_back is racing with the method call on Metadata::updating::update_partitions inside Metadata::update_partitions. There's also a second race between Metadata::update_keyspaces and Metadata::update_partitions probably on the unprotected read-and-increment of Metadata::schema_snapshot_version_ inside Metadata::update_keyspaces.
[0]
WARNING: ThreadSanitizer: data race (pid=663384)
Write of size 8 at 0x7b7c000b4a58 by thread T59:
#0 cass::Metadata::clear_and_update_back(cass::VersionNumber const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/metadata.cpp:955:13 (libcassandra.so.2+0x19f950)
#1 cass::ControlConnection::on_query_meta_schema(cass::ControlConnection*, cass::ControlConnection::UnusedData const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, cass::SharedRefPtr<cass::Response>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, cass::SharedRefPtr<cass::Response> > > > const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/control_connection.cpp (libcassandra.so.2+0x154186)
#2 cass::ControlConnection::ControlMultipleRequestCallback<cass::ControlConnection::UnusedData>::on_set(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, cass::SharedRefPtr<cass::Response>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, cass::SharedRefPtr<cass::Response> > > > const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/control_connection.cpp:1171:3 (libcassandra.so.2+0x158192)
#3 cass::MultipleRequestCallback::InternalCallback::on_internal_set(cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_callback.cpp:227:14 (libcassandra.so.2+0x1eab40)
#4 cass::SimpleRequestCallback::on_set(cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_callback.cpp:258:3 (libcassandra.so.2+0x1eb00c)
#5 cass::Connection::consume(char*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/connection.cpp:472:25 (libcassandra.so.2+0x13fd0c)
#6 cass::Connection::on_read(uv_stream_s*, long, uv_buf_t const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/connection.cpp:670:15 (libcassandra.so.2+0x1408fd)
#7 uv__read /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/stream.c:1257:7 (libuv.so+0x2aa2b)
#8 uv__stream_io /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/stream.c:1324:5 (libuv.so+0x2aa2b)
#9 uv__io_poll /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/linux-core.c:401:11 (libuv.so+0x32b94)
#10 uv_run /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/core.c:370:5 (libuv.so+0x1abe3)
#11 cass::LoopThread::on_run_internal(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:128:5 (libcassandra.so.2+0x21031d)
Previous read of size 8 at 0x7b7c000b4a58 by thread T60 (mutexes: write M790798982087133632, write M789391607203580552):
#0 cass::Metadata::update_partitions(int, cass::VersionNumber const&, cass::ResultResponse*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/metadata.cpp:895:3 (libcassandra.so.2+0x19d4e2)
#1 cass::Session::refresh_metadata_callback(CassFuture_*, void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:500:23 (libcassandra.so.2+0x20792f)
#2 cass::Future::internal_set(cass::ScopedLock<cass::Mutex>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/future.cpp:190:5 (libcassandra.so.2+0x17cfb0)
#3 cass::ResponseFuture::set_response(cass::Address, cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.hpp:64:7 (libcassandra.so.2+0x1f1945)
#4 cass::RequestHandler::set_response(cass::SharedRefPtr<cass::Host> const&, cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:142:16 (libcassandra.so.2+0x1efc90)
#5 cass::RequestExecution::set_response(cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:538:21 (libcassandra.so.2+0x1efc90)
#6 cass::RequestExecution::on_result_response(cass::Connection*, cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:369:7 (libcassandra.so.2+0x1efc90)
#7 cass::RequestExecution::on_set(cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:252:7 (libcassandra.so.2+0x1ef51d)
#8 cass::Connection::consume(char*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/connection.cpp:472:25 (libcassandra.so.2+0x13fd0c)
#9 cass::Connection::on_read(uv_stream_s*, long, uv_buf_t const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/connection.cpp:670:15 (libcassandra.so.2+0x1408fd)
#10 uv__read /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/stream.c:1257:7 (libuv.so+0x2aa2b)
#11 uv__stream_io /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/stream.c:1324:5 (libuv.so+0x2aa2b)
#12 uv__io_poll /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/linux-core.c:401:11 (libuv.so+0x32b94)
#13 uv_run /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/core.c:370:5 (libuv.so+0x1abe3)
#14 cass::LoopThread::on_run_internal(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:128:5 (libcassandra.so.2+0x21031d)
Location is heap block of size 3128 at 0x7b7c000b4400 allocated by main thread:
#0 operator new(unsigned long) /opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1633143152-bdb147e6-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (yb-admin-snapshot-schedule-test+0x385cc6)
#1 cass_session_new /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:36:26 (libcassandra.so.2+0x200139)
#2 yb::CassandraSession::Connect(CassCluster_*) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:333:23 (libcql_test_util.so+0x31534)
#3 yb::CassandraSession::Create(CassCluster_*) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:340:3 (libcql_test_util.so+0x3164c)
#4 yb::CppCassandraDriver::CreateSession() /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:483:10 (libcql_test_util.so+0x328e8)
#5 yb::tools::AdminTestBase::CqlConnect(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/admin-test-base.cc:77:17 (libadmin-test-base.so+0x1d343)
#6 yb::tools::YbAdminSnapshotScheduleFailoverTests::ClusterRestartTest(bool) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:2761:17 (yb-admin-snapshot-schedule-test+0x3c656e)
#7 yb::tools::YbAdminSnapshotScheduleFailoverTests_ClusterRestartDuringRestoreWithReplayUncommitted_Test::TestBody() /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:2868:3 (yb-admin-snapshot-schedule-test+0x3c8885)
#8 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x67b8f)
#9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2438:14 (libgmock.so+0x67b8f)
#10 testing::Test::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x47cc8)
#11 testing::TestInfo::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x48f5d)
#12 testing::TestCase::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x49c06)
#13 testing::internal::UnitTestImpl::RunAllTests() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x54606)
#14 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x6899f)
#15 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2438:14 (libgmock.so+0x6899f)
#16 testing::UnitTest::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x53d59)
#17 RUN_ALL_TESTS() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/installed/tsan/include/gtest/gtest.h:2233:46 (libyb_test_main.so+0x754b)
#18 main /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/util/test_main.cc:105:13 (libyb_test_main.so+0x70e7)
Mutex M790798982087133632 is already destroyed.
Mutex M789391607203580552 is already destroyed.
Thread T59 (tid=663903, running) created by main thread at:
#0 pthread_create /opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1633143152-bdb147e6-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:965:3 (yb-admin-snapshot-schedule-test+0x3148bb)
#1 uv_thread_create /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/thread.c:202:9 (libuv.so+0x2e4a3)
#2 cass::LoopThread::run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:106:14 (libcassandra.so.2+0x200f5e)
#3 cass::Session::connect_async(cass::Config const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, cass::SharedRefPtr<cass::Future> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:381:3 (libcassandra.so.2+0x200f5e)
#4 cass_session_connect_keyspace_n /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:68:12 (libcassandra.so.2+0x2008ff)
#5 cass_session_connect_keyspace /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:57:10 (libcassandra.so.2+0x2006f9)
#6 cass_session_connect /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:51:10 (libcassandra.so.2+0x2006f9)
#7 yb::CassandraSession::Connect(CassCluster_*) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:334:26 (libcql_test_util.so+0x31580)
#8 yb::CassandraSession::Create(CassCluster_*) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:340:3 (libcql_test_util.so+0x3164c)
#9 yb::CppCassandraDriver::CreateSession() /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:483:10 (libcql_test_util.so+0x328e8)
#10 yb::tools::AdminTestBase::CqlConnect(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/admin-test-base.cc:77:17 (libadmin-test-base.so+0x1d343)
#11 yb::tools::YbAdminSnapshotScheduleFailoverTests::ClusterRestartTest(bool) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:2761:17 (yb-admin-snapshot-schedule-test+0x3c656e)
#12 yb::tools::YbAdminSnapshotScheduleFailoverTests_ClusterRestartDuringRestoreWithReplayUncommitted_Test::TestBody() /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:2868:3 (yb-admin-snapshot-schedule-test+0x3c8885)
#13 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x67b8f)
#14 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2438:14 (libgmock.so+0x67b8f)
#15 testing::Test::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x47cc8)
#16 testing::TestInfo::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x48f5d)
#17 testing::TestCase::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x49c06)
#18 testing::internal::UnitTestImpl::RunAllTests() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x54606)
#19 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x6899f)
#20 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2438:14 (libgmock.so+0x6899f)
#21 testing::UnitTest::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x53d59)
#22 RUN_ALL_TESTS() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/installed/tsan/include/gtest/gtest.h:2233:46 (libyb_test_main.so+0x754b)
#23 main /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/util/test_main.cc:105:13 (libyb_test_main.so+0x70e7)
Thread T60 (tid=663904, running) created by thread T59 at:
#0 pthread_create /opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1633143152-bdb147e6-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:965:3 (yb-admin-snapshot-schedule-test+0x3148bb)
#1 uv_thread_create /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/thread.c:202:9 (libuv.so+0x2e4a3)
#2 cass::LoopThread::run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:106:14 (libcassandra.so.2+0x2082e5)
#3 cass::Session::on_run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:555:12 (libcassandra.so.2+0x2082e5)
#4 cass::LoopThread::on_run_internal(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:127:13 (libcassandra.so.2+0x210312)
SUMMARY: ThreadSanitizer: data race /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/metadata.cpp:955:13 in cass::Metadata::clear_and_update_back(cass::VersionNumber const&)
==================
==================
WARNING: ThreadSanitizer: data race (pid=663384)
Write of size 4 at 0x7b7c000b4a80 by thread T59:
#0 cass::Metadata::update_keyspaces(int, cass::VersionNumber const&, cass::ResultResponse*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/metadata.cpp:799:27 (libcassandra.so.2+0x1966c3)
#1 cass::ControlConnection::on_query_meta_schema(cass::ControlConnection*, cass::ControlConnection::UnusedData const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, cass::SharedRefPtr<cass::Response>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, cass::SharedRefPtr<cass::Response> > > > const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/control_connection.cpp:597:27 (libcassandra.so.2+0x1541ec)
#2 cass::ControlConnection::ControlMultipleRequestCallback<cass::ControlConnection::UnusedData>::on_set(std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, cass::SharedRefPtr<cass::Response>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, cass::SharedRefPtr<cass::Response> > > > const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/control_connection.cpp:1171:3 (libcassandra.so.2+0x158192)
#3 cass::MultipleRequestCallback::InternalCallback::on_internal_set(cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_callback.cpp:227:14 (libcassandra.so.2+0x1eab40)
#4 cass::SimpleRequestCallback::on_set(cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_callback.cpp:258:3 (libcassandra.so.2+0x1eb00c)
#5 cass::Connection::consume(char*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/connection.cpp:472:25 (libcassandra.so.2+0x13fd0c)
#6 cass::Connection::on_read(uv_stream_s*, long, uv_buf_t const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/connection.cpp:670:15 (libcassandra.so.2+0x1408fd)
#7 uv__read /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/stream.c:1257:7 (libuv.so+0x2aa2b)
#8 uv__stream_io /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/stream.c:1324:5 (libuv.so+0x2aa2b)
#9 uv__io_poll /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/linux-core.c:401:11 (libuv.so+0x32b94)
#10 uv_run /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/core.c:370:5 (libuv.so+0x1abe3)
#11 cass::LoopThread::on_run_internal(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:128:5 (libcassandra.so.2+0x21031d)
Previous write of size 4 at 0x7b7c000b4a80 by thread T60 (mutexes: write M790798982087133632, write M789391607203580552):
#0 cass::Metadata::update_partitions(int, cass::VersionNumber const&, cass::ResultResponse*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/metadata.cpp:894:27 (libcassandra.so.2+0x19d4d6)
#1 cass::Session::refresh_metadata_callback(CassFuture_*, void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:500:23 (libcassandra.so.2+0x20792f)
#2 cass::Future::internal_set(cass::ScopedLock<cass::Mutex>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/future.cpp:190:5 (libcassandra.so.2+0x17cfb0)
#3 cass::ResponseFuture::set_response(cass::Address, cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.hpp:64:7 (libcassandra.so.2+0x1f1945)
#4 cass::RequestHandler::set_response(cass::SharedRefPtr<cass::Host> const&, cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:142:16 (libcassandra.so.2+0x1efc90)
#5 cass::RequestExecution::set_response(cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:538:21 (libcassandra.so.2+0x1efc90)
#6 cass::RequestExecution::on_result_response(cass::Connection*, cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:369:7 (libcassandra.so.2+0x1efc90)
#7 cass::RequestExecution::on_set(cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:252:7 (libcassandra.so.2+0x1ef51d)
#8 cass::Connection::consume(char*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/connection.cpp:472:25 (libcassandra.so.2+0x13fd0c)
#9 cass::Connection::on_read(uv_stream_s*, long, uv_buf_t const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/connection.cpp:670:15 (libcassandra.so.2+0x1408fd)
#10 uv__read /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/stream.c:1257:7 (libuv.so+0x2aa2b)
#11 uv__stream_io /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/stream.c:1324:5 (libuv.so+0x2aa2b)
#12 uv__io_poll /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/linux-core.c:401:11 (libuv.so+0x32b94)
#13 uv_run /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/core.c:370:5 (libuv.so+0x1abe3)
#14 cass::LoopThread::on_run_internal(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:128:5 (libcassandra.so.2+0x21031d)
Location is heap block of size 3128 at 0x7b7c000b4400 allocated by main thread:
#0 operator new(unsigned long) /opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1633143152-bdb147e6-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64:3 (yb-admin-snapshot-schedule-test+0x385cc6)
#1 cass_session_new /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:36:26 (libcassandra.so.2+0x200139)
#2 yb::CassandraSession::Connect(CassCluster_*) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:333:23 (libcql_test_util.so+0x31534)
#3 yb::CassandraSession::Create(CassCluster_*) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:340:3 (libcql_test_util.so+0x3164c)
#4 yb::CppCassandraDriver::CreateSession() /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:483:10 (libcql_test_util.so+0x328e8)
#5 yb::tools::AdminTestBase::CqlConnect(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/admin-test-base.cc:77:17 (libadmin-test-base.so+0x1d343)
#6 yb::tools::YbAdminSnapshotScheduleFailoverTests::ClusterRestartTest(bool) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:2761:17 (yb-admin-snapshot-schedule-test+0x3c656e)
#7 yb::tools::YbAdminSnapshotScheduleFailoverTests_ClusterRestartDuringRestoreWithReplayUncommitted_Test::TestBody() /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:2868:3 (yb-admin-snapshot-schedule-test+0x3c8885)
#8 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x67b8f)
#9 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2438:14 (libgmock.so+0x67b8f)
#10 testing::Test::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x47cc8)
#11 testing::TestInfo::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x48f5d)
#12 testing::TestCase::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x49c06)
#13 testing::internal::UnitTestImpl::RunAllTests() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x54606)
#14 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x6899f)
#15 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2438:14 (libgmock.so+0x6899f)
#16 testing::UnitTest::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x53d59)
#17 RUN_ALL_TESTS() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/installed/tsan/include/gtest/gtest.h:2233:46 (libyb_test_main.so+0x754b)
#18 main /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/util/test_main.cc:105:13 (libyb_test_main.so+0x70e7)
Mutex M790798982087133632 is already destroyed.
Mutex M789391607203580552 is already destroyed.
Thread T59 (tid=663903, running) created by main thread at:
#0 pthread_create /opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1633143152-bdb147e6-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:965:3 (yb-admin-snapshot-schedule-test+0x3148bb)
#1 uv_thread_create /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/thread.c:202:9 (libuv.so+0x2e4a3)
#2 cass::LoopThread::run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:106:14 (libcassandra.so.2+0x200f5e)
#3 cass::Session::connect_async(cass::Config const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, cass::SharedRefPtr<cass::Future> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:381:3 (libcassandra.so.2+0x200f5e)
#4 cass_session_connect_keyspace_n /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:68:12 (libcassandra.so.2+0x2008ff)
#5 cass_session_connect_keyspace /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:57:10 (libcassandra.so.2+0x2006f9)
#6 cass_session_connect /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:51:10 (libcassandra.so.2+0x2006f9)
#7 yb::CassandraSession::Connect(CassCluster_*) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:334:26 (libcql_test_util.so+0x31580)
#8 yb::CassandraSession::Create(CassCluster_*) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:340:3 (libcql_test_util.so+0x3164c)
#9 yb::CppCassandraDriver::CreateSession() /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/integration-tests/cql_test_util.cc:483:10 (libcql_test_util.so+0x328e8)
#10 yb::tools::AdminTestBase::CqlConnect(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/admin-test-base.cc:77:17 (libadmin-test-base.so+0x1d343)
#11 yb::tools::YbAdminSnapshotScheduleFailoverTests::ClusterRestartTest(bool) /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:2761:17 (yb-admin-snapshot-schedule-test+0x3c656e)
#12 yb::tools::YbAdminSnapshotScheduleFailoverTests_ClusterRestartDuringRestoreWithReplayUncommitted_Test::TestBody() /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/tools/yb-admin-snapshot-schedule-test.cc:2868:3 (yb-admin-snapshot-schedule-test+0x3c8885)
#13 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x67b8f)
#14 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2438:14 (libgmock.so+0x67b8f)
#15 testing::Test::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2474:5 (libgmock.so+0x47cc8)
#16 testing::TestInfo::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2656:11 (libgmock.so+0x48f5d)
#17 testing::TestCase::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2774:28 (libgmock.so+0x49c06)
#18 testing::internal::UnitTestImpl::RunAllTests() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:4649:43 (libgmock.so+0x54606)
#19 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2402:10 (libgmock.so+0x6899f)
#20 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:2438:14 (libgmock.so+0x6899f)
#21 testing::UnitTest::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/gmock-1.8.0/googletest/src/gtest.cc:4257:10 (libgmock.so+0x53d59)
#22 RUN_ALL_TESTS() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/installed/tsan/include/gtest/gtest.h:2233:46 (libyb_test_main.so+0x754b)
#23 main /nfusr/dev-server/zdrudi/code/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/util/test_main.cc:105:13 (libyb_test_main.so+0x70e7)
Thread T60 (tid=663904, running) created by thread T59 at:
#0 pthread_create /opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1633143152-bdb147e6-almalinux8-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:965:3 (yb-admin-snapshot-schedule-test+0x3148bb)
#1 uv_thread_create /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/libuv-1.23.0/src/unix/thread.c:202:9 (libuv.so+0x2e4a3)
#2 cass::LoopThread::run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:106:14 (libcassandra.so.2+0x2082e5)
#3 cass::Session::on_run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:555:12 (libcassandra.so.2+0x2082e5)
#4 cass::LoopThread::on_run_internal(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/loop_thread.hpp:127:13 (libcassandra.so.2+0x210312)
SUMMARY: ThreadSanitizer: data race /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181611-c1907426fc-almalinux8-x86_64-clang12/src/cassandra-cpp-driver-2.9.0-yb-13/src/metadata.cpp:799:27 in cass::Metadata::update_keyspaces(int, cass::VersionNumber const&, cass::ResultResponse*)