Skip to content

Add QueryTypeFilter compatibility to CassandraSinkCluster#1995

Merged
justinweng-instaclustr merged 8 commits intoshotover:mainfrom
justinweng-instaclustr:1922-cassandrasinkcluster-querytypefilter-compatibility
Mar 26, 2026
Merged

Add QueryTypeFilter compatibility to CassandraSinkCluster#1995
justinweng-instaclustr merged 8 commits intoshotover:mainfrom
justinweng-instaclustr:1922-cassandrasinkcluster-querytypefilter-compatibility

Conversation

@justinweng-instaclustr
Copy link
Collaborator

@justinweng-instaclustr justinweng-instaclustr commented Mar 18, 2026

This PR first adds a new integration test to reproduce the incompatibility problem between QueryTypeFilter and CassandraSinkCluster. This new test places a QueryTypeFilter in front of CassandraSinkCluster in the transform chain which causes the error below (ref: example failed CI test). The problem here is that QueryTypeFilter replaces the filtered request with a Frame::Dummy which doesn't have a stream_id, and passes the dummy frame to the downstream CassandraSinkCluster which then runs x.stream_id().unwrap(), leading to the error of calling Option::unwrap() on the None value.

shotover   23:46:34.417319Z PANIC panicked at shotover/src/transforms/cassandra/sink_cluster/connection.rs:30:59:
called `Option::unwrap()` on a `None` value
   0: __rustc::rust_begin_unwind
             at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/std/src/panicking.rs:689:5
   1: core::panicking::panic_fmt
             at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panicking.rs:80:14
   2: core::panicking::panic
             at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/panicking.rs:150:5
   3: core::option::unwrap_failed
             at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/option.rs:2199:5
   4: core::option::Option<T>::unwrap
             at /rustc/4a4ef493e3a1488c6e321570238084b38948f6db/library/core/src/option.rs:1016:21
      shotover::transforms::cassandra::sink_cluster::connection::CassandraConnection::send::{{closure}}
             at /home/runner/work/shotover-proxy/shotover-proxy/shotover/src/transforms/cassandra/sink_cluster/connection.rs:30:59

To resolve this incompatibility issue, extra logic for handling dummy message is added to [‎shotover/src/transforms/cassandra/sink_cluster/connection.rs and ‎shotover/src/transforms/cassandra/sink_cluster/mod.rs, allowing the new integration test to pass.

Additionally, the Cassandra error type for filtered responses is changed from ErrorType::Server to ErrorType::Invalid because the Datastax C++ driver defuncts the connection on Server errors which breaks subsequent queries (other drivers handle server errors more gracefully and are fine).

Specifically, the C++ driver produced this warning (WARN datastax::internal::core::RequestExecution::on_error_response: Received server error 'Message was filtered out by shotover' from host 127.0.0.1. Defuncting the connection...) in the integration test, which then led to the error thread 'cassandra_int_tests::cluster_single_rack_v4_query_type_filter::case_1_cpp' (5539) panicked at test-helpers/src/connection/cassandra/connection/cpp.rs:164:25: Unexpected cassandra_cpp error: Cassandra error LIB_NO_HOSTS_AVAILABLE: All hosts in current policy attempted and were either unavailable or failed because the driver regarded nodes as down after defuncting the connection. Example CI run here.

A filtered query is a rejection instead of a server bug and so Invalid is more appropriate than Server error here. A new protocol-agonistic MessageErrorType enum is hence introduced to support passing the error type from the common QueryTypeFilter to the protocol-specific response mapper.

@justinweng-instaclustr justinweng-instaclustr linked an issue Mar 18, 2026 that may be closed by this pull request
@justinweng-instaclustr justinweng-instaclustr changed the title #1922 Support QueryTypeFilter for CassandraSinkCluster #1922 Add QueryTypeFilter compatibility to CassandraSinkCluster Mar 18, 2026
@justinweng-instaclustr justinweng-instaclustr changed the title #1922 Add QueryTypeFilter compatibility to CassandraSinkCluster Add QueryTypeFilter compatibility to CassandraSinkCluster Mar 18, 2026
@codspeed-hq
Copy link

codspeed-hq bot commented Mar 18, 2026

Merging this PR will not alter performance

✅ 36 untouched benchmarks
⏩ 2 skipped benchmarks1


Comparing justinweng-instaclustr:1922-cassandrasinkcluster-querytypefilter-compatibility (6386101) with main (836af57)

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes an incompatibility between QueryTypeFilter (which replaces filtered requests with Frame::Dummy) and CassandraSinkCluster (which previously assumed all requests had a Cassandra stream_id). It also introduces protocol-agnostic error classification so filtered Cassandra responses map to ErrorType::Invalid (avoiding defunct connections in the Datastax C++ driver), and adds an integration test reproducing the original failure.

Changes:

  • Add MessageErrorType and thread it through error-response construction so transforms can classify errors as Internal vs Rejected (Cassandra maps to Server vs Invalid).
  • Update CassandraSinkCluster / connection tracking to tolerate dummy messages instead of panicking on missing stream_id.
  • Add a Cassandra cluster integration test and config that places QueryTypeFilter in front of CassandraSinkCluster.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
shotover/src/transforms/tee.rs Updates Tee’s mismatch error construction to pass MessageErrorType.
shotover/src/transforms/null.rs Updates NullSink error construction to pass MessageErrorType.
shotover/src/transforms/filter.rs Updates QueryTypeFilter to classify filtered requests as Rejected.
shotover/src/transforms/cassandra/sink_cluster/rewrite.rs Updates internal Cassandra rewrite error construction to pass MessageErrorType.
shotover/src/transforms/cassandra/sink_cluster/mod.rs Avoids version detection using dummy frames; generates dummy responses for dummy requests during routing.
shotover/src/transforms/cassandra/sink_cluster/connection.rs Stops unwrapping stream_id for requests; skips missing stream_ids.
shotover/src/source_task.rs Updates internal “bug” error responses to use MessageErrorType::Internal.
shotover/src/message/mod.rs Introduces MessageErrorType and adds it to to_error_response / from_*_to_error_response APIs with protocol mappings.
shotover/src/frame/cassandra.rs Allows Cassandra error responses to specify ErrorType instead of hardcoding Server.
shotover-proxy/tests/test-configs/cassandra/cluster-v4/topology-query-type-filter.yaml New test config chaining QueryTypeFilter -> CassandraSinkCluster.
shotover-proxy/tests/cassandra_int_tests/mod.rs New integration test validating filtered writes return ErrorType::Invalid and subsequent queries still succeed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Collaborator

@yallen-ic yallen-ic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@justinweng-instaclustr justinweng-instaclustr merged commit 62843e3 into shotover:main Mar 26, 2026
43 checks passed
@justinweng-instaclustr justinweng-instaclustr deleted the 1922-cassandrasinkcluster-querytypefilter-compatibility branch March 26, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CassandraSinkCluster + QueryTypeFilter compatibility

4 participants