Enabling DUAL_ASYNC_ON_SECONDARY results in write timeouts

[dna-tsc]

We are currently experiencing issues using ZDM and we are not sure what might be the cause.

We use ZDM to replicate data between clusters.
When we use the setting read_mode: DUAL_ASYNC_ON_SECONDARY we start to observe the following error message:
Async Request failed with error code ErrorCode WriteTimeout [0x00001100]. Error message: Operation timed out - received only 1 responses.

The error message is found in the ZDM log files and there are no corresponding related messages found on the actual Cassandra nodes' log files.

We are currently having 3 nodes in the cluster and the replication factor is set to 3.
We are testing this in our pre-production environment, so the traffic is low and there are no big hits.

When we use read_mode: PRIMARY_ONLY the above error message stops being generated.

I am confused of why would the read async option cause write timeouts error messages.

I would appreciate if you can give us some guidance what can be root the cause of these write timeouts or some ideas about what and where to look for.

If you need any more information, please let me know and I would try to provide what I can.

[joao-r-reis]

I don't know what causes Cassandra to return a write timeout for a read query, maybe that's something you could ask folks in the Cassandra user mailing list or #cassandra channel on ASF slack. If those folks say that it's not possible then we have to assume ZDM is incorrectly parsing a statement as a read when in reality it is a write. Do you know which CQL statement it is?

[joao-r-reis]

Also,

We use ZDM to replicate data between clusters.

Is this a permanent replication scenario or just for a migration? The proxy was designed for migrations where its use is temporary.

[dna-tsc]

Hello, @joao-r-reis , thanks for the responses.

We are using it for a migration and we are soaking our new cluster while are migrating the old data.

Do you know which CQL statement it is?

No. Do you know how we can get to that information? The log message above is the only thing we can see in ZDM. Our target Cassandra cluster does not have any error or timeout related messages in the log file.

maybe that's something you could ask folks in the Cassandra user mailing list or #cassandra channel on ASF slack

I already did that 💯 They are speculating that the load of the async read that the target cluster has is resulting in write timeouts because ZDM is forwarding data to that cluster. But because we are in pre-production environment the load is extremely small.
We did speculate that it can be a bug in ZDM incorrectly parsing the statement.

[joao-r-reis]

Do you know how we can get to that information?

Hmm nothing comes to mind. Do you have a way of reproducing this in a test environment? You could just swap the primary cluster setting on the zdm config so that the request is sent to that cluster in a synchronous manner and therefore the client receives the error

[joao-r-reis]

They are speculating that the load of the async read that the target cluster has is resulting in write timeouts because ZDM is forwarding data to that cluster.

Where did this discussion happen? I only see your slack thread on #cassandra but I don't see that part of the discussion there

[joao-r-reis]

If you are referring to the message that Wei sent I don't think he was saying that that specific read will cause a write timeout, I think he was saying that sending those reads to the target cluster might cause the target cluster to return write timeouts to other write requests.