Issue with cassandra ruby driver, timecop and ione

[alexggordon]

Hello @iconara!

Figured this would be as good of a spot as any to ask this question, but we discovered an issue when using ione with the ruby cassandra driver and timecop. Specifically, this is the situation.

1. Configure and launch a cassandra server locally.
2. Configure an rspec suite for integration tests against the cassandra server.
3. Ensure the cassandra idle_timeout is set to the default value (60)
4. Adjust time using timecop to sometime in the future past 60 seconds.

Result:
The test will fail with a cassandra timeout error, as the scheduled ione timer for the idle_timeout has not been rescheduled, and the on_value timeout function has been called.

Given this, I could see an argument for either calling this a cassandra ruby driver issue, or a ione issue--if you'd like me to report this somewhere else, please let me know.

However, my question is: What is the best way to utilize timecop with ione?

[iconara]

Hello! This is a tricky one. I haven't looked at the Ruby driver code for a long time, and also not done much work on Ione for a while, so I don't have all the details in my head at the moment.

The Ione timers use Time.now to determine if they should fire, which should work with timecop, for some definition of should. However, the timers are only checked when the reactor ticks, which it does every second by default, or when there is IO. I'm not entirely sure what it is that you're trying to achieve, and how all the pieces interact. Do I understand you correctly that the problem is that the timer fires when it shouldn't? In other words, you're time traveling into the future with timecop to avoid the timer firing?

I'm no fan of using timecop. Especially not for integration tests. In a single threaded environment and if you can be sure that nothing else runs when you change the time I guess it can work, if everyone plays ball (but there are better ways to do it in that situation). In a system with moving parts like Ione (which runs the reactor in a separate thread), there's no telling what will happen.

If you tell me a little bit more about what you're trying to achieve, perhaps I can give you some advice on how to do it in a different way and work around the problem.

[alexggordon]

@iconara let me push up a repo with a demonstration

[iconara]

Sounds good.

[brendar]

I also encountered this issue, with errors like

Cassandra::Errors::NoHostsAvailable: All attempted hosts failed: 127.0.0.1 (Cassandra::Errors::IOError: Terminated due to inactivity)

and

Cassandra::Errors::NoHostsAvailable: All hosts down

I suspect that using a monotonic clock for timers e.g. Process.clock_gettime(Process::CLOCK_MONOTONIC) rather than Time.now would fix the issue since Timecop doesn't modify the monotonic clock.

Regardless, I was able to work around the issue by disabling the Cassandra driver heartbeat and idle timeout in the test environment with options:

heartbeat_interval: null,
idle_timeout: null

[Roguelazer]

I spent some time investigating this; it's definitely an issue with monotonic clocks and timecop.

There are two issues:

(1) cassandra-driver doesn't pass a monotonic clock to Ione
(2) Ione doesn't actually respect the clock argument to IoReactor.

I'll send over a PR for #2.

[Roguelazer]

datastax/ruby-driver#271 fixes ruby-cassandra-driver to use the newly-released Ione along with Monotime to fix this issue.