Need to filter better the inputs?

[felipekk]

I have setup a server running dynosaur with NewRelicInputPlugin and every day I get many exceptions like these:

comparison of Fixnum with nil failed
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/inputs/abstract_input_plugin.rb:65:in `each'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/inputs/abstract_input_plugin.rb:65:in `max'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/inputs/abstract_input_plugin.rb:65:in `max_recent_values'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/inputs/abstract_input_plugin.rb:32:in `estimated_resources'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/controllers/abstract_controller_plugin.rb:49:in `block in get_combined_estimate'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/controllers/abstract_controller_plugin.rb:48:in `each'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/controllers/abstract_controller_plugin.rb:48:in `get_combined_estimate'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/controllers/abstract_controller_plugin.rb:74:in `run'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/autoscaler.rb:73:in `block in run_loop'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/autoscaler.rb:72:in `each'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/autoscaler.rb:72:in `run_loop'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/autoscaler.rb:59:in `start'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/bin/dynosaur:24:in `'
/app/vendor/bundle/ruby/2.1.0/bin/dynosaur:23:in `load'
/app/vendor/bundle/ruby/2.1.0/bin/dynosaur:23:in `

comparison of NilClass with 2 failed
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/inputs/abstract_input_plugin.rb:65:in `each'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/inputs/abstract_input_plugin.rb:65:in `max'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/inputs/abstract_input_plugin.rb:65:in `max_recent_values'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/inputs/abstract_input_plugin.rb:32:in `estimated_resources'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/controllers/abstract_controller_plugin.rb:49:in `block in get_combined_estimate'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/controllers/abstract_controller_plugin.rb:48:in `each'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/controllers/abstract_controller_plugin.rb:48:in `get_combined_estimate'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/controllers/abstract_controller_plugin.rb:74:in `run'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/autoscaler.rb:73:in `block in run_loop'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/autoscaler.rb:72:in `each'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/autoscaler.rb:72:in `run_loop'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/lib/dynosaur/autoscaler.rb:59:in `start'
/app/vendor/bundle/ruby/2.1.0/bundler/gems/dynosaur-eb2ae5c61d4c/bin/dynosaur:24:in `'
/app/vendor/bundle/ruby/2.1.0/bin/dynosaur:23:in `load'
/app/vendor/bundle/ruby/2.1.0/bin/dynosaur:23:in `

[werkshy]

We're seeing similar issues with mild flakiness in the API causing lots of exceptions. I just released the https://github.com/harrystech/dynosaur#bugsnag error handler to at least fix the '200 emails from dynosaur' issue. @felipekk that might help you in the short term.

I think what we really need here is a way to silently ignore API failures for a short time, then raise an exception (or whatever) when the outage goes on too long. I don't have a clear idea yet how to do that cleanly.

[felipekk]

@werkshy I would store the datetime of the last "clean" result from the API in a variable, and raise an exception when that is older than X minutes.