HTTP::Parser::Error: Could not parse data entirely (EDIT: with a fix)

[thbar]

Tested against master today - when I do this to change what I track:

stream.options[:content] = new_content
stream.immediate_reconnect

I get this error (after 2 on_reconnect calls):

[GEM_ROOT]/gems/twitter-stream-0.1.14/lib/twitter/json_stream.rb:121:in `<<'
[GEM_ROOT]/gems/twitter-stream-0.1.14/lib/twitter/json_stream.rb:121:in `receive_data'
[GEM_ROOT]/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:in `run_machine'
[GEM_ROOT]/gems/eventmachine-1.0.0.beta.4/lib/eventmachine.rb:179:in `run'

I updated to master and it still applies.

Does anyone else meet this issue?

[thbar]

Just realized that the first line in the code I maintain was doing a bit of monkey patching:

module Twitter
  class JSONStream < EventMachine::Connection
    attr_reader :options
  end
end

My guess is that the use case may not be currently supported by the gem, then. I'll dive more and see if I can make this work somehow.

Any hint welcome if you read this!

[thbar]

Figured out: on reconnect, the @parser also needs to be reset, otherwise it will attempt to continue parsing as defined by the previous connection.

I made a fix that works for me, although beware, I'm only discovering this gem so there will be a better way IMO.

https://github.com/thbar/twitter-stream/compare/fix-for-27-could-not-parse-data-entirely

How to use:

Signal.trap('USR1') do
  @should_restart = true
end

EventMachine::add_periodic_timer(1) do
  if @should_restart
    tracked = get_updated_keywords.map { |e| CGI.escape(e) }.join(',')
    stream.reconnect_with_updated_config do |options|
      options[:content] = "track=#{tracked}"
    end
    @should_restart = false
  end
end

Let me know if you find a better way to do this.

[thbar]

@mgreen @rud @abuiles if this stuff makes sense, let me know and I'll wrap a proper pull-request! (I suspect there's probably a built-in way to achieve that already, but unsure).

[tibbon]

So how does one use this again? Just put in that code anywhere and it runs?

[cromulus]

n'thing tibbon: can you provide some examples of a configuration or implementation of this fix or is this something that is going to be included in a release of the gem?

[thbar]

@tibbon see my sample above; basically you need a way to detect you need to change the keywords (for my customer's app here I used a signal because it was convenient to inform the program from outside), then use reconnect_with_updated_config. (all this after applying my patch at https://github.com/thbar/twitter-stream/compare/fix-for-27-could-not-parse-data-entirely).

Hope this helps!