Screen control stops working if bandshell-timer.rb crashes

[asquared]

I noticed on one of the RPI screens that was getting stuck "off" that bandshell-timer.rb wasn't running. It had crashed with what appears to be a HTTP timeout:

bandshell-timer.rb: process with pid 2370 started.

bandshell-timer.rb (2015-12-07 04:03:22 +0000): Bandshell timer starting up.
bandshell-timer.rb (2015-12-07 04:03:22 +0000): connecting to bandshell at http://localhost:4567
/usr/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill': Timeout::Error (Timeout::Error)
    from /usr/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
    from /usr/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
    from /usr/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
    from /usr/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
    from /usr/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
    from /usr/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1293:in `request'
    from /usr/lib/ruby/1.9.1/net/http.rb:1195:in `request_get'
    from /usr/lib/ruby/1.9.1/net/http.rb:455:in `block in get_response'
    from /usr/lib/ruby/1.9.1/net/http.rb:745:in `start'
    from /usr/lib/ruby/1.9.1/net/http.rb:454:in `get_response'
    from /var/lib/gems/1.9.1/gems/bandshell-1.6/bin/bandshell-timer.rb:36:in `block in <top (required)>'
    from /var/lib/gems/1.9.1/gems/bandshell-1.6/bin/bandshell-timer.rb:27:in `loop'
    from /var/lib/gems/1.9.1/gems/bandshell-1.6/bin/bandshell-timer.rb:27:in `<top (required)>'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/application.rb:218:in `load'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/application.rb:218:in `start_load'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/application.rb:297:in `start'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/controller.rb:56:in `run'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons.rb:144:in `block in run'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/cmdline.rb:88:in `call'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons/cmdline.rb:88:in `catch_exceptions'
    from /var/lib/gems/1.9.1/gems/daemons-1.2.3/lib/daemons.rb:143:in `run'
    from /var/lib/gems/1.9.1/gems/bandshell-1.6/bin/bandshelld:22:in `<top (required)>'
    from /usr/local/bin/bandshelld:23:in `load'
    from /usr/local/bin/bandshelld:23:in `<main>'

It's probably not good that this is crashing the whole program, but more importantly, perhaps we should look into running this every 10 seconds or so from crontab and avoiding the long-running process. I'm less worried about the actual Bandshell app crashing and burning because Sinatra should have some error handling that limits that to one request.

Also of note is that this seems to have happened just seconds after boot. Maybe it's racing against the main bandshell process's startup?

[mikldt]

I think that's likely: Sinatra likely takes longer to come up, and the way bandshelld is set up it won't wait for Sinatra to boot to start the timer. I agree with the catch-all in f344578 - this is the second time we've come across a one-time exception that would be resolved by keeping the process alive a little longer.

My only issue with cron would be that it makes it a little harder to run bandshell standalone for development, but if we can come up with something that can work either way, I'd be ok with that.

[asquared]

I'm not really opposed to the daemon as long as there's a mechanism to keep
it running. Perhaps once we move the player to Debian jessie we can have
systemd supervise the two daemons and restart them if they crash.

On Fri, Dec 18, 2015 at 11:10 AM, Mike notifications@github.com wrote:

I think that's likely: Sinatra likely takes longer to come up, and the way
bandshelld is set up it won't wait for Sinatra to boot to start the timer.
I agree with the catch-all in f344578
f344578

• this is the second time we've come across a one-time exception that would
  be resolved by keeping the process alive a little longer.

My only issue with cron would be that it makes it a little harder to run
bandshell standalone for development, but if we can come up with something
that can work either way, I'd be ok with that.

—
Reply to this email directly or view it on GitHub
#50 (comment).