Scraper stalls on random websites

[m1cr0man]

Occasionally the scraper will stall indefinitely on a website. If I go into the screen session and hit ctrl+c it kills something associated with the requests library and then continues to the next website.

My guess would be that to reproduce this all you would need to do is open 100 or so random URLs in series and watch for one of them not to respond.

This is in turn stopping the stream page updating and also stalling the email summaries. It needs to be fixed

[CianLR]

Can you check the logs and let us know what feed?
On 21 May 2016 6:17 p.m., "Lucas Savva" notifications@github.com wrote:

Occasionally the scraper will stall indefinitely on a website. If I go
into the screen session and hit ctrl+c it kills something associated with
the requests library and then continues to the next website.

My guess would be that to reproduce this all you would need to do is open
100 or so random URLs in series and watch for one of them not to respond.

This is in turn stopping the stream page updating and also stalling the
email summaries. It needs to be fixed

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#280

[m1cr0man]

The last time it ran it crashed on http://erikbern.com/atom.xml, but it seems since then it hasn't run as someone actually killed scraper (double ctrl+c). I saw it freeze on Reddit once too. Like I said it's random (IE any feed).

[CianLR]

Hm, I'll try and reproduce this evening
On 21 May 2016 6:21 p.m., "Lucas Savva" notifications@github.com wrote:

The last time it ran it crashed on http://erikbern.com/atom.xml, but it
seems since then it hasn't run as someone actually killed scraper (double
ctrl+c). I saw it freeze on Reddit once too. Like I said it's random (IE
any feed).

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#280 (comment)

[CianLR]

And if I have no luck with that I'll come home and try to get the crash to occur on a test server

[devoxel]

It seems to be dead 100%. Can't ssh into it. @m1cr0man, is m1cr0man.com down