运行一段时间后报错 #37
Description
爬虫运行一段时间后报错如下，然后就中断无法运行了
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/scrapy/core/downloader/middleware.py", line 44, in process_request
return (yield download_func(request=request, spider=spider))
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure twisted.internet.error.ConnectionDone: Connection was closed cleanly.>]