scrapy.spidermiddlewares.httperror INFO: Ignoring response 999 #6
Description
Hi,
I tried scrapy code and getting following response from server :
c:\python27\lib\site-packages\scrapy\settings\deprecated.py:27: ScrapyDeprecationWarning: You are using the following settings which are deprecated or obsolete (ask scrapy-users@googlegroups.com for alternatives): BOT_VERSION: no longer used (user agent defaults to Scrapy now) warnings.warn(msg, ScrapyDeprecationWarning)C:\Drive D\Work\Python\crawlers\linkedInScraper-master\linkedIn\linkedIn\spiders\linkedIn_spider.py:1: ScrapyDeprecationWarning: Module scrapy.spideris deprecated, usescrapy.spiders instead from scrapy.spider import BaseSpider C:\Drive D\Work\Python\crawlers\linkedInScraper-master\linkedIn\linkedIn\spiders\linkedIn_spider.py:20: ScrapyDeprecationWarning: linkedIn.spiders.linkedIn_spider.linkedInSpider inherits from deprecated class scrapy.spiders.BaseSpider, please inherit from scrapy.spiders.Spider. (warning only on first subclass, there may be others) class linkedInSpider(BaseSpider): 2018-03-15 16:34:42 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: linkedIn) 2018-03-15 16:34:42 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'linkedIn.spiders', 'FEED_URI': 'items.json', 'SPIDER_MODULES': ['linkedIn.spiders'], 'BOT_NAME': 'linkedIn', 'DEFAULT_ITEM_CLASS': 'linkedIn.items.LinkedinItem', 'FEED_FORMAT': 'json'}2018-03-15 16:34:42 [scrapy.middleware] INFO: Enabled extensions:['scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.corestats.CoreStats']2018-03-15 16:34:44 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy.downloadermiddlewares.stats.DownloaderStats'] 2018-03-15 16:34:44 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2018-03-15 16:34:44 [scrapy.middleware] INFO: Enabled item pipelines: ['linkedIn.pipelines.LinkedinPipeline'] 2018-03-15 16:34:44 [scrapy.core.engine] INFO: Spider opened 2018-03-15 16:34:44 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2018-03-15 16:34:44 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-a> (referer: None) 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-c> (referer: None) 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-d> (referer: None) 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-f> (referer: None) 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-e> (referer: None) 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-a>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-h> (referer: None) 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-c>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-d>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-f>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-e>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-h>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-b> (referer: None) 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-i> (referer: None) 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-k> (referer: None) 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-b>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-j> (referer: None) 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-l> (referer: None) 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-n> (referer: None) 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-i>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-m> (referer: None) 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-k>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-j>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-o> (referer: None) 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-l>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-g> (referer: None) 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-n>: HTTP status code is not handled or not allowed 2018-03-15 16:34:45 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-m>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-p> (referer: None) 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-q> (referer: None) 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-o>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-g>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-s> (referer: None) 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-r> (referer: None) 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-t> (referer: None) 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-p>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-u> (referer: None) 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-q>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-w> (referer: None) 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-v> (referer: None) 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-s>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-r>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-t>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-u>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-y> (referer: None) 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-w>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-x> (referer: None) 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-v>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.core.engine] DEBUG: Crawled (999) <GET https://www.linkedin.com/directory/people-z> (referer: None) 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-y>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-x>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <999 https://www.linkedin.com/directory/people-z>: HTTP status code is not handled or not allowed 2018-03-15 16:34:46 [scrapy.core.engine] INFO: Closing spider (finished) 2018-03-15 16:34:46 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'downloader/request_bytes': 8770, 'downloader/request_count': 26, 'downloader/request_method_count/GET': 26, 'downloader/response_bytes': 53336, 'downloader/response_count': 26, 'downloader/response_status_count/999': 26, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2018, 3, 15, 11, 4, 46, 403000), 'httperror/response_ignored_count': 26, 'httperror/response_ignored_status_count/999': 26, 'log_count/DEBUG': 27, 'log_count/INFO': 33, 'response_received_count': 26, 'scheduler/dequeued': 26, 'scheduler/dequeued/memory': 26, 'scheduler/enqueued': 26, 'scheduler/enqueued/memory': 26, 'start_time': datetime.datetime(2018, 3, 15, 11, 4, 44, 414000)} 2018-03-15 16:34:46 [scrapy.core.engine] INFO: Spider closed (finished)
getting the code scrapy.spidermiddlewares.httperror INFO: Ignoring response 999, please can you provide how to handle this error code from server.
Thanks