Search results

HTTP request User-Agent header styled incorrectly

Status: Open.

#30 In ekalinin/robots.js;

· midnightfreddie opened on Nov 9, 2017

Provide a CLI interface to test if a specific URL is accessible using a specific user agent

Status: Open.

#28 In ekalinin/robots.js;

· gajus opened on Jul 23, 2017

setUrl() is blocked

Status: Open.

#23 In ekalinin/robots.js;

· fprivitera opened on Apr 8, 2016

Access true while robots.txt disallows all

Status: Open.

#22 In ekalinin/robots.js;

· basvdijk opened on Sep 1, 2015

'Allow all' statement or blank line returns true for disallowed URLs

Status: Open.

#19 In ekalinin/robots.js;

· sanderheilbron opened on Mar 21, 2015

Don't return first rule match for canFetch

Status: Open.

#18 In ekalinin/robots.js;

· sebastianwessel opened on Jan 11, 2015