Search results

Acquire random URL from the to_visit_q (not queue anymore, strictly speaking..)

enhancement

Status: Open.

#10 In digawp/cs3103-crawler;

· digawp opened on Sep 27, 2016

Limit each domain visit by depth not by number

enhancement

Status: Open.

#9 In digawp/cs3103-crawler;

· digawp opened on Sep 25, 2016

Check protocol too before pushing to queue of URL to be visited

Status: Open.

#8 In digawp/cs3103-crawler;

· digawp opened on Sep 25, 2016

Log Errors

enhancement

Status: Open.

#4 In digawp/cs3103-crawler;

· digawp opened on Sep 24, 2016

Customize parameters

enhancement

Status: Open.

#3 In digawp/cs3103-crawler;

· digawp opened on Sep 24, 2016

Read robots.txt

enhancement

Status: Open.

#2 In digawp/cs3103-crawler;

· digawp opened on Sep 24, 2016

Crawler termination logic

Status: Open.

#1 In digawp/cs3103-crawler;

· digawp opened on Sep 24, 2016