Python Web Crawler

This script will gather all dead links in a particular website.

Pre-requisites

• python 3

Install the python packages.

• pip3 install -r requirements.txt

Run application

• python3 ./app/main.py
• Enter a website to crawl (Default is [http://www.pythonanywhere.com/])
• Go for coffee
• Check ./dead_link_crawler/dead_link_crawler_files/dead.txt for a list of all dead links

Run unit tests

• cd app
• python3 -m pytest