webcrawler This is to crawl a given website and provide the sitemap with internal links and the media