Wextracto is a toolkit for command-line web data extraction.
$ pip install wextracto$ echo -e "[wex]\nsitemaps=wex.sitemaps:urls_from_sitemaps" > entry_points.txt
$ wex "http://www.ebay.com/robots.txt"The documentation can be found here:
http://wextracto.readthedocs.org/en/latest/index.html