Consider moving robots.txt checks into this module #36
Description
Currently we check this in the consuming application, but it might make sense to move that functionality into this module
{{ message }}
Currently we check this in the consuming application, but it might make sense to move that functionality into this module