chore: block robots.txt on all mfes, idas, and edxapp#300

deborahgu merged 2 commits intomasteredx/configuration:masterfrom

dkaplan1/APER-4252_block-robots.txt-on-all-mfes,-idas,-and-edxappedx/configuration:dkaplan1/APER-4252_block-robots.txt-on-all-mfes,-idas,-and-edxappCopy head branch name to clipboard