Skip to content

Commit 29b52ce

Browse files
author
Ajay kumar
committed
[Tech] [Ajay]: Add robots.txt to manage web crawler access
1 parent 38fbec5 commit 29b52ce

File tree

1 file changed

+6
-0
lines changed

1 file changed

+6
-0
lines changed

TestArena/wwwroot/robots.txt

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
User-agent: *
2+
Allow: /
3+
4+
User-agent: GPTBot
5+
Disallow: /admin
6+
Sitemap: https://devcodex.in/sitemap.xml

0 commit comments

Comments
 (0)