Skip to content

Update robots.txt #162

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
github-actions wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update robots.txt #162

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion public/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ User-agent: ICC-Crawler
User-agent: IndeedJobBot
User-agent: InternetArchiveBot
User-agent: Jagged Pixel UptimeBot
User-agent: Jetpack
User-agent: Kangaroo Bot
User-agent: Kinsta
User-agent: KlaviyoAIBot
Expand All @@ -94,6 +95,7 @@ User-agent: Manus Bot
User-agent: MediaMonitoringBot
User-agent: Medialogia Bot
User-agent: Mediatoolkitbot
User-agent: Mediavine Medatada Parser
User-agent: Meta-ExternalAds
User-agent: Meta-ExternalAgent
User-agent: MirrorWebCrawler
Expand All @@ -116,11 +118,13 @@ User-agent: Parse.ly
User-agent: Perplexity-User
User-agent: PerplexityBot
User-agent: PetalBot
User-agent: PhindBot
User-agent: PingPing Bot
User-agent: Pingdom
User-agent: PinterestBot
User-agent: Poggio-Citations
User-agent: Productsup Crawler
User-agent: Proximic
User-agent: Pulsetic
User-agent: QualifiedBot
User-agent: Quantcastbot
Expand All @@ -129,18 +133,20 @@ User-agent: Reelevant
User-agent: SBIntuitionsBot
User-agent: SE Ranking Bot
User-agent: SalesViewerBot
User-agent: SemrushBot-OCOB
User-agent: Sentry Uptime Monitoring
User-agent: Site24x7
User-agent: SiteUpTimeBot
User-agent: Slickstream
User-agent: SolarWinds Observability
User-agent: Spider
User-agent: Splunk
User-agent: Splunk Synthetics
User-agent: StatsDroneBot
User-agent: StatusCake
User-agent: Synthetic Bot
User-agent: TTD Content
User-agent: Taboola
User-agent: TavilyBot
User-agent: Thousand Eyes Cloud Agent
User-agent: Timpibot
User-agent: Trendiction Bot
Expand Down