@rsiddle wrote a JSON schema in this blogpost. @merj also publishes a list of web crawlers linking to the corresponding crawlerips.json.

I merged the list from Merj with another one at hand. I looks like many crawlers already follow the JSON schema when publishing their IP addresses:

Name	Size (kiB)	IP Address Location(s)	References
Ahrefs	3	crawler-ip-ranges (schema validation failed: no creationTime)	What is the list of your IP ranges?
	377	crawler-ips (different schema: single IPs, not prefixes)
Amazon		(verification via reverse DNS)	About Amazonbot
Apple	1	applebot.json	About Applebot
Baidu		(verification via reverse DNS)	FAQs of Baiduspider
Bing	2	bingbot.json	Verify Bingbot
Common Crawl	1	ccbot.json	CCBot
DuckDuckGo	16	duckduckbot.json	Is DuckDuckBot related to DuckDuckGo?
	16	duckassistbot.json	Is DuckAssistBot related to DuckDuckGo?
Google	20	googlebot.json	Verifying Googlebot and other Google crawlers
	18	special-crawlers.json
	60	user-triggered-fetchers.json
	30	user-triggered-fetchers-google.json
Majestic		("IP range distributed, worldwide)	About MJ12Bot
Meta		(verification via routing / IP ranges published as CSV)	Meta Web Crawlers
Mistral	1	mistralai-user-ips.json	Mistral AI Crawlers
Naver	2	naverbot.json	https://searchadvisor.naver.com/guide/seo-basic-firewall
OpenAI	2	searchbot.json	Overview of OpenAI Crawlers
	4	chatgpt-user.json
	1	gptbot.json
Perplexity	1	perplexitybot.json	Perplexity Crawlers
	1	perplexity-user.json
Pinterest		(IP range published not as JSON)	What does Pinterestbot do?
Semrush		(no verification documented)	SemrushBot
Yandex		(verification via reverse DNS)	How to check that a robot belongs to Yandex

Please feel free to update and complete this table!

Standardize or figure out the existing format used for publishing IP ranges #4

Description