2026-07-01
For our second Content Independence Day, we’re giving website owners finer options to manage AI traffic. Instead of a one-size-fits-all block, all customers can now easily distinguish and manage Search, Agent, and Training bots, alongside the new ability to protect ad-monetized pages....
Continue reading »
2022-08-12
Cloudflare is uniquely positioned to help give crawlers hints about when they should recrawl, if new content has been added, or if content on a site has recently changed...
2021-12-16
Cloudflare Is reducing the environmental impact of web searches with 20+ billions crawler hints delivered so far. This blog describes the technical solution of how we built the Crawler Hints system that makes all this possible. ...
2021-10-18
Crawler Hints now supports IndexNow, a new protocol that allows websites to notify search engines whenever content on their website content is created, updated, or deleted. ...
2021-07-28
Crawler Hints provide high quality data to search engine crawlers on when content has been changed on sites using Cloudflare, allowing them to precisely time their crawling, avoid wasteful crawls, and generally reduce resource consumption of customer origins....
