Cloudflare will now block AI web scrapers that don't pay for data

Declaring "Content Independence Day," Cloudflare CEO Matthew Prince announced massive changes to the company's web service infrastructure, which will now block all AI web crawler bots by default. 

In a blog post, Prince elaborated that the modern web search landscape is now increasingly defined by AI chatbots such as Google's Gemini or OpenAI's ChatGPT. While these are useful, they also freely scrape available data online without any repercussions and fail to compensate the original creators. 

Prince argued that due to recent changes with Google Search, it was now ten times "more difficult for a content creator to get the same volume of traffic" as they did ten years ago. 

"Instead of being a fair trade," said Prince. "The web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value."

Prince said the scraped content "is the fuel that powers AI engines," and it was only fair that the original creators get compensated for it. 

Cloudflare also announced plans for a marketplace that will bring creators and AI companies together in one place. The marketplace will score available content on not only the traffic it generates but "on how much it furthers knowledge." Prince believes this will help AI engines get faster and "potentially facilitate a new golden age of high-value content creation."  

Prince admitted he doesn't have all the answers yet, but the company was working with "leading computer scientists and economists to figure them out." 

Recently, SourceHut, an open-source git-hosting service, said they experienced disruptions due "to aggressive LLM crawlers" and blocked several cloud providers, including Google Cloud and Microsoft Azure, due to high volumes of incoming traffic from their networks. 

In January, DoubleVerify, a platform for web analytics, reported an 86% increase in General Invalid Traffic (GIVT) from AI scrappers and other automated tools online compared to 2024. 

Despite earlier promises, OpenAI's GPTbot has also found ways to ignore or bypass a site's robot.txt entirely, causing a massive surge in traffic for domain owners and expensive bills.