Website owners make money from human views, so AI crawlers need to pay their fair share, too.
(Image credit: Getty Images / Sundry Photography)
Cloudflare CEO Matthew Prince said that his company has blocked over 416 billion AI bot requests since making it the default option in July of this year after it announced the Content Independence Day initiative. Prince said in an interview with Wired that this feature allows website owners to block AI crawlers by default, unless the AI company pays them to gain access to their content.
“The business model of the internet has always been to generate content that drive traffic and then sell either things, subscriptions, or ads, Prince told Wired. “What I think people don’t realize, though, is that AI is a platform shift. The business model of the internet is about to change dramatically. I don’t know what it’s going to change to, but it’s what I’m spending almost every waking hour thinking about.”
While Cloudflare blocks almost all AI crawlers, there’s one particular bot it cannot block without affecting its customers’ online presence — Google. The search giant combined its search and AI crawler into one, meaning users who opt out of Google’s AI crawler won't be indexed in Google search results. “You can’t opt out of one without opting out of both, which is a real challenge — it’s crazy,” Prince continued. “It shouldn’t be that you can use your monopoly position of yesterday in order to leverage and have a monopoly position in the market of tomorrow.”
Human-generated content is crucial for AI companies to train their models on, as research has proven that AI models turn to slop when trained on AI-generated data. AI summaries have been proven to reduce traffic on websites — especially impacting those that heavily rely on visibility and views for ad revenue — but licensing deals can help offset this, helping online publications remain a viable source of income for creators and publishers.
Cloudflare will also benefit from a varied internet that hosts content from real humans. Its CEO told Wired that the company is aiming for a future where creators and businesses grow on a level playing field, as there would be more websites that need protection, resulting in more potential clients for Cloudflare. This makes it one of the biggest content delivery networks in the world, owning 79.9% of the market as of 2022. However, this also makes the internet vulnerable, like when a single misconfigured file knocked out a huge chunk of the web in November.
This highlights the current problem we have with the global web infrastructure, which relies on just a few big companies — AWS, Azure, Cloudflare, CrowdStrike, and Google, to name a few — to serve the entire planet. While these institutions have made things easier and more streamlined for corporations that rely on the internet, it also means that just one of these services going down will cost billions in losses and severe disruption across the globe.
Jowi Morales is a tech enthusiast with years of experience working in the industry. He’s been writing with several tech publications since 2021, where he’s been interested in tech hardware and consumer electronics.
