How OpenAI and Perplexity are Undermining Publishers
The Impact of AI Search Engines on Publisher Traffic
Introduction to AI in Search Engines
In recent years, companies like OpenAI and Perplexity have touted their AI-driven search engines as game-changers for publishers, claiming that these platforms would significantly boost referral traffic to news sites and blogs. However, recent findings tell a different story.
According to a report by content licensing platform TollBit, AI search engines are actually sending a staggering 96% less referral traffic to content publishers compared to traditional search engines like Google. This data sheds light on the discrepancies between expectations and reality in the evolving landscape of online content distribution.
Increased Scraping Activity
The report reveals that AI companies have ramped up their web scraping activities, averaging around 2 million scrapes per company in the last quarter of 2024. This study analyzed 160 websites, including major news outlets and consumer blogs, confirming that each page was scraped roughly seven times on average.
Toshit Panigrahi, CEO of TollBit, noted that bots are inundating these sites each time users pose questions to the AI tools. This surge in demand for publisher content is creating challenges for website owners, who often struggle to manage the influx of traffic from bots while losing organic traffic from human users.
Publishers Express Concerns
Several publishers and content-driven companies have voiced their concerns regarding the declining traffic due to AI search engines. For example, Chegg, an education technology company, reported a 49% drop in traffic in January, which they attribute to Google’s use of AI-generated summaries that pulled content from their site without proper attribution. The CEO of Chegg, Nathan Schultz, expressed frustration over what he perceives as a violation of a longstanding agreement between search engines and content creators, where quality content should be linked rather than simply summarized.
Legal Actions Against AI Companies
In response to these issues, various news publishers have started taking legal action against AI firms like OpenAI and Perplexity. These lawsuits typically cite infringement of intellectual property rights, as publishers claim that their content is being reused or summarized without permission.
Legal representatives for Chegg have pointed out that if companies are driven out of business due to AI misuse, it could dumb down the quality of search results on the internet, leading to an "AI slurry" that harms everyone involved.
The Challenge of Identifying Scrapers
AI developers often use "user agents" to crawl web pages for data. However, many of these agents do not properly identify themselves, making it challenging for website operators to monitor how their content is accessed. Olivia Joslin, co-founder of TollBit, mentioned that publishers are hesitant to block Google entirely because it could hurt their search engine optimization (SEO) efforts. This predicament complicates the quest for a balanced relationship between publishers and AI companies.
Referral Traffic from Scrapers
Even when some publishers attempt to restrict access to their sites, platforms like Perplexity continue to send back referral traffic, suggesting they may still be scraping content without detection. For instance, one study showed that while a publisher’s website was scraped 500 times, it received over 10,000 referrals, indicating that AI bots might operate differently than intended.
Economic Solutions for Publishers
To address the financial implications of AI scraping, some organizations have started creating licensing agreements directly with publishers. Established entities such as the Associated Press, Axel Springer, and the Financial Times have engaged in content deals with AI companies like OpenAI.
Furthermore, TollBit has emerged as an innovative solution by charging AI companies each time they scrape content. This model aims to mitigate the adverse effects of content scraping on publishers, providing them with an alternative revenue source.
The Future of AI Search Engines
As AI-driven search engines continue to evolve, their methods of content extraction pose significant challenges to publishers and content creators. The financial burden created by scraping activities, compounded by decreasing referral traffic, suggests a pressing need for actionable solutions that ensure fair compensation and protect the interests of content owners.
