In a groundbreaking move that has sent shockwaves throughout the digital landscape, OpenAI has introduced its latest innovation – the Web Crawling GPTBot this week designed to collect website content for training their large language models (LLMs). This cutting-edge technology is set to revolutionize the way search engines index and understand web content. However, this advancement has also led to a brewing conflict between website owners and creators who are now facing a new challenge in ensuring their online presence remains intact.
Once the news about the bot spread, a wave of resistance emerged. Website owners and creators rapidly shared advice on how to prevent GPTBot from gathering data from their sites.
OpenAI also unveiled a support page for GPTBot, along with a method to prevent the service from scraping a website. By making a minor adjustment to a website’s robots.txt file, the flow of content to OpenAI could be halted. Nevertheless, considering the extensive nature of web scraping in general, it remains uncertain whether solely blocking GPTBot will entirely prevent the incorporation of website content into LLM training data.
“We gather publicly available internet data at intervals to enhance the potential, precision, and security of upcoming models,” conveyed an OpenAI representative via email. “We furnish guidance on preventing our collection bot from reaching a site on our platform. Our filtering process excludes web pages with paywalls, those that collect personal identifiable information (PII), and content that contravenes our guidelines.”
Websites raise their defenses
News platforms like The Verge have already implemented the robots.txt directive to prevent the OpenAI model from extracting content for inclusion in its LLMs. Casey Newton, through his Substack newsletter, Platformer, sought input from readers on whether to disallow OpenAI’s content collection. Neil Clarke, the editor of the science fiction magazine Clarkesworld, declared on X (formerly known as Twitter) that it would block GPTBot.
Shortly after GPTBot’s public launch, OpenAI unveiled a grant of $395,000 and a collaboration with New York University’s Arthur L. Carter Journalism Institute. Headed by former Reuters editor-in-chief Stephen Adler, NYU’s Ethics and Journalism Initiative aims to assist students in developing ethical and responsible approaches to incorporating AI in the field of news.
Tom Rubin, OpenAI’s Chief of Intellectual Property and Content, expressed enthusiasm for the potential of the Ethics and Journalism Initiative, highlighting its objective of addressing various challenges journalists encounter when striving for ethical and responsible practices, particularly in the realm of AI implementation. However, the release made no mention of public web scraping or the associated controversy.
Although having greater control over content usage on the open internet is beneficial, it remains uncertain how effective it would be to simply block GPTBot in completely preventing LLMs from assimilating content not protected by paywalls. LLMs and similar generative AI platforms have already employed extensive collections of publicly available data to train the datasets they currently utilize.