Last week, Perplexity CEO Aravind Srinivas responded to WIRED’s investigation by first stating that the questions we asked the company “reflect a deep and fundamental misunderstanding of how Perplexity and the internet work.” Srinivas then told Fast Company that the secret IP address WIRED observed scraping Condé Nast’s websites and a test page we created was operated by a third-party company that performs web crawling and indexing services. He declined to name it, citing a confidentiality agreement. When asked if he would ask the third party to stop crawling WIRED, Srinivas replied, “It’s complicated.”
Perplexity spokeswoman Sara Platnick tells WIRED that the company responded to Amazon’s questions on Wednesday and called the investigation standard procedure. Platnick says Perplexity has not made any changes to its operations in response to Amazon’s concerns.
“Our PerplexityBot, which runs on AWS, respects robots.txt, and we have confirmed that the services controlled by Perplexity do not crawl in any way that violates the AWS Terms of Service,” Platnick says. He adds, however, that PerplexityBot will ignore robots.txt when a user enters a specific URL in their query, a use case Platnick describes as “very rare.”
“When a user enters a specific URL, that doesn’t trigger crawling behavior,” Platnick notes. “The agent acts on behalf of the user to retrieve the URL. It works the same as if the user themselves went to a page, copied the text of the article, and then pasted it into the system.”
This description of Perplexity’s functionality confirms WIRED’s findings that its chatbot ignore robots.txt in certain cases.
Digital Content Next is a trade association for the digital content sector that counts among its members The New York Times, The Washington Post and Condé Nast. Last year, the organization shared draft principles to govern generative AI to prevent potential copyright infringements. Jason Kint, its CEO, tells WIRED that if the accusations against Perplexity are true, the company is violating many of those principles.
“By default, AI companies must assume that they do not have the right to take and reuse publishers’ content without permission,” Kint emphasizes. If Perplexity ignores the terms of service or robots.txt, she adds, “it should raise alarm bells that something improper is happening.”
Article originally published in WIRED. Adapted by Andrei Osornio.
How Perplexity was born, the powerful AI search engine that wants Google’s business
Aravind Srinivas grew up in the same city as Sundar Pichai and developed an obsession with the company long before launching his own startup AI search.
