Perplexity wants to change the way we use the Internet, but the AI search startup backed by Jeff Bezos You could be breaking their rules by doing so. The company appears to be ignoring a widely accepted web standard, the Robot Exclusion Protocol, to remove parts of the web that operators don’t want robots to access, according to a report from developer Robb Knight this week that was confirmed by Wired.
Why your clothes shrink when you wash them (and how you can avoid it)
Perplexity’s service summarizes articles on the web and claims to offer “reliable answers” without the need to click on different links, as noted. in a blog post. To do that, Wired and Knight discovered that Perplexity ignores code (robots_files.txt) deliberately written to block web trackers. The two reports found that Perplexity uses an unlisted IP address to circumnavigate these robots.txt files and scrape the websites in detail anyway. Wired claims that its website blocked the Perplexity web crawler in early 2024, but the AI search engine is still able to summarize its articles in detail.
Despite this, Perplexity claims to respect the Robot Exclusion Protocol in documentation on their website. Perplexity CEO Aravind Srinivas told Wired that reporters had “a deep and fundamental misunderstanding of how Perplexity and the Internet work.” ”, But he did not dispute the findings directly. Gizmodo reached out to Perplexity for a more detailed response and will update the article if we receive a response.
On the other hand, Perplexity is currently facing legal threats for violating other Internet rules: copyright infringement. Forbes Reportedly threat of legal action against Perplexity this week, after accusing the AI startup of ripping off Forbes reports without proper attribution. Forbes had done original reporting on Former Google CEO Eric Schmidt’s AI Drone Adventure, and Perplexity created AI-generated articles, podcasts, and videos using text and images from Forbes. Forbes executive editor called Perplexity on X earlier this month.
Perplexity’s product, while useful, redirects traffic on the Internet. Google also indexes web pages and provides brief AI summaries, but points traffic directly to the web pages from which the information comes. The puzzle in practice is to write detailed articles about AI, so that users do not click on websites, which breaks the business model of digital media.
OpenAI has forged partnerships with media companies To address this, pay them upfront to license content, and Perplexity is supposedly working in associations of similar content, but instead of paying a flat fee for content like OpenAI, Perplexity aimed to share the revenue. But these associations do not yet exist. So for now, Perplexity appears to be jumping paywalls and scraping websites to grab all the data it needs to power its AI responses.
This content has been automatically translated from the original material. Due to the nuances of machine translation, there may be slight differences. For the original version, click here.
