Is the era of the open Internet over?

Reddit has apparently blocked numerous search engines to prevent them from displaying content from the platform. Google seems to be the only major exception. The reason for this is probably that the two companies have signed a licensing deal worth millions.

Reddit began blocking numerous search engines and their web crawlers in early July 2024. This is according to a report by the online magazine 404 Media According to this, Google is currently one of the few search engines that displays current Reddit content.

Reddit blocks search engines – because they don’t pay?

The background: Google had concluded a deal with Reddit in early 2024. For around 60 million US dollars, the company secured exclusive access to the content of the Internet forum in order to be able to train its in-house AI systems.

But Google now seems to be the only search engine that shows current results from Reddit in its search. For example, if you enter “site:reddit.com” on Bing or DuckDuckGo and select “last week” as the date in the search settings, you will not receive any search results.

A Reddit spokesperson denied this to the tech magazine The Verge officially a connection with the Google deal, but indirectly confirmed it:

This is unrelated to our recent partnership with Google. We have been in discussions with several search engines. We have not been able to reach agreements with all of them because some are unable or unwilling to make enforceable commitments regarding the use of Reddit content, including its use for AI.

Microsoft confirms: Reddit blocks search engines

In concrete terms, Reddit not only demands money for other companies to be able to train their AI systems with the content of the Internet forum, but also apparently wants to be paid for basic access – for example via search engines.

To prevent Bing, DuckDuckGo and Co. from accessing Reddit, the social news aggregator updated its robots.txt file in early July 2024 to prevent search engines from crawling. Ben Lee, Reddit's Chief Legal Officer, said The Verge in an earlier report: “It sends a signal to those who don’t have an agreement with us that they shouldn’t access Reddit data.”

Microsoft spokeswoman Caitlin Roulston said: “Microsoft respects the robots.txt standard and we respect the instructions of websites that do not want content on their pages to be used with our generative AI models. She also confirmed that Bing stopped crawling Reddit after the platform updated its robots.txt file.

Reddit's decision to block some of the most popular search engines seems drastic, but strictly speaking, it is not really surprising. The platform had already started protecting its data more and more strictly last year in order to generate additional sources of income and attract investors.

Is the era of the open Internet over?

Reddit reportedly even threatened to deny Google access to its content if the company did not stop using the data to train its AI systems for free. As more and more AI content floods the internet, human content like that on Reddit is likely to become more important in the future.

Many users add the name “Reddit” to their search queries in order to receive specific answers from people. But in the future, this seems to only be possible on Google. Reddit itself has thus acquired an extremely powerful position.

However, licensing deals such as those with Google also mean that the Internet becomes more closed when platforms such as Reddit close themselves off to other search engines. Since forums are generally becoming more important, this trend could continue and also expand to the social media sector.

A few years ago, such a development would have been unthinkable. Search engines primarily provided website operators with traffic and thus indirectly also advertising revenue. However, due to AI and spam content, search engines and the Internet have been getting worse for some time.

The key question is therefore not only how search engines will deal with Reddit's move, but also whether it will be copied – not to mention how to deal with AI spam, which even Google cannot get under control.

Also interesting: