AI crawlers occasionally have higher coverage than Google Search

For decades, Google has been the undisputed leader in collecting information from the internet and is arguably the most prominent example of a website crawler. These automated programs crawl the internet and record the content they find, allowing search engines to make websites discoverable. But Google Search is now facing competition, as AI systems also require data from the internet, which is why AI companies such as OpenAI are deploying their own bots to search the internet for information.

At the end of August 2025, web hosting provider Hostinger conducted a study of the accessibility of 5 million websites to crawlers. Particularly striking is that, for example, OpenAI's GPT bot reached 4.4 of the 5 million websites, thus achieving greater coverage than the Google bot, which accessed "only" 3.9 million websites. Lesser-known bots such as Ahrefs' SEO crawler, Anthropic's Claude bot, and crawlers from Meta, TikTok, Bing and Apple were also very active, generating a combined total of around 1.4 billion daily requests for the 5 million websites.

The fact that some bots achieve a lower percentage of coverage than others does not mean that they ignore certain parts of the internet. Rather, the programs rotate their targets and thus create an almost complete map of the internet over a longer period of time (but still within a few weeks).

The study also shows that around 80% of queries originate from US tech companies, around 10% from China, with other countries accounting for a negligible proportion. This means that the indexing of the internet is primarily dominated by providers from the US, and within that country by a few large technology companies. As a result, a small number of platforms have a major influence on what content is visible and what responses AI systems generate.