Exclusive: Geekbench founder responds to Pixel 8 benchmark blocking controversy

Notebookcheck recently broke the story about the fact that Google appears to have blocked writers reviewing its new Pixel 8 and Pixel 8 Pro devices from being able to easily install popular benchmark apps like Geekbench 6 and 3D Mark during the embargo period. It was a highly unusual move and raised questions about Google’s motivations for doing so. Google has made about the capabilities of the Tensor G3 chip at the heart of its latest Pixels, in particular its AI performance. However, in being unable to install the app, many reviewers were unable to test Google’s claims by running the benchmarks, as is standard practice for most reviews as it provides an objective and standardized way of assessing chip performance.

One of the principal benchmarks used by reviewers is Geekbench. It is a popular cross-platform benchmarking utility that is primarily used to assess CPU performance, although it also has a GPU compute suite too. The test assesses both a chip’s single-core performance and its multi-core performance, and is it is also a “cross-platform” test. This means the results it produces can be compared across different devices like mobile phones and laptops, operating systems like Android and Windows, and across chips made by the likes of Apple, Intel, Qualcomm, Google, Samsung, MediaTek, and AMD.

When Geekbench was recently updated to version 6, it added the ability to test new application areas including AR performance and also putting a greater emphasis on machine learning, which powers a chip’s AI potential. It is also used and endorsed by companies including Samsung, Microsoft, AMD, Dell, HP, LG, MediaTek and more. As such, you might think that given the way Google touts the AI performance of its Tensor chipsets, this would be a benchmark that Google would be happy for reviewers to install on their Pixel 8 and Pixel 8 Pro units. There were reviewers with the requisite knowledge to by-pass Google’s Play Store block by simply downloading the app package for Geekbench 6 and installing it manually using a process known as side-loading.

The results Geekbench 6 produced by the Tensor G3 were unflattering - both during the review embargo period and following. Some Pixel fans have posted theories that there was a compatability issue with the benchmarks and Android 14, and that Google hadn’t deliberately blocked their easy installation. This however fails to explain why the apps could be sideloaded and could be run successfully, or why the results pre- and post-block were on no different. We have reached out to Google for comment, but have not heard back at the time of writing. We also reached out to Primate Labs, the makers of Geekbench 6, which was able to provide us with a response.

In reaching out to Primate Labs, we asked whether they were aware that of any compatibility issues that would have prevented Geekbench 6 from being installed on the Pixel 8 and Pixel 8 Pro during the review embargo period. John Poole, creator of the original verison of Geekbench and company founder provided Notebookcheck with this response:

We do not know why Geekbench 6 was unavailable on Pixel 8 devices (although we suspect it had something to do with embargoes). We are not aware of any compatibility issues with Geekbench 6 on Pixel 8 or Pixel 8 Pro.

Following our coverage of this issue, Arun Maini, a YouTuber better known by his handle of @Mrwhosetheboss revealed that key new AI-powered features that incorporate generative AI techniques including Magic Editor, AI Wallpaper and Best Take require a persistent internet connection to function as advertised. During Google’s launch presentation, it implied that these functions were being processed by the Tensor G3 chip. This is a message reinforced on the official Google Blog extolling the virtues of the chip and positioning it as being “AI-first.” 

In the blog, Google VP of Product Management asserts the following:

Our work with Tensor has never been about speeds and feeds, or traditional performance metrics. It’s about pushing the mobile computing experience forward. And in our new Tensor G3 chip, every major subsystem has been upgraded, paving the way for on-device generative AI. It includes the latest generation of ARM CPUs, an upgraded GPU, new ISP and Imaging DSP and our next-gen TPU, which was custom-designed to run Google’s AI models.

We tested Maini’s claims regarding the requirement for a persistent internet connection to make some of these new generative AI-functions work. Despite Google’s claims about the ability for the Tensor G3 to power on-device generative AI, Google is in fact off-loading the processing requirements for these functions to the cloud for processing by its much more powerful servers - even if some of the processing may initially be done on device. However, given the relatively poor performance of the Google Tensor G3 in benchmarks such as Geekbench 6, it is clear why Google has had to resort to off-loading the processing.

As chip designer Arm explains on its website (keeping in mind that the Tensor G3 utilizes Arm-designed CPU cores), its entire architecture is designed with AI performance in mind, and that machine learning algorithms are performed across the entire chip. This means that if a chip does not perform well in CPU and GPU benchmarks, it is highly unlikely to be demonstrate strong AI performance. This is because machine learning tasks are handled by the CPU and GPU in conjunction with the AI accelerator on board the SoC known as the Neural Processing Unit (NPU) -- or, as Google calls it, the Tensor Processing Unit (TPU). You will often read on forums and chats that “benchmarks don’t matter,” it is how the device performs “in the real world” that matters. And to some extent, there is truth to that view - benchmarks are one part of the overall equation.

However, there is no escaping that raw processing power is also a necessity for a device that is being promoted as being AI-first. Generative AI clearly requires more processing power than the Tensor G3 is able to deliver, despite Google’s claims to the contrary – otherwise, there would be no requirement for a persistent internet connection and off-loading these tasks to the cloud for processing before the result is then returned to the user’s device. It would be processed on board. It appears that Google’s AI-powered software algorithms have hit their limit with the Tensor G3’s ability to handle them locally, and Google’s narrative regarding the unimportance of “traditional performance metrics” has come unstuck. 

Update: Article updated with screenshots from Google's Pixel 8 Pro product page with screenshots of Google descibing advanced new AI features including the generative AI features mentioned in the article. A screenshot is also included of the small print footnote reference '7' that explains the requirements and limitations. Note, no reference is made by Google to the requirement for a persistent internet connection for the features to work, or that the content needs to be off-loaded to Google's servers in the cloud for AI processing.

Purchase the Google Pixel 8 Pro from Amazon starting from $999.