Exclusive: Google confirms with Notebookcheck it blocked benchmarks during Pixel 8, Pixel 8 Pro review embargo period

In an exclusive, Google has confirmed with Notebookcheck that it did indeed block reviewers of the Pixel 8 and Pixel 8 Pro from downloading and installing benchmarks including Geekbench 6 during the review embargo period. Last week we were the first to report that some reviewers found that they couldn’t install benchmark applications through the Google Play Store, but were unaware about the reason it was unavailable for installation.

It had speculated that there was an “API mismatch,” or compatibility issue that might have been to blame for the issue, as a message accompanying next to any search for benchmarks includes a message stating “This app won’t work for your device.” In response to these claims, we reached out the Primate Labs, the markers of Geekbench 6 and received a personal reply from founder John Poole who said that they were unaware of any incompatibility issues. However, as we now know, this has been a deliberate strategy undertaken by Google to stop, if not make it more difficult, for reviewers, to easily install benchmarking apps on Pixel 8 or Pixel 8 Pro review units.

While Google provided Notebookcheck with a statement, it requested that we not quote directly from it, but rather paraphrase it. According to the company, the reason for blocking reviewers from easily installing and running benchmarks on their Pixel 8 and Pixel 8 Pro review units was to help stop benchmark scores and device specs from ‘leaking’. However, at the same time, Google did acknowledge to us that those reviewers who knew how, could still sideload the benchmarking apps and run the tests regardless.

The rationale behind Google’s explanation seems flawed. As we know, Pixel 8 series leaks have been ongoing for the past several months with images of the design itself surfacing back in March. Google then leant into these leaks itself by officially revealing the design of its Pixel 8 and Pixel 8 Pro in late August. As for the specifications of the Tensor G3 chipset, accurate leaks of its core configuration and clock speeds also surfaced back in April and even as far back as September last year. The first early benchmark results appear to have surfaced in June. 

So while Google has no trouble with officially ‘leaking’ the Pixel 8 series designs several weeks before its launch, it does, it seems, have serious reservations about any details of its performance surfacing - even if it is only a week or so before launch. In the comments we received from Google, the company also indicated this is not the first time that it has blocked benchmarks from being installed on its Pixel phones. In fact, for Google, blocking the installation of benchmarks is normal practice.

With reviews only going live on launch day, reviewers and website publishers want to get their reviews out the door at the first available opportunity. Generally, most won’t wait for the embargo period to end before they can then easily download and install benchmarks to run, gather and publish the data. The effect of this is to minimize the spread of this information on the internet, while also helping to keep this information out of the early launch reviews — from the Google hand-picked selection of reviewers given special access to the devices ahead of launch no less.

Google also claimed in the comments provided to us that the block on benchmarks is lifted ‘ahead’ of the consumer availability of the devices. However, we found that we were ourselves unable to install benchmarks in our own unit purchased through a retail store on launch day. This continued for the next 24-48 hours after launch which meant that even reviewers who might have wanted to run the benchmarks before publishing their reviews were still unable to do so. It also stopped the average users who didn’t know how to sideload the benchmarks from posting their results all over social media.

As we now know of course, the benchmark results for the Pixel 8 Pro’s Tensor G3 chip in popular benchmarks like Geekbench 6, graphics tests like the 3D Mark Wild Life Extreme Stress Test and performance in popular games like Genshin Impact is very poor. They pale in comparison to the performance delivered by the A17 Pro found in the iPhone 15 Pro and the Snapdragon 8 Gen 2 found in Android competition such as the Galaxy S23 Ultra.

Google has been at pains to point out that Tensor semi-custom chips are not ostensibly about outright performance, but for performing AI tasks. Instead of focusing on the performance of the Tensor CPUs and GPUs, they have shifted focus to the Tensor’s AI processing accelerator the Tensor Processing Unit or TPU. However, the reality is that all high-end smartphone chips from Apple, Qualcomm and MediaTek are much more powerful at processing AI tasks because the AI performance of a chip also actually depends upon CPU and GPU performance. This isn’t always as apparent in comparisons with other devices, however, because the software on the devices with these more powerful chips don’t often have quite the same level of advanced AI software algorithms that Google’s software engineers specialize in.

Perhaps highlighting the relative shortcomings of the Tensor G3 from an AI performance perspective, @Mrwhosetheboss revealed that most if not all of the new generative AI features introduced via (not ‘on’) the Pixel 8 Pro are in fact processed in the cloud by Google’s powerful AI data centers. During its Pixel 8 and Pixel 8 Pro launch, Google only made it clear that Video Boost was being off-loaded to the cloud for processing. Given how Google repeatedly touted the AI performance of the Tensor G3 during the presentation, it would be easy for viewers to assume the other new generative AI features were being processed by the chip.

Similarly, Google’s product pages for the Pixel 8 and Pixel 8 Pro don’t make any reference at all to the fact that the new generative AI features require a persistent internet connection to work. Not even in the small print. In blocking benchmarking apps, overstating the AI performance of the Tensor G3, and failing to clearly explain when AI processing is taking place on device or in the cloud, Google appears determined to make it difficult for reviewers and average customers to come to a fully informed conclusion about its latest devices.

Update: In response to questions regarding why we did not publish verbatim the statement from Google’s media team, this is because we were advised that it was provided as “background and as additional context.” Google also advised Notebookcheck that its official comment on the issue was “okay to paraphrase, but should not be directly quoted,” which we have abided by.

Purchase the Google Pixel 8 Pro from Amazon starting from $999.