Principled Technologies responds to criticism of its i9 9900k study

Although critics Steve Burke and Steve Walton have called its study deeply flawed, hardware validator Principled Technologies has put forth a defense for its testing methodology when testing Intel's newest CPU the i9 9900k. Burke interviewed Bill Catchings, co-founder of Principled Technologies, while the company itself made a public statement to criticism such as Walton's video "Intel's New Low: Commissioning Misleading Core i9-9900K Benchmarks."

Burke's interview with co-founder Bill Catchings primarily focused on the testing methodology and aimed to answer questions about certain curious aspects of the study. When asked about the cooling arrangement of the CPUs, Principled Technologies argues that since the 2700X was the only CPU to come with a cooler, it was necessary to procure other coolers for the 7 other CPUs. Burke points out that the Noctua cooler used with the 7 other CPUs is far better for cooling performance, as it has a bigger heatsink, more optimal placement, and more copper heatpipes. Burke says AMD was "irresponsibly punished by including a cooler in its packaging."

Burke also questioned the use of 64 GB of memory, which Catchings described as "normal." According to Steam's Hardware Survey, nearly 40% of users use 8 GB of RAM; even normalizing for higher end, DIY systems, it seems far more likely that 16 GB of RAM is closer to the norm, with 31% of Steam users having that amount and less than 4% using any amount more. For extensive testing, Burke later recommended 32 rather than 64 GB.  They also defended the use of Ryzen's "Game Mode," which disabled half of the cores on the 2700X, saying it improved Threadripper's performance "across the board." Catchings admitted he wasn't sure what settings the typical gamer ran with on Ryzen CPUs; Burke informed him that usually gamers usually don't even install Ryzen Master at all, let alone enable "Game Mode."

Later, the company made a public statement about its testing methodology, of which they directly notified Hardware Unboxed. Principled Technologies made similar remarks in its public document as co-founder Catchings made in the interview. The usage of "Game Mode" was again defended; the company's rationale was that in order to keep results reliable, it had to be enabled on all AMD platforms and that the company expected better results from the 2700X since the Threadripper processors benefited from the setting. Principled Technologies also expressed its confidence in the study, saying:

Because our goal is always to do the right thing and get the answers right, we are currently doing additional testing. We will share that data and will certainly call out if something is significantly different from what we've already published. We are confident in our test methodology and results. We welcome questions and we are doing our best to respond to questions from our interim report, but doing so takes time. We will add responses if other issues come up. Thanks for listening.

Walton says the testing methodology Principled Technologies used "don't come close to meeting the standards used by respectable media outlets in 2018, not to mention they don't come close to representing the configurations used by the vast majority of gamers." The reviewer then stated Principled Technologies's updated results without using "Game Mode" on the 2700X should improve performance significantly. Walton reiterated his criticism of Intel in facilitating this study and believes most of the blame rests on the processor company, saying it is likely that Principled Technologies used "Game Mode" at Intel's behest.

Source(s)