Benchmark gate: Are smartphone manufacturers secretly cheating?

Modern smartphones are fast. They're absolutely blazing fast! But as tech reviewers, my colleagues and I do need to know how fast exactly.

This is also important to many people around the world who don't make a living by testing phones. They just want to see the numbers and get a quantitive representation of that "speed."

That's why benchmarks are quite important not only for professional reviewers but also for all people interested in tech. And when something starts smelling fishy, we should pay attention.

Benchmark gate — what happened?

The first part of the puzzle appeared on April 7th, when NotebookCheck reported that two RedMagic phones had been delisted from the UL Solution's benchmark database. That's the company behind many popular benchmark tools, such as 3DMark, PCMark, and Testdriver.

Our friends at Android Authority picked up on the news and got an official statement from RedMagic a day later. What was the reason for the delisting, which morels got affected by, and what was the official answer? Let's get to it!

What models got delisted from the 3DMark database?

The RedMagic 11 Pro and the RedMagic 11 Pro Plus results got removed from UL Benchmarks' database. The exact benchmarks are 3DMark Sling Shot Extreme (OpenGL ES 3.1), 3DMark Solar Bay Extreme, 3DMark Solar Bay, and 3DMark Steel Nomad Light.

The reason for the delisting posted on UL's site is that "the manufacturer has not complied with our benchmark rules." Part of these rules read,

"Benchmark specific optimizations are not allowed. Additionally, all generic optimizations that change the work specified by the benchmark are prohibited."

And furthermore, "The platform may not detect the launch of the benchmark executable. The platform must not alter, replace or override any parameters or parts of the test, nor modify the usual functioning of the platform based on the detection of the benchmark."

Was RedMagic cheating, and how did they do it?

The Japanese-language Saityo YouTube channel found that the RedMagic 11 Pro phone had much higher scores when running the standard benchmark app compared to a stealth/disguised version.

This suggested that RedMagic detects the benchmark software and optimizes the hardware of the phone that's being tested in order to achieve the highest possible score. Which is in violation of the UL Solutions benchmark rules quoted in the previous paragraph.

RedMagic's response

Shortly after Android Authority published the news, RedMagic got in contact and addressed the cheating allegations. Here's the full response:



It's true that many phones have such systems that detect the load on the CPU and GPU and put the phone in high-performance mode to offset the additional strain and offer smooth performance.

These app-specific tweaks and enhancements are clearly in violation of UL's rules, hence the delisting. Is it really cheating, though?

RedMagic's delisting is not a black and white story

On one hand, optimizing a smartphone to score high in benchmarks is clearly giving companies advantage over their immediate competitors. But it's not that simple.

If RedMagic phones do this exact same optimization when running a heavy 3D game to give you better performance, then the higher 3DMark scores are kind of justified. You are getting better performance at the end of the day.

The grey area is that the phone does it automatically. If it were just a manual mode you'd have to toggle yourself, before you start gaming or before you run a benchmark for that matter, it probably would not have been considered cheating.

My personal take

It's a delicate area. I can offer some insight from our time testing the RedMagic 11 Pro. The phone did score pretty high in the tests we use at PhoneArena, but it has also been performing very smoothly in real-world scenarios, gaming, and other heavy applications.

I've been using this phone as a daily driver for more than six months now and can't fault it. You make out of all this what you will.