Qualcomm responds to benchmark cheating allegations — Snapdragon X Elite/Plus benchmarks claimed to be fraudulent (Updated)

Qualcomm's HQ sign
(Image credit: Qualcomm)

The notoriously confrontational tech site SemiAccurate claims that Qualcomm is cheating on the benchmarks of its new Snapdragon X Elite and Plus laptop processors, and Qualcomm has now responded to those accusations. The Snapdragon X Plus, announced yesterday, joins the previously announced X Elite in Qualcomm's upcoming processor lineup for Windows notebooks. The chips stand out as Qualcomm's first potentially competitive laptop processor against Apple, Intel, and AMD, and the best chance for Windows on Arm to take off. However, SemiAccurate.com claims that Qualcomm has been far from forthcoming with its curated benchmark results that have been presented to the press. 

SemiAccurate, claiming two "major" OEMs and a "deep source at Qualcomm" for their benchmark cheating claims, insists that Qualcomm benchmarks are impossible to recreate even by class-leading OEM developers.

A Qualcomm representative sent Tom's Hardware an official comment on the matter, saying succinctly, "We stand behind our performance claims and are excited for consumers to get their hands on Snapdragon X Elite and X Plus devices soon.”

Back at Qualcomm's Snapdragon Summit conference held last October, the public saw the first benchmarks of the forthcoming X Elite processors. These numbers, claimed to be clean by SemiAccurate's sources, were slower than expected, though the Qualcomm source claimed the software wasn't final. And those benchmarks were a complete black box — no one at the conference could see the settings used for testing, and vague charts showing performance against competitors Apple and Intel were unverifiable. 

After OEMs got their hands on the X Series chips, SemiAccurate claims no OEM has been able to replicate Qualcomm's touted test numbers. In fact, the site reports OEMs first saw numbers "far sub-50%" of the numbers Qualcomm claimed. Qualcomm engineers reportedly told SemiAccurate that they blamed poor optimization from Windows running on Arm, and poor cooling. Even after some time had passed and cooling could be improved, however, testers still reportedly never came close to Qualcomm's curated benchmarks, with one anonymous source comparing the Snapdragon X Elite to Intel Celeron chips. 

SemiAccurate makes no claims about specifics of how Qualcomm is allegedly cheating its benchmark numbers. If "tier 1" OEMs are reportedly unable to even come close to recreating the X Series benchmarks on their own, wrongdoing by Qualcomm, the site alleges, must be the only other option.

The site also claims that Qualcomm's product slides are presenting "tainted" numbers to the press and its OEM partners. At the Snapdragon Summit, the company avoided answering technical questions on its forthcoming X Series, promising more answers and deep technical briefings before launching its chips. It also apparently promised reviewers the opportunity and ability to independently benchmark the new chips before launch; neither of these promises came to be, though there's still time before the chips launch in mid-2024. (The press could run benchmarks on devices at hands-on events, but the software was all pre-installed in tightly controlled conditions.)

SemiAccurate says Qualcomm's stats, revealed today, on its new line of chips lacked the deep technical elements originally promised, with more time spent on potential camera specs enabled by the SoC processors than any real technical stats or deep analysis. We did get core counts, cache, and clock speeds, but nothing deeply technical outside of spec sheets.

If the claims prove true, it would certainly rock a laptop world that looked to the X Series as Windows' response to Apple's M Series processors, but for now, these claims are unsubstantiated. Naturally, we can't test the veracity of these huge claims until the silicon comes to market.

Microsoft is expected to make big announcements in this space at its Build developer conference in May, hopefully giving us a better idea of the shape of Windows on Arm.

Edit 04/25/2024 3:30am PT: Added comment from Qualcomm.

Dallin Grimm
Contributing Writer

Dallin Grimm is a contributing writer for Tom's Hardware. He has been building and breaking computers since 2017, serving as the resident youngster at Tom's. From APUs to RGB, Dallin has a handle on all the latest tech news. 

  • ThomasKinsley
    I read the article by SemiAccurate. It can be distilled (by me, not AI) into three points:
    Qualcomm has not been forthcoming with detailed specs of their chip.
    OEMs reported poor performance with samples - sometimes 50% slower; other times much more.
    An insider at Qualcomm claims they cooked the benchmarking, yet at the same time by manipulating the benchmarks they made the chip look worse in other areas than it really is.On the first two points SemiAccurate states that Qualcomm worked with them months ago. They provided more information on the chip and blamed Windows on ARM for being unoptimized.

    I'm trying to see every side here. It could be Qualcomm is maliciously lying about their benchmarks, but if it's really as slow as a Celeron (as one OEM stated), then that will be discovered very quickly by tech journalists.

    What if Qualcomm is struggling to get the chip to play nicely with Windows 11 on Arm? That would make sense why Qualcomm is delaying the benchmarks and test samples to journalists. Nobody at Qualcomm wants to hear, "The Snapdragon X Elite runs like trash! What a loser!" when they believe it's MS's fault. Even if MS fixes their OS after launch, everyone will still think of the SD X Elite as junk and the initial image can't be fixed. That might explain it all.
    Reply
  • Bruhme2985
    I checked the article tonshardware linked and man did it sound incredibly petty and childish. The website for one doesn't look even remotely professional and the article is just a dude ranting that he didn't get an inside peek at it.

    A lot of things including the fact that the dude messed things up in the article (for instance snapdragon doesn't have an x Pro chipset), shows that it is more of a rant then anything special, the dude wasn't even bothered to reread his own rant.

    Honestly his article imo has very little weight to it, even if it was true his article is too unprofessional and salty. It makes sense why snapdragon doesn't want to talk to you, you sound too bitter.
    Reply
  • bcweir
    Wait, you call the author's remarks petty and childish, yet refer to the other person as 'dude,' undercutting your own argument to treat people with respect. Unless you're either a Valley Girl or a professional surfer, try following your own rules before calling out someone else for the same sins you're committing
    Reply
  • bit_user
    Bruhme2985 said:
    I checked the article tonshardware linked and man did it sound incredibly petty and childish. The website for one doesn't look even remotely professional and the article is just a dude ranting that he didn't get an inside peek at it.
    I can confirm that SemiAccurate (mostly the work of Charlie Demerjian) has fallen far from whatever grace it had. The articles try way too hard to strike a clever and informal tone, while self-congratulating and calling out every slight the author believes he received, at every possible opportunity. It turns them into rants that I find virtually unreadable. Add to that the jealous paywall-guarding of the few actual nuggets of information the author managed to collect and the entire site is basically just a frustrating waste of time.

    I really don't know why anyone in the industry still talks to Charlie, at this point. It's probably just a shrinking number of disgruntled tech workers trying to settle a score, which makes his information very subject to exaggeration and mischaracterization.

    If you want to see what SemiAccurate could have been, head on over to SemiAnalysis:
    https://www.semianalysis.com/
    Reply
  • bit_user
    Oh, and as for the allegations, I'm probably not as bothered by them as I should be. I try to disregard both leaks and curated benchmark scores, instead just waiting until review samples get into the hands of independent tech journalists.

    That said, even the best review I saw of the Thinkpad X13S, which I think is probably the leading implementation of the Snapdragon 8cx Gen 3 we've seen to date, is still quite lacking in benchmarks of native apps, even where they exist (i.e. web, productivity, etc.). So, I do understand Qualcomm's concerns about how it will be treated by the tech press and influencers looking to score points with their audience at its expense.
    Reply
  • Alvar "Miles" Udell
    You mean a first party manufacturer is presenting numbers that were probably obtained in a very specific way on order to maximize the performance numbers for publicity? That NEVER happens with AMD, Intel, nVidia, or any other company...
    Reply
  • JamesJones44
    Assuming everything SemiAccurate said ends up being true, the minute independent benchmarks come out Qualcomm would be in big trouble. Maybe they are crazy enough to try, but that seems like the fastest way to have your product killed and lawsuits thrown at your company.

    Guess we'll find out in the July timeframe.
    Reply
  • Bruhme2985
    bcweir said:
    Wait, you call the author's remarks petty and childish, yet refer to the other person as 'dude,' undercutting your own argument to treat people with respect. Unless you're either a Valley Girl or a professional surfer, try following your own rules before calling out someone else for the same sins you're committing


    I don't know why you are getting triggered over the word dude. It's informal English, and the guy is a dude a male. I am a dude, a male human. How is dude derogatory, it really isn't. :unsure:
    Reply
  • Bruhme2985
    bit_user said:
    I can confirm that SemiAccurate (mostly the work of Charlie Demerjian) has fallen far from whatever grace it had. The articles try way too hard to strike a clever and informal tone, while self-congratulating and calling out every slight the author believes he received, at every possible opportunity. It turns them into rants that I find virtually unreadable. Add to that the jealous paywall-guarding of the few actual nuggets of information the author managed to collect and the entire site is basically just a frustrating waste of time.

    I really don't know why anyone in the industry still talks to Charlie, at this point. It's probably just a shrinking number of disgruntled tech workers trying to settle a score, which makes his information very subject to exaggeration and mischaracterization.

    If you want to see what SemiAccurate could have been, head on over to SemiAnalysis:
    https://www.semianalysis.com/
    Alright. Man, I wonder what happened to him that made him change I guess. Maybe stress or something. I read the article and it was very weird, he acted as if qualcomn owed him to get a sneek peak of their chip.
    Reply
  • CmdrShepard
    In the past Charlie was right on Intel SNAFUs so I will give them a benefit of a doubt (I don't trust Qualcomm as far as I can throw them).
    Reply