Sign in with
Sign up | Sign in

AMD Won't Endorse SYSmark 2012 Benchmark

By - Source: AMD | B 34 comments

AMD claims that SYSmark 2012 is "misleading" and doesn't offer "clear and reliable measurements." Ouch.

Tuesday AMD made it officially well-known that it has no plans to endorse BAPCo's upcoming SYSmark 2012 benchmark. In fact, America's #2 processor giant has resigned from the BAPCo organization altogether, and even hinted that SYSmark 2012 doesn't provide "clear and reliable measurements" and is "misleading." AMD's Nigel Dessau followed up with a little clarification on the company's stance Tuesday in his blog.

"The heart of our complaint is this: the SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor’s designs), but it actually generates misleading results that can lead to very poor purchasing decisions, causing governments worldwide to historically overspend somewhere in the area of approximately $8B!" he said.

Dessau went on to explain that AMD tried to work within the BAPCo consortium in order to get the next-generation benchmark, SYSmark 2012, correct. AMD even stressed that it needed to be open, transparent and processor-neutral. Unfortunately, that didn't happen.

"Our good intentions were met with an outcome that we believe does a disservice to the industry and our customers," he said. "We weren’t able to effect positive change within BAPCo, and the resulting benchmark continues to distort workload performance and offers even less transparency to end users."

That said, AMD pulled itself out of BAPCo and asked that the AMD brand be removed from marketing materials promoting SYSmark 2012. According to the company's official press release, it will only endorse benchmarks based on real-world computing models and software applications, and which provide useful and relevant information.

"AMD believes benchmarks should be constructed to provide unbiased results and be transparent to customers making decisions based on those results," AMD said in an official statement. "Currently, AMD is evaluating other benchmarking alternatives, including encouraging the creation of an industry consortium to establish an open benchmark to measure overall system performance."

"We are committed to working with likeminded companies that want to give consumers and business users an accurate, honest measure of what they can expect from their PCs and mobile devices," Dessau added.

Display 34 Comments.
This thread is closed for comments
Top Comments
  • 21 Hide
    fleeb , June 22, 2011 7:30 AM
    AMD, nVidia, and VIA already left BAPCo. What does that say?
  • 18 Hide
    mkrijt , June 22, 2011 8:24 AM
    I found some interesting read over here. Ok, It's old but I think it does cover the problems with SYSmark...

    Quote:
    In fact, Fog points out that even benchmarking programs are affected by this, up to a point where benchmark results can differ greatly depending on how a processor identifies itself. Ars found out that by changing the CPUID of a VIA Nano processor to AuthenticAMD you could increase performance in PCMark 2005's memory subsystem test by 10% - changing it to GenuineIntel yields a 47.4% performance improvement!
  • 18 Hide
    reggieray , June 22, 2011 1:06 PM
    To all the AMD haters, if AMD fails (folds) then expect chip prices to double.
Other Comments
  • 21 Hide
    fleeb , June 22, 2011 7:30 AM
    AMD, nVidia, and VIA already left BAPCo. What does that say?
  • 16 Hide
    noob2222 , June 22, 2011 7:40 AM
    fleebAMD, nVidia, and VIA already left BAPCo. What does that say?

    sysmark is still based on dual core testing, ie ... crap
  • 4 Hide
    mactronix , June 22, 2011 8:09 AM
    We are committed to working with like minded companies that want to give consumers and business users an accurate, honest measure of what they can expect from their PCs and mobile devices,"

    Yes thats all very noble but the point is if AMD are involved then Nvidia people will start shouting biased.
    We already have the info needed to make an informed decision its out there on this and various other sites. Common sense says you look across a selection of sites to get a clearer more accurate picture. Some sites are after all leaning one way or the other.
    Mactronix
  • 18 Hide
    mkrijt , June 22, 2011 8:24 AM
    I found some interesting read over here. Ok, It's old but I think it does cover the problems with SYSmark...

    Quote:
    In fact, Fog points out that even benchmarking programs are affected by this, up to a point where benchmark results can differ greatly depending on how a processor identifies itself. Ars found out that by changing the CPUID of a VIA Nano processor to AuthenticAMD you could increase performance in PCMark 2005's memory subsystem test by 10% - changing it to GenuineIntel yields a 47.4% performance improvement!
  • 1 Hide
    crisan_tiberiu , June 22, 2011 8:50 AM
    mkrijtI found some interesting read over here. Ok, It's old but I think it does cover the problems with SYSmark...

    ermm...i believe that ^^
  • 15 Hide
    KT_WASP , June 22, 2011 9:41 AM
    SYSmark should have been abandoned years ago by the companies that are leaving it now (its basically an Intel marketing tool anyway). What I hate is that tech/review sites still use it, even though they know how skewed it is towards Intel.

    SYSmark = synthetic benchmark that falsely gives Intel the edge every time. Intel controls it and no input/suggestions are ever implemented from the other companies that finally said "enough".
    Why no mention of Nvidia and VIA leaving at the same time as AMD? It was a mass exodus leaving only Intel. The deal is, these companies are finally fed up and are dropping this sham of a benchmark.

  • 9 Hide
    palladin9479 , June 22, 2011 9:43 AM
    Yep, the big benchmarking companies have been in bed with the highest payer (usually Intel) for years now. Usually it takes the form of checking CPUID and using that to determine code path and optimizations. You then make the code for your preferred CPU more "efficient" and leave everyone else on a non-optimized path. This isn't only in benchmarking but also in software development. Lets say publisher A is making a game, and decides to "work with" Intel to "get the most" out of their code. Intel provides a SDK with a compiler they say will help optimize their code. The compiler does indeed optimize the code, but only for Intel branded systems, and further the compiler may intentionally compile the program so that it runs a very inefficient non-optimized path if it detects its running on a "non-Intel" system. After all Intel can claim they only support their own products and have no requirement to support competitors products.

    This is why I take ~all~ "benchmarks" with a grain of salt. Different code paths can swing performance 10~40% depending.
  • 0 Hide
    warmon6 , June 22, 2011 9:52 AM
    crisan_tiberiuStill, where is Bulldozer? i have expected it on 20.06 ^^. I went i7 2600k on month ago, could not wait anymore


    sometime between next month to late September area is all the info i got. Although this info been out for a little while.
  • 6 Hide
    back_by_demand , June 22, 2011 11:08 AM
    fleebAMD, nVidia, and VIA already left BAPCo. What does that say?

    Intel own Sysmark.
  • 4 Hide
    verbalizer , June 22, 2011 11:19 AM
    I just want competition and the best performing products that can be pushed out.
    just for the record, I'm not a SYSmark fan as well..
  • 5 Hide
    goodguy713 , June 22, 2011 11:24 AM
    yea this is pretty interesting although its common knoledge that those bechmarks are rigged .. it would be interesting to see how well some of the current gen amd processors really fair with un-biased benchmarks. not that i think they would surprise a lot but it would still be interesting .. i mean in the end i guess if you can play pretty much any game at 60fps or shrink and convert a dvd in 5-6 mins then you have a pretty fast system but both of those are either pretty gpu intensive or hard drive intensive better off with a raid or dual gpu set up. lol
  • 5 Hide
    WR2 , June 22, 2011 11:56 AM
    mactronixWe already have the info needed to make an informed decision its out there on this and various other sites. Common sense says you look across a selection of sites to get a clearer more accurate picture.
    Correct.
    Thankfully we're not limited to just the AMD or Intel marketing hype (or any other MFGR for that matter). We have many good independent review sites that test and report what they find.
  • 8 Hide
    fir_ser , June 22, 2011 12:25 PM
    This definitely indicates that SYSmark benchmark is a biased one, and I believe it overstates Intel’s CPUs performance.
  • 9 Hide
    Anonymous , June 22, 2011 12:47 PM
    http://www.phoronix-test-suite.com/
    http://openbenchmarking.org/
  • 18 Hide
    reggieray , June 22, 2011 1:06 PM
    To all the AMD haters, if AMD fails (folds) then expect chip prices to double.
  • 5 Hide
    demonhorde665 , June 22, 2011 2:31 PM
    Maybe it's jsut me but i think sysmark arnt teh only "intel-optimized" bench marking tools , i see this same pattern with futuremark and 3d mark for graphics , it always reads intel cpu as better regradless of the model (unless it is jsut a model taht is that much oplder than the amd model you test with. also whehn testing itnergarted graphic chips it will quite oddly favor , intel over invida and nivdia over ati NOW HOW CRAZY THAT !.

    for this reason i gave up longa gout even bothering to read synthetic bench marks and i only look at the real world test , and in all hoensty i think tom's should DROP synthetic test
Display more comments