- Aug 23, 2006
- 19,840
- 0
- 50,860
I just read this article, and it looks like Intel needed help back in the day. I think all benchmarks need to be checked out for things like this, as Intel gets a good nod from PCMark. Also, the Atom isnt looking that great http://arstechnica.com/reviews/hardware/atom-nano-review.ars/6 Quote: This, gentle reader, is where things get fun. I've heard rumors for years that performance in PCMark 2005 could change depending on what CPUID was handed to the benchmark, but this is the first opportunity I've ever had to test that theory. The term CPUID refers to a processor-specific character string that stores information on the chip's manufacturer, available features, make, and model. Different manufacturers use different CPUIDs, including GenuineIntel, AuthenticAMD, CentaurHauls, and the now-obsolete CyrixInstead. Intel and AMD both lock their CPUIDs to prevent them being changed by a third party, but VIA doesn't—and that gives us an opportunity to explore a question that normally can't be explored.
By changing Nano's CPUID, we can change what value is handed off to FutureMark and expose any irregularities in the benchmark results. If everything is five by five, we shouldn't see any meaningful performance variation at all. According to the PCMark 2005 whitepaper, "The cornerstones of our design process are transparency and neutrality. We make a strong effort to document all processes that make up the benchmark...Also, we always maintain the highest standards of neutrality, neither favoring nor dis-favoring any party. I'd say that lays out the company's position in no uncertain terms, so lets take a look at how different CPUIDs impact Nano's performance.
The graph above covers all of PCMark 2005's test suites except for the memory benchmark. As you can see, everything here is as it should be; PCMark doesn't care if Nano identifies itself as GenuineIntel or CentaurHauls. Memory subsystem performance, on the other hand, looks a wee bit different.
Also : My my. Swap CentaurHauls for AuthenticAMD, and Nano's performance magically jumps about 10 percent. Swap for GenuineIntel, and memory performance goes up no less than 47.4 percent. This is not a test error or random occurance; I benchmarked each CPUID multiple times across multiple reboots on completely clean Windows XP installations. The gains themselves are not confined to a small group of tests within the memory subsystem evaluation, but stretch across the entire series of read/write tests. Only the memory latency results remain unchanged between the two CPUIDs.
At the very least, this suggests some incredibly sloppy coding on Futuremark's part, as the company may be enabling or disabling CPU optimizations based on a processor's vendor name in CPUID instead of actually checking CPUID for SIMD support. In this case, PCMark 2005's memory subsystem test doesn't appear to be aware that Nano supports SSE2 and SSE3, and is instead running a ecidedly less-optimized code path. There are two factors, however, that make this explanation a bit difficult to swallow.
First, there's the issue of timing. PCMark 2005 was released (obviously) in 2005, and was obviously coded with an eye towards supporting current and future processors. This is standard operating procedure for Futuremark, which always builds benchmarks designed to last for at least a year, and often two. VIA's C5N-T (Nehemiah) core may have only supported MMX and 3DNow!, but the C7 launched in 2005, and that processor supported SSE2 and SSE3 from day one. Even if proper extension support wasn't built into the first version of PCM2K5, we tested version 1.2.0, and that patch was released on or around 11-29-2006.
Second, there's the issue of performance when Nano is identified as AuthenticAMD. If performance between the AMD and Intel CPUIDs was identical, there wouldn't really be a story here, but it isn't, and that's curious. Futuremark could plausibly argue that VIA's C3/C7 processors weren't exactly on the radar back in 2004-2005, but AMD and K8 certainly were, and K8 launched with full SSE and SSE2 support, with SSE3 added in 2005.
None of this constitutes proof of wrongdoing, but it flies in the face of Futuremark's neutrality claims. Bad code is a fact of life, but companies that write benchmarks for a living and sell those benchmarks as evaluation tools have a responsibility to ensure that their software delivers the neutral framework that it promises. Based on the information I've gathered thus far, it seems Futuremark may have created three paths—one for Intel, one for AMD, and one generic "other" path. There's nothing wrong with optimized code paths, but our results would seem to indicate that some paths are decidedly more optimized than others. This doesnt look good for PCMark, nor Intel. I know its a dated benchmark, but Ive heard other things about other benchmarks. Opinions welcome
By changing Nano's CPUID, we can change what value is handed off to FutureMark and expose any irregularities in the benchmark results. If everything is five by five, we shouldn't see any meaningful performance variation at all. According to the PCMark 2005 whitepaper, "The cornerstones of our design process are transparency and neutrality. We make a strong effort to document all processes that make up the benchmark...Also, we always maintain the highest standards of neutrality, neither favoring nor dis-favoring any party. I'd say that lays out the company's position in no uncertain terms, so lets take a look at how different CPUIDs impact Nano's performance.
The graph above covers all of PCMark 2005's test suites except for the memory benchmark. As you can see, everything here is as it should be; PCMark doesn't care if Nano identifies itself as GenuineIntel or CentaurHauls. Memory subsystem performance, on the other hand, looks a wee bit different.
Also : My my. Swap CentaurHauls for AuthenticAMD, and Nano's performance magically jumps about 10 percent. Swap for GenuineIntel, and memory performance goes up no less than 47.4 percent. This is not a test error or random occurance; I benchmarked each CPUID multiple times across multiple reboots on completely clean Windows XP installations. The gains themselves are not confined to a small group of tests within the memory subsystem evaluation, but stretch across the entire series of read/write tests. Only the memory latency results remain unchanged between the two CPUIDs.
At the very least, this suggests some incredibly sloppy coding on Futuremark's part, as the company may be enabling or disabling CPU optimizations based on a processor's vendor name in CPUID instead of actually checking CPUID for SIMD support. In this case, PCMark 2005's memory subsystem test doesn't appear to be aware that Nano supports SSE2 and SSE3, and is instead running a ecidedly less-optimized code path. There are two factors, however, that make this explanation a bit difficult to swallow.
First, there's the issue of timing. PCMark 2005 was released (obviously) in 2005, and was obviously coded with an eye towards supporting current and future processors. This is standard operating procedure for Futuremark, which always builds benchmarks designed to last for at least a year, and often two. VIA's C5N-T (Nehemiah) core may have only supported MMX and 3DNow!, but the C7 launched in 2005, and that processor supported SSE2 and SSE3 from day one. Even if proper extension support wasn't built into the first version of PCM2K5, we tested version 1.2.0, and that patch was released on or around 11-29-2006.
Second, there's the issue of performance when Nano is identified as AuthenticAMD. If performance between the AMD and Intel CPUIDs was identical, there wouldn't really be a story here, but it isn't, and that's curious. Futuremark could plausibly argue that VIA's C3/C7 processors weren't exactly on the radar back in 2004-2005, but AMD and K8 certainly were, and K8 launched with full SSE and SSE2 support, with SSE3 added in 2005.
None of this constitutes proof of wrongdoing, but it flies in the face of Futuremark's neutrality claims. Bad code is a fact of life, but companies that write benchmarks for a living and sell those benchmarks as evaluation tools have a responsibility to ensure that their software delivers the neutral framework that it promises. Based on the information I've gathered thus far, it seems Futuremark may have created three paths—one for Intel, one for AMD, and one generic "other" path. There's nothing wrong with optimized code paths, but our results would seem to indicate that some paths are decidedly more optimized than others. This doesnt look good for PCMark, nor Intel. I know its a dated benchmark, but Ive heard other things about other benchmarks. Opinions welcome
Facebook
Twitter
Instagram