NVidia respond to Futuremark RE 3dmark03 cheats

[rain_king_uk]

With the introduction of the GeForce FX - we built a sophisticated real-time compiler called the Unified Compiler technology. This compiler does real-time optimizations of code in applications to take full advantage of the GeForce FX architecture.

Game developers LOVE this - they work with us to make sure their code is written in a way to fully exploit the compiler.

The end result - a better user experience.

One of the questions we always get is what does this compiler do? The unified compiler does things like instruction reordering and register allocation. The unified compiler is carefully architected so as to maintain perfect image quality while significantly increasing performance. The unified compiler a collection of techniques that are not specific to any particular application but expose the full power of GeForce FX. These techniques are applied with a fingerprinting mechanism which evaluates shaders and, in some cases substitutes hand tuned shaders, but increasingly generates optimal code in real-time.

Futuremark does not consider their application a "game". They consider it a "synthetic benchmark". The problem is that the primary use of 3DMark03 is as a proxy for game play. A website or magazine will run it as a general predictor of graphics application performance. So it is vital that the benchmark reflect the true relative performance of our GPUs versus competitors.

And, while they admit that our unified compiler is behaving exactly the way it behaves in games and that it produces accurate image quality, they do not endorse the optimizations for synthetic use. Hence, Futuremark released a patch that intentionally handicapped our unified compiler.

So, we advocate that when reviewers are using 3DMark as a game proxy, they must run with the unified compiler fully enabled. All games run this way. That means running with the previous version of 3DMark, or running with a version of our drivers that behave properly.

Derek Perez
Director of Nvidia PR

 

[rain_king_uk]

With the introduction of the GeForce FX - we built a sophisticated real-time compiler called the Unified Compiler technology.

So sophisticated that it doesn't work when the driver can't detect the application that is running in order to substitute the manually written shader routines etc.

The unified compiler is carefully architected so as to maintain perfect image quality while significantly increasing performance.

Well I think it's safe to say the "unified compiler" (also known as "NVidia driver engineers manually coding replacement graphical routines that are substituted when a specific application is detected as running" - not quite as catchy) many times achieves this significant increase in performance by degrading image quality. It did in 3DMark03 and I am pretty sure it does in alot of games. And when it's not doing that, many times it's not quite showing what the developer of the application intended.

So it is vital that the benchmark reflect the true relative performance of our GPUs versus competitors.

...

So, we advocate that when reviewers are using 3DMark as a game proxy, they must run with the unified compiler fully enabled. All games run this way. That means running with the previous version of 3DMark, or running with a version of our drivers that behave properly.

I think actually the benchmark should run however the hell Futuremark want it to run - it's their benchmark. In this case Futuremark want to test how well their application runs without application specific "optimisations".

And this is the crux of the issue - Futuremark have programmed their application to run how they want and NVidia, against Futuremark's wishes, are tampering with that, in some cases causing degraded results - same as they do with games. This is not what a driver is for.

NVidia really are now scraping the bottom of the barrel even more in my eyes.

<P ID="edit"><FONT SIZE=-1><EM>Edited by RAIN_KING_UK on 11/13/03 07:22 AM.</EM></FONT></P>

 

[shadus]

Has anyone done an IQ comparison on the new drivers and 3dmark03? Not that it would suprise me if they're cheating, but what they are describing as being disabled is exactly what they've been working on that is giving better performance across the board for the fx cards.

Perhaps a bit quick to jump to conclusions? The new drivers have *NO* iq problems *I* can see in comparisons with with the radeon except of course for having a poor fsaa alg compared to the radeon (vert lines.) Alot of sites note pretty much the same thing (or note that in some cases the radeon is rendering things incorrectly, ala is ati cheating crap that went around a bit ago.)

Shadus

 

[Ion]

<A HREF="http://www.beyond3d.com/articles/3dmark03/340/index.php?p=6" target="_new">Image Comparison from B3D</A>

 

[Ion]

Bascially that reply is saying we would like to control whats been display on the screen regardless if it is a benchmark or game because we think it is the best way to gain performance.

However the reply fail to solve the issue surrounding the case when certain game/application is not included in the optimise list and it runs crap.

 

[HarrY]

weeeeeeee..........look at me.......i've fallen into a ocean
i cant swim
I m drowning
.........no wait
...........i think i can gulp down the water
and walk out of here........i m so smart

 

[TheGreatGrapeApe]

Which of course is the biggest drawback to their strategy.

The nV's require this extra time and effort to make their cards run standard code to the same level that the ATI cards run out of the box.

And since there are IQ differences, how does this not fall outside the guidelines of the Futuremark enforcement paper?

I kinda thought there would be mention of their run-time compiler, but really the investigations so far show that the issue is more broad than just that.


- You need a licence to buy a gun, but they'll sell anyone a stamp <i>(or internet account)</i> ! <A HREF="http://www.redgreen.com" target="_new"><font color=green>RED</font color=green> <font color=red>GREEN</font color=red></A> GA to SK

 

[shadus]

The images are slightly different. Image quality wise, when you get the full blown up pic on the screen the only one that looks worse is the hair one. nVidia needs to just put a strict policy in place they don't test on 3dmark03, they don't do any opts for it, they don't do anything with it whatsoever. I don't think they decreased the quality so much as are reworking the way it is done causing a slightly differnt image (which is bad in its own way) the sad thing is nvidia seems so obsessed with 03 which isnt a game and doesnt matter for a [-peep-]. Not worth the bad press.

Shadus

 

[phial]

your vague humor eludes me

-------


<A HREF="http://www.albinoblacksheep.com/flash/you.html" target="_new">please dont click here! </A>

 

[cleeve]

I think perhaps he was alluding to the attitude of the Nvidia PR staff.

------------------
Radeon 9500 (hardmodded to PRO, o/c to 322/322)
AMD AthlonXP 2400+ (o/c to 2600+ with 143 fsb)
3dMark03: 4055

 

[TheGreatGrapeApe]

Vague, like une Vague en Français?

Puny!

Go on, Wave to the nice people!




- You need a licence to buy a gun, but they'll sell anyone a stamp <i>(or internet account)</i> ! <A HREF="http://www.redgreen.com" target="_new"><font color=green>RED</font color=green> <font color=red>GREEN</font color=red></A> GA to SK

 

[TheGreatGrapeApe]

A response from FutureMark @ the Inquirer;

<A HREF="http://www.theinquirer.net/?article=12641" target="_new">http://www.theinquirer.net/?article=12641</A>


And Tero Sarkkinen's personal (or on his behalf) reply to me;
Dear X.X. (edited)

Thank you for your email. We are now executing the enforcement policy:

"The next step in enforcing the guidelines will be a release of a new build
of 3DMark03. There will be both full version, and a patch. The patched
version with drivers that are listed on Futuremark's web site as 'Reviewed',
will produce a valid 3DMark result."

So, patch 340 enables you to get the valid 3DMark03 score with drivers we
have posted on page:
<A HREF="http://www.futuremark.com/community/drivers/?approved" target="_new">http://www.futuremark.com/community/drivers/?approved</A>

Going forward, we will only add drivers to the page, which we have
determined, produce a valid 3DMark03 score. I hope that all new drivers will
be valid and that we will not have resort to launching new patches because
of offending drivers.

Please let me know if there's anything else I can help you with.

Best regards,

Tero Sarkkinen
Executive Vice President of Sales and Marketing

Futuremark Corporation


COOL!


Also this response took less than 24hrs to comeback; it wasn't there this morning but I just checked when I read the Inquirer article, and voila.

Just thought I'd share that with y'all.


- You need a licence to buy a gun, but they'll sell anyone a stamp <i>(or internet account)</i> ! <A HREF="http://www.redgreen.com" target="_new"><font color=green>RED</font color=green> <font color=red>GREEN</font color=red></A> GA to SK

 

[rain_king_uk]

Cool - thanks for sharing.

I am hearing there are already new NVidia drivers leaked that turn the cheats back on - I guess they won't be getting approved.

It would be really great if MS took this attitude when certifying drivers too - only then I think would nVidia stop this crap.

 

[coolsquirtle]

Lemme show u guys the ORIGINAL DOCUMENT

With the introduction of the GeForce FX - we built a shitty ass compiler but we're going to say it's sophisticated real-time compiler called the Unified Compiler technology. This compiler does shorttcuts that is in optimizations of code in applications to take full advantage of the [-peep-] GeForce FX architecture.

We FORCE THE Game developers TO LOVE this - we make them work with us to make sure their code is written in a way that it will performs more than 10FPS in a DX9 game and to force the code to run slower on ATi hardware.
The end result - a shiitty user experience, we get shitloads of money, ATi gets WASTED, but it doesn't seem so.

One of the questions we always get is what does nVidia do to come up with such a shitty complier? The unified compiler does things like instruction reordering and register allocation except it does it the OPPOSITE of how it suppose to be. The unified compiler is carefully architected so as to maintain the cheats to not let stupid hardware people to findout about them. The unified compiler a collection of techniques that are not specific to any particular application and it doesn't do dipshiit but we're stupid and that's the only way to expose the full power of GeForce FX. These techniques are applied with a lot of cash and make the game developer uses our f***ed up codes to substitutes the hand tuned one, but still make it run good on nVidia Hardware.

Futuremark does not consider their application a "game". They consider it a "synthetic benchmark". We tried to make them suck our diick, but they gave up half way, simply because we didn't give them enough cash. The problem is that the primary use of 3DMark03 is as a proxy for game play. A website or magazine will run it as a general predictor of graphics application performance which we can't stop because we dont have enought money to shutup every single review site. So it is vital that the benchmark reflect the "true" performance that we want the public to believe.

And, while they admit that our unified compiler is [-peep-] the way it behaves in games and that it produces fake accurate image quality, they do not endorse the fact that we bribed them millions of $ to put in these optimizations for synthetic use. Hence, Futuremark decided to suck ATi's diick and release released a patch that intentionally [-peep-] up our cheating compiler.

So, we advocate that when reviewers to suck our diick and stop using 3DMark as a game proxy, they must run with the cheating compiler fully enabled. All games that we bribed to modify run this way. That means running with the previous version of 3DMark, where me made futuremark use the cheating complier or running with a version of our drivers that behave properly.

Coolsquirtle
Director of Nvidia BS



RIP Block Heater....HELLO P4~~~~~
120% nVidia Fanboy
bring it fanATics~~ nVidia PWNS all!
SCREW aBOX! LONG LIVE nBOX!!

 

[Guest]

Captain Obvious gives your post the thumbs up!

<b><font color=red>Captain Obvious To The Rescue!!!</font color=red></b>

 

[TheGreatGrapeApe]

A little more...

from the Inquirer;

<A HREF="http://www.theinquirer.net/?article=12673" target="_new">http://www.theinquirer.net/?article=12673</A>

and from

[H];

<A HREF="http://www.hardocp.com/article.html?art=NTQ5" target="_new">http://www.hardocp.com/article.html?art=NTQ5</A>

I agree with Tero, it's not about predicting games, or how things will run in games, it's generic accross the board performance based on the DX standard.

I'm surprised that Kyle isn't harder on the way optimizations work, especially since I think it shows how nV will always be last out of the gate, even if optimizations do come to the recue a few driver releases later, while the ATIs will perform to expectation straight from the start. I guess Kyle's dislike of Futuremark is greater than towards nV. Surprising as he seemed so ticked off at the bi/trilinear issue.

Good reads none the less.


- You need a licence to buy a gun, but they'll sell anyone a stamp <i>(or internet account)</i> ! <A HREF="http://www.redgreen.com" target="_new"><font color=green>RED</font color=green> <font color=red>GREEN</font color=red></A> GA to SK

 

[TheGreatGrapeApe]

Also ATI's reaction is enclosed in this article by GamersDepot;

<A HREF="http://www.gamersdepot.com/ed/mkting_hype/001.htm" target="_new">http://www.gamersdepot.com/ed/mkting_hype/001.htm</A>

I have to agree, seriously as much as the run-time comiler helps in the games we benchmark most often, what about the games that doesn't attract nV's attention?

Futuremark is 'synthetic' and not a game benchmark, and not playable. But in all likelyhood it is a better reflection of how your card will react with new or unpopular games than the benchmark standards that get finessed by game-specific optimizations.


- You need a licence to buy a gun, but they'll sell anyone a stamp <i>(or internet account)</i> ! <A HREF="http://www.redgreen.com" target="_new"><font color=green>RED</font color=green> <font color=red>GREEN</font color=red></A> GA to SK

 

[TheGreatGrapeApe]

And nV's retracting their initial statements about 3Dmk03 disabling their run-time compiler;

Xbit has this blurb covering the retraction;;
<A HREF="http://www.xbitlabs.com/news/video/display/20031114041519.html" target="_new">http://www.xbitlabs.com/news/video/display/20031114041519.html</A>

It appears even nV has trouble understanding how their drivers work/interact with other apps.

- You need a licence to buy a gun, but they'll sell anyone a stamp <i>(or internet account)</i> ! <A HREF="http://www.redgreen.com" target="_new"><font color=green>RED</font color=green> <font color=red>GREEN</font color=red></A> GA to SK

 

[HardWareBoss]

What a bunch of cry babies.

 

[HardWareBoss]

Makes me want to run out and spend my hard earned money on Nvidia. Yeah right.

 

[TheGreatGrapeApe]

It's funny I just got an e-mail response from Kyle Bennett of [H] about an e-mail I sent yesterday regarding the 3Dmark situation and here's his VERY brief response (hey he's a busy man);

<font color=blue>
"
Then I would suggest that you use its results to help your buying decision
if you see value in it. :tongue:
"
</font color=blue>

I did not add the tongue (except to make it visible here), that's how it came. I guess I shouldn't have expected much seeing as he's not a fan of Futuremark. I don't see how one can use a benchmark to help one's buying decision if you don't have the card to benchmark because you haven't been able to decide yet. Perhaps he was refering to my reference that we are soon going to be forced to buy both cards and then benchmark, and then 'finalize' our buying decision. In that case I won't need review sites like [H] or THG.

I guess we have to rely on ourselves to do our own benchmarking, because reviewers will stick to the older gaming benchmarks, and not look towards future products with synthetic benchmarks that stress specific parts. That's ok for all the current buyers, but I'm not going to base my buying decision for the R42X or RV380 based on Kyle and other's Quake3 and UT2K3 Benchmarks. Great, yeah I know it can play 2 year old games just fine, thanks.

I have a feeling that those sites currently against 3Dmk03 will be equally against Futremarks ineviteable PS3.0/VS3.0 benchmark because it's synthetic. Oh well I guess the only forward looking info will come from the early adopters who post in the forums. I'm surprised that these guys have such a problem with 3Dmk03, but seem to have no problem with Shadermark. I guess if it didn't look so much like a game, then more people wouldn't criticise it for not being representative of all games.

Oh well. So be it.


- You need a licence to buy a gun, but they'll sell anyone a stamp <i>(or internet account)</i> ! <A HREF="http://www.redgreen.com" target="_new"><font color=green>RED</font color=green> <font color=red>GREEN</font color=red></A> GA to SK

 

[rain_king_uk]

Clearly our compiler has gotten much better, as image quality remains exactly the same, the only thing that happens is a 10-15% drop in performance.

We're not sure why anyone would want to reduce their performance by 10-15% for the same image quality, but apparently Futuremark feels that is something relevant.

What a liar. Do you think they believe their own lies? It DOES CLEARLY effect IQ.

 

[rain_king_uk]

It's obvious that NVidia's compiler isn't as swish as they pretend and there is alot of app detection and manually coded graphics routines substituted for certain apps. Futuremark were absolutely correct to disallow this.

 

[TheGreatGrapeApe]

Yeah I think that people are also missing what 3Dmark is supposed to be. Yes, NOT A GAME, but a DX9 benchmark. Not a 'how can we construe X in such a way that our shortcomings in DX standards isn't as apparent and we can acheive almost the same with our own tweaks whihc we won't apply to every game since we don't have that much interest in the games so much as the benchmarks that sell our products' (which is a long sentence/idea)

Seriously nV wants to do everything it can to have a single standard of anything, be it benchmarks, be it shader language, or architecture. Even 'The Way It's Meant To Be Played' isn't enough now. Funny how many of those games play better on ATI's standardized hardware without game specific tweaks nV enjoys.

This really bothers me the most because I beleive in standards, especially as someone who likes competition and realistic level playing field (chance for Matrox to come back ?)

Special Circumstances/Codepaths/etc. = Someone getting $crewed IMO.


- You need a licence to buy a gun, but they'll sell anyone a stamp <i>(or internet account)</i> ! <A HREF="http://www.redgreen.com" target="_new"><font color=green>RED</font color=green> <font color=red>GREEN</font color=red></A> GA to SK

 

[sargeduck]

Game developers LOVE this - they work with us to make sure their code is written in a way to fully exploit the compiler.

I don't think they have a choice. If a game developer wants to sell a game, they have to make it work on as many possible platforms as possible. Last I heard, it took Valve 5x longer to code for Nvidia. Even John Carmack had to add extra stuff to make Doom3 run better.
Game developers love our stuff.......sure they do. They just have to do 5x the amount of work first.


Error: Keyboard not attached. Press F1 to continue