Sign in with
Sign up | Sign in
Your question

Anyone think more Objectivity is needed?

Last response: in Graphics & Displays
Share
a b U Graphics card
July 11, 2007 2:33:53 PM

I have a question for you all.

It has come to my attention that many sites seem to not post the whole story. They omit game results that would show their sponsors in a bad light (at least that's m observation). Now I don't want to make any accusations just yet but take the info I'm giving you bellow at face value. Before I start I have to make a confession. I currently only own Geforce 8 series cards (2x8800GTX + 1x8800GTS) I thought I should mention this beforehand.

Perfect examples are:

HardOCP
FalconNorthwest
Fudzilla
Bjorn3D

What I mean is that.. how many of you play S.T.A.L.K.E.R? How many of you play Flight Simulator X?

Would you deem those two games to be more popular then:
Quake 4
Battlefield 2
Doom 3
Half Life 2
and
F.E.A.R ?

I don't think so.. not if you look at the sales. Sales figures are posted on a monthly basis and S.T.A.L.K.E.R./Flight Simulator X have never even been on the radar. As such most gamers have the other titles I've mentioned above and play those instead.

So why does HardOCP only test on those games which are known to show the Radeon HD 2900XT in a bad light? (games which are the least popular as well) And why does HardOCP not test those other games I mentioned where the Radeon HD 2900XT 512MB is on par or faster then an 8800GTS 640MB? (this is a question coming from a Geforce 8 series owner).

Here's the HardOCP review in question:
here with the link I just provided above.

Now have a look at this:
here.

Both those two reviews painted a single one sided story each using titles that performed best on each respective architecture.

Now for some Objectivity look here:
here?

Notice anything?

It's quite apparent that HardOCP and Fudzilla are both misleading the public from their own respective ends. Oddly enough nVIDIA card partner banners are all over HardOCP's site whereas ATi/AMD banners are all over Fudzilla's site. Coincidence? Maybe. I've also heard some folks over at HardOCP claim that HardOCP does benchmarking different. Well.. how so? They claim HardOCP uses AA+AF + High resolutions. Guess what.. so did Firingsquad.

It seems that Kyle Bennett (yes I got in a HUGE argument with him over this) won't admit to using un-popular games that seem to only paint one side of the story.

Now taking it further I visited the sites in question and took screenies.

These images are large so bare with me (i've made thumbnails):

Fudzilla's Advertising=

They lean ATi/AMD in all their articles.. perfect match

HardOCP's Advertising=

They lean nVIDIA in all their articles.. perfect match

Firingsquad's Advertising=

They don't lean to either side and paint a fair and balanced picture testing MANY games. They clearly show that as a whole the Radeon HD 2900XT is a match for the GeForce 8800GTS 640MB (unlike what HardOCP show) but that it trails the GeForce 8800GTX 768MB (unlike what Fudzilla show). And their advertising is nVIDIA thus clearly highlighting objectivity.

So what do you think about Objectivity in today's PC Hardware reviews?

I just gave you the information to consider what conclusions you come too are your own. I expect that fanboyism won't poison this thread and instead we can discuss this as adults.

You decide.

More about : objectivity needed

a b U Graphics card
July 11, 2007 2:39:42 PM

BTW I'd like to see Tom's Hardware visit this as well. Seeing as DAAMIT drivers have improved considerably since the release of the Radeon HD 2900XT 512MB.

Toms tends to show a wide array of games (just like Anandtech, Firinsquad and Tweaktown). As well as many in game settings.

If anyone at Tom's see's this.. would it be possible?
a b U Graphics card
July 11, 2007 2:53:10 PM

I agree 100% . I was extremely disappointed by the HardOCP review. Since we all use references here in the forums for advice to buy and check out te ch info, Ive had this thrown in my face a few times. Most of us here take FUaDs site with a grain of salt, so I guess Ill have to do the same for HardOCP. Its a shame really,especially reading that the newer drivers for the 2900 are more optimised PER game than ever before. Good work , this is common sense, and we all know, thing about common sense is it aint so common
Related resources
July 11, 2007 2:59:36 PM

i mainly use 3 sites: toms, hexus and firingsquad

July 11, 2007 3:05:19 PM

well, I agree that there may be "suspect" issues here, I am not sure you can really prove it beyond the tin-foil hat conspiracy level that you have it at so far.

Honestly, I read from most of the sites you mention and usually find that the truth is somewhere in between. the [H] and firingsquad have always been at odds. (so has [H] and toms TBH) and so it does not surprise me that you find issue with how they test. ;) 

stalker and flightsimX both tax a machine hard, and I believe that is why the [H] uses them. They used to have WoW on there which I thought was dumb, but meh...

Honestly, quake4 and the other games you mention are not hard for ANY current card to render except perhaps FEAR. (WoW too honestly) Heck, my 9700pro rocks on HL2 and can even be servicable on FEAR and Q4... how much easier is it for a 7600gt or 2900?

I like that they use games that are "not normal" but I agree w/ you that if they claim "real world" as the reason, then games that are actually used more would be better.

I like seeing the games that favor one card over another just as much as the ones that show them even. Oblivion favored the x1900 cards heavily over the gf7 while FEAR tended towards the Nv camp. Big elder scrolls fans needed to know which card to buy... it all helped man.

Regardless, every person has a bias. Whether they admit it or not. If you read all those articles (even firingsquad and toms) with that in mind you can sift out the hype and find the truth.

rock on.
July 11, 2007 3:31:32 PM

Several sites have the appearance of building there tests around a predetermined outcome. To get the true picture you simply need listen to all the witnesses and find the truth somewhere in the middle.

It is no secret that the hardware review business is not always honest. Earlier in the year there was the controversy with PC World.
http://www.boston.com/business/globe/articles/2007/05/11/a_week_after_he_resigned_editor_returns_to_pc_world/

Then Dailytech did the story: Pay to Play: Uncovering Online Payola

http://www.dailytech.com/Pay+to+Play+Uncovering+Online+Payola/article7510.htm

Faud is basically fun to read for rumours but I wouldn't go there for reviews.

HardOCP has been one of the better sites for updating there reviews when new drivers come out etc. For example they've gone into depth on drivers under Vista and have concluded that ATI has done better in that regard. I see your point though that they could have a wider selection of benches.
Regarding you're choice of benches though I'm not sure I agree. For new hardware you want titles that make them sweat. If you run older games for benches you find yourself testing the cpu not the gpu. For example you have bf2 on your list. I still play bf2 and it does nothing to my video card. Sometimes I launch the game and forget to bump my GPU out of it's under clocked desktop state and don't even notice. I default speed it locks on to 99.9 fps at 1680*1050 max everything. That title is now useless for testing high end cards.

Another site whose integrity I question is Tweaktown. I love the one where they put a 2900xt up against an 8800 ultra and concluded the 2900 is a better value. If I owned a hundred thousand shares of Google I wouldn't see the value in buying an ultra. They also have run ATI review after review of 2900's with no AA and then conclude with a couple of AA benches.

Edit to add. I've repeated a bit of what you said Sojrner. I was posting at the same time, I was slow looking up links.

a b U Graphics card
July 11, 2007 3:44:17 PM

No1sFanboy said:
Several sites have the appearance of building there tests around a predetermined outcome. To get the true picture you simply need listen to all the witnesses and find the truth somewhere in the middle.

It is no secret that the hardware review business is not always honest. Earlier in the year there was the controversy with PC World.
http://www.boston.com/business/globe/articles/2007/05/11/a_week_after_he_resigned_editor_returns_to_pc_world/

Then Dailytech did the story: Pay to Play: Uncovering Online Payola

http://www.dailytech.com/Pay+to+Play+Uncovering+Online+Payola/article7510.htm

Faud is basically fun to read for rumours but I wouldn't go there for reviews.

HardOCP has been one of the better sites for updating there reviews when new drivers come out etc. For example they've gone into depth on drivers under Vista and have concluded that ATI has done better in that regard. I see your point though that they could have a wider selection of benches.
Regarding you're choice of benches though I'm not sure I agree. For new hardware you want titles that make them sweat. If you run older games for benches you find yourself testing the cpu not the gpu. For example you have bf2 on your list. I still play bf2 and it does nothing to my video card. Sometimes I launch the game and forget to bump my GPU out of it's under clocked desktop state and don't even notice. I default speed it locks on to 99.9 fps at 1680*1050 max everything. That title is now useless for testing high end cards.

Another site whose integrity I question is Tweaktown. I love the one where they put a 2900xt up against an 8800 ultra and concluded the 2900 is a better value. If I owned a hundred thousand shares of Google I wouldn't see the value in buying an ultra. They also have run ATI review after review of 2900's with no AA and then conclude with a couple of AA benches.

Edit to add. I've repeated a bit of what you said Sojrner. I was posting at the same time, I was slow looking up links.

Hey man,

I agree with you that making a video card sweat is great. But if games that make these cards sweat aren't being played by gamers who buy these cards then how is that indicative of the cards performance one can expect?

Also we've seen that ATi/AMD have been updating their drivers at a rapid pace targeting the more popular games first as well. We've seen what that has done in objective tests. Wouldn't that be a better indication of performance in the long run seeing as it's only a matter of time before they start updating for not-so-popular titles?

So although I agree partially with what you're saying, I totally disagree that showing tests of games not used by gamers showing performance of high end hardware being bought by gamers makes any sense.

That's my opinion. Another thing that needs to be added to these reviews it VISTA vs XP performance. Perhaps a mouse over. Because many gamers (such as myself) run VISTA. Looking at the reviews posted (which were run under XP though) I bought an 8800GTX SLI setup only to find out nVIDIA drivers were horrid under VISTA (they've gotten better, especially their SLI which did not work for the longest time as we can see in the falcon Northwest tests).

My 2 Cents though.

BTW I like everyone's input.. keep 'em coming. Some good arguments are being brought forth :) 
July 11, 2007 6:24:57 PM

No1sFanboy said:
I've repeated a bit of what you said Sojrner. I was posting at the same time, I was slow looking up links.


no worries
a b U Graphics card
July 11, 2007 8:49:55 PM

Yeah, this is quite common and for me, [H] lost alot of it's appearance of objectivity when they started doing sponsored reviews, and when they also had early access to things, and their reviews then spoke glowingly about the sponsor, and then clinically about the others. Recently in their HD2900XT(512MB) review [H] goes through the trouble of telling how nV's CSAA isn't up to ATi's MSAA, but that nV's other option of quality AA is equal to Ati's IQ, and then in the apples to apples section they go on to benchmark the crippled AA and then say the quality AA was about the same speed (if you're only going to benchmark one it seems pretty obvious which one you chose for an Apples to Apples comparison, Duh! :pfff:  )

For me Anandtech has always been an issue because reoslutions and setting suddenly dissapear from one test to another, and when you go elsewhere it's the one situation where some IHV doesn't do as well and it doesn't put them in a good light. Where there's 5 benchmarks and they test all let's say 7 cards at low and high in 4 test, but for one of the test suddenly there's only 5 candidates with 2 missing. :heink: 

When there's such selective testing it begs the question of WTF!?! And don't tell me 'it's unplayable' SHOW ME the numebrs and let ME figure that out.

This issue was explored mildly when [H] started doing sponsored reviews and Kyle went onto the B3D thread to defend their use of sponsorship. My reply to that is pretty much the sentiment I still have, regardless of IHV' Here's what I wrote way back then:

--+--

"Ever think of getting ASUS or Gigabyte (both sellers of both ATi and nV) to sponsor? Or maybe to them sponsorship isn't as vital because they know that molding the reviews and the community isn't as important because in the end they have you regardless of which camp people choose.

While I agree that it's easier to avoid the optimization/IQ debate the way B3D does with their reviews of just ATi and nV cards, but if you're trying to scale the cards and cpu you don't think people would accept a little compromise, especially since there hasn't been much focus on the issue in prior reviews for them to care in the first place?

I guess page-hits and advertising alone brought from good reviews isn't enough to avoid the need for 'sponsored reviews'. However I'd never take one of those any more seiously than I would the Pepsi challenge. Why should I think you (or anyone else, doesn't matter whom) wouldn't omit a poor result for the whole line if you should come across a performance hole?

Too bad there's no equivalent Consumer Reports for tech, unfortunately everyone in the hardware industry is worried about all the PR money out there they can get a chunck of and they risk losing."


--+--


But I think the most telling part is that if the sponsors didn't think they had influence over the process (ensure the testing puts them in a good light), then why would they bother sponsoring the tests. And considering how easily the Advertising money shifts to/from sites. I only trust sties that truely explore the cards and do enough tests that they couldn't hide things even if they tried. Xbit is one of the few sites that test and tests so you get a far bigger picture. FiringSquad is similar in that respect for me especially when it comes to resolutions. Also Brandon's pretty good for feedback and then updating his review if there's something questionable.

The other thing that gets me, more in the 2D video playback area is the number of times that tests occur just before the competition brings out a new product, as if rushing to do the review will help make one product look better than the other. So even an objective review that's properly timed so that their one look per year falls just perfectly before the situation changes so that people will keep refering to that outdated information for the following 11months.

Some times it's just over-sensitivity, but other times it just makes you say 'hmm wonder if the logo int the top/bottom left/right corner had anything to do with the choices in this review. And that feeling is annoying when looking for objective information. And in the end the only way to look at ti is that if all the sponsored reviews are looked at hopefully the sponsored one for A and the sponsored ones for B add up to a balanced picture when you weed out the conflicting info. Of course that may be overly optimistic itself.
a b U Graphics card
July 11, 2007 9:31:49 PM

TheGreatGrapeApe that is exactly my sentiment on the issue. I tried to reason with Kyle but he would not listen and went on about insulting readers of THG, Firingsquad, Anandtech etc. claiming in no such words that they were all simple minded (actually he insulted Firinsquad readers because of the way Firinsquad reviews which is the same way as THG and Anandtech).

That alone was enough to seal the deal for me. So I started to look at the sponsored reviews and the financial problems that HardOCP has had (they closed [H]Consumer and [H]Console due to money issues). I then began to speculate that HardOCP seem to be in it for the money right now and not exactly caring that their reviews paint but a small part of a larger picture.

So I opted to post this in several forums trying to get people's opinions on the matter.

I want to know if they see what I see.
July 11, 2007 10:23:20 PM

There should be a poll of 20 games and see what the top 5 games come up and then bench those games, i think that if atleast 200ppl from the THfourms community were to take this poll; like a thread styled poll. This would show what the majoraty of ppl play and what games truely matter to gamers. the CPU should be run at stock speeds and then at 3.2Ghz, most ppl run theyre comp speeds between those clocks from what ive noticed.

that would be awsome because then we wont see games benched that most ppl EVER play and it would make the results more realistic.

that will keep the results unbias IMO
July 11, 2007 10:34:43 PM

TheGreatGrapeApe said:
And in the end the only way to look at ti is that if all the sponsored reviews are looked at hopefully the sponsored one for A and the sponsored ones for B add up to a balanced picture when you weed out the conflicting info. Of course that may be overly optimistic itself.


therein lies my point earlier. Sure, it may be optimistic. But if you have some that are "more trustworthy" like firingsquad and then have the others that are "suspect"... if the trustworthy falls in the middle of those extremes then it kinda backs all this up and you can still make a solid educated decision on it.

I also mentioned earlier about the [H] having issues with THG and firingsquad which I honestly thought were childish. (going waaay back for big THG fights and within a year for the "firingsquid" fight) The [H] has lost alot it seems lately, but I still use it as a reference for alot of hardware as it is one more view (however biased) that I can compare to.

Also agree w/ stranger on the presentation. Toms is a bit lame compared to that... but it works.

TBH, I would love to see toms get back to the raw hardware "meat and potatoes" tests that used to happen pretty regularly here. Get rid of the clutter and do the stuff that got us all reading and posting here in the first place.

for what it's worth.
a b U Graphics card
July 11, 2007 11:21:25 PM

I agree, and I agree with you and SS, I like the hystograms, I think they provide alot of information about the fluidity of test, but like SS says, their great information layout / chart is ruined by their mediocre methods. It's like have a very beautiful cover page, synopsis and folder for a report, only for the content to be terrible.

If Xbit had [H]'s information, but put into all their tests, then to me that would be the perfect review. And while Xbit's test usually follow 3-5 days after everyone else's, I find them to just offer so much by testing non-standard or at least a wider variety of games. And when Xbit included the min fps in their charts that was almost as good as [H]'s info and put them leaps ahead of most reviews.

'Best playable' is interesting an I liked it when BitTech first started it, but [H]'s implementation leaves alot to be desired (if the performance is close, then show me apples to apples, don't make grass 25% different! :heink:  and then have higher min framerates on the one you say is worse, show me.

It's usually takes me about 10 reviews to get a good feel for a card/chip, but lately it's getting harder because as we see with this generation, the games are so different and the architecture so different and even XP vs Vista is so different that things can easily change especially in areas most reviewers don't even look at.

Personally I'd prefer the opposite of what Charb wants, and I'd prefer more games, especially extoic ones that test the non-optimized situations. I agree with the stock vs OC'ed concept, but the reviews doing on 5 games tends to give a very myopic pcture of an ever more complex situation, especially heading into the DX9 vs DX10 and XP vs Vista divides.

The more information the better IMO.
July 12, 2007 5:02:37 AM

TheGreatGrapeApe said:
The more information the better IMO.

bingo. That is why I can even appreciate the biased reviews. If you read it for what it is, you can find info even there.

lol @ 25% grass... I also thought that was retarded.

I agree about using more games. I also agree about using ones that are "non standard". I just don't agree w/ using OLD games for video card tests. WoW? c'mon, that engine was not taxing when it was NEW! lol. FEAR is still ok, and Source can be good in its newer versions. (ep2...?) I do like seeing flightsimX. That one is very different. Oblivion I think is used the best on firingsquad where they have an "indoor" and "outdoor" test. Both gf7 and x1900 were swapping places on those tests for a while and yet it was the same game.

but really, you are right, more info is better.
July 12, 2007 6:31:24 AM

Is it just me or has 3Dmark lost it's marbles too. Some cards out there perform much much better on 3D mark than their counterparts, but then across 90% of real applications they aren't nearly as good.
!