Sign in with
Sign up | Sign in
Your question

Benchmarking suggestion

Tags:
  • Future Article Ideas
  • Performance
  • Benchmark
  • Tom's Hardware
Last response: in Site Feedback
Share
October 18, 2009 1:27:15 PM

PREEDIT: Gah, dear toms hardware, please make it so that when I don't choose a sub-category for my post and then select 'submit' I don't have to re-enter my entire comment text!

Ok so here (again) is my suggestion for benchmark results, which I've been whoring round a few sites, designed to give a fuller picture of component performance than mere minimum/maximum and average results that are fairly common now.

It's the 'sustained performance' chart.

What it shows is the % of time a certain level of performance was achieved during a single benchmark (not an aggregate over several different benchmarks). It might not lean itself towards some benchmark types but should be easily applied to things such as gaming/FPS results. I don't know how easily these stats could be gathered but I'd have thought there must be some FPS tool out there that can output a log of FPS figures which could be hauled in to a spreadsheet for analysis.

Consider 2 graphics cards (I made up) the XYG 260GT and the ABC 3900S.

I dummied up some benchmark results on a spreadsheet. I don't pretend that they're realistic (I just used random number generators and formulae and crazy math functions) but they do make a point and show how these charts might be useful.

The figures show that both complete a benchmark with the same maximum and minimum FPS of 10/82 FPS, the XYG has an average FPS of 47.38 and the ABC 44.91.... so the XYG is the clear winner, right?

Let's take a look at the sustained performance chart:


[<edit> further explanation of how the graph works

If you look along the bottom it shows a certain FPS. The left axis shows the % of time that FPS or higher was achieved.

So if the line for a card/chip passed through the point that corresponds to 30 FPS at the bottom and 90% on the left, it would show that when the benchmark ran, the card achieved 30FPS for 90% of benchmark.

In other words, it only dipped below 30FPS 10% of the time.

So if the benchmark ran for a total of 100 seconds, for 10 of those seconds it ran at less than 30FPS and for 90 seconds it was producing greater than 30FPS.

Make Sense?

</edit>]

Here we can see that, despite it's higher average FPS, the XYG barely manages to limp above 30FPS 30% of the time, spending 20% of the time below about 19 FPS. Perhaps the problems is with a certain graphical effect or the XYG can't cope when a scene comes particularly crowded as the manufacturer, for some unknown reason, despite giving it 1Gb of superfast DDR5 memory, decided to lump it with a 128-bit data bus. Of course such a design mistake would never happen in reality :sarcastic: , but you get my point.

Conversely the ABC has a much smoother performance curve, maintaining 30+ FPS 90% of the time.

So ... which would you rather game with? Ok, I can hear cries of 'neither' from the hardcore enthusiast '100FPS or die!' gamers, but let's pretend this is a Crysis 4 (TBA) benchmark, so 30+ FPS isn't that bad.

Meh, just a thought. Any takers/comments?

It's not uncommon to read a review where the reviewer rates a card highly, making comments about the 'overall experience' or some such, despite seemingly unimpressive benchmark scores, or denigrates a product for 'choppy gameplay'. These charts provide the data to back such claims up.

More about : benchmarking suggestion

October 18, 2009 1:41:07 PM

They do have charts similar in concept.

October 18, 2009 1:44:26 PM

I just checked. Can't see them....
Related resources
October 18, 2009 2:02:22 PM

It's a type of graph with fps on the x-axis and time on the y-axis.

Not sure if Tom's uses it.

Can't find an example right now, sorry.
October 18, 2009 2:05:41 PM

HardOCP show FPS over time, but that's not the same and shows trends/performance less clearly. This is like a summary of those graphs which clearly shows how the card performs overall.
October 18, 2009 2:08:10 PM

I'm really sorry but I don't understand your nice little graph... don't understand how the vertical axis works.
October 18, 2009 2:12:59 PM

If you look along the bottom it shows a certain FPS. The left axis shows the % of time that FPS or higher was achieved.

So if the line for a card/chip passed through the point that corresponds to 30 FPS at the bottom and 90% on the left, it would show that when the benchmark ran, the card achieved 30FPS for 90% of benchmark.

In other words, it only dipped below 30FPS 10% of the time.

So if the benchmark ran for a total of 100 seconds, for 10 of those seconds it ran at less than 30FPS and for 90 seconds it was producing greater than 30FPS.

Make sense?
October 18, 2009 2:35:24 PM

That's a very cool chart. You are really on to something! Keep it up!

I'm positive the Tom's Hardware people would love such a lovely innovative graph design.
October 18, 2009 2:59:05 PM

It's a good idea, but it would be a lot of work on the testing team's part.
October 18, 2009 3:30:43 PM

Shadow703793 said:
It's a good idea, but it would be a lot of work on the testing team's part.


It would take some extra work but FRAPS will output the time at which each frame in the benchmark was rendered in to a log file in CSV format. They could construct a generalised spreadsheet in to which these files could be imported, automatically displaying the required charts (as I expect they do for existing benchmark figures).

From these times the intervals between frames can be calculated and from these the number of FPS each frame effectively gave rise to.

FPS = 1000/(interval between frames in ms).

On each row of the spread sheet we'd have 2 important columns, one for interval between frames and one for FPS.

Add a third column to each row containing a formula to SUMIF all the interval times that are lower or equal it's the interval time and you get the total time spent at each FPS or higher.

The formulae used would be something like:

=SUMIF(<entire interval data column for all frames>, CONCATENATE("<="; <this row's interval data cell>))

Divide this by the duration of the benchmark and you get the % time the benchmark ran at a specific FPS or faster.

Plot this on an XY or Line graph, with X as FPS and the % as Y, sorted by X values (which open office can do) and you have your sustained performance graph. All this, other than the importing of the data in to the spreadsheet, could be automated. If course the spreadsheet would have to be fancier to reduce the workload further and allow the handling of more than one data source, but I'm trying to outline the general principles here.
October 18, 2009 4:45:01 PM

Quote:
FRAPS will output the time at which each frame in the benchmark was rendered in to a log file in CSV format.

I did not know that. In that case, it's not that much work.
!