Web Browser Grand Prix: Chrome 18, Firefox 11, Windows XP

WBGP10 Test Suite And Methodology

We restart the computer and allow it to idle for a few minutes before benchmarking each browser. Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variance. 

All tests are placed into one of four groups: core, observation, dated, and quarantine. Core tests are considered current. These tests are usually trusted industry standards or our own creations, and they make up the foundation of the WBGP suite. Tests that are either generally unknown, mostly untested, or just too bleeding-edge are placed under observation. Tests classified as dated are either outdated, losing relevance, or otherwise need replacing. We are actively seeking community feedback and contributions regarding alternatives to these benchmarks. The final group is for quarantined benchmarks. Benchmarks find their way into quarantine by delivering dubious results or by being gamed. Whenever benchmarks that test the same thing yield conflicting results, more weight is given to tests with a better rating when creating the analysis tables.

The table below lists all 51 of the tests currently in our suite (along with a version number and link, where applicable), number of iterations performed, and current rating:

Swipe to scroll horizontally
Tom's Hardware Web Browser Grand Prix Test Suite v9.0
Test NameIterationsRating
Performance Tests (44)
Cold Startup Time: Single Tab3Core
Cold Startup Time: Eight Tabs3Core
Hot Startup Time: Single Tab3Core
Hot Startup Time: Eight Tabs3Core
Uncached Page Load Times (9 Test Pages)5Core
Cached Page Load Times (9 Test Pages)5Core
Kraken v1.12Core
Google SunSpider v0.91 Mod2Core
FutureMark Peacekeeper 2.02Core
Dromaeo DOM2Core
Maze Solver5Core
GUIMark2 Flash Vector Charting3Core
GUIMark2 Flash Bitmap Gaming3Core
GUIMark2 Flash Text Columns3Core
Flash Benchmark 2008 v1.09.12Core
GUIMark Java3Dated
Encog Silverlight3Dated
Facebook JSGameBench v4.12Core
GUIMark 2 HTML5 Vector Charting (1 pixel variant)3Core
GUIMark 2 HTML5 Bitmap Gaming3Core
GUIMark 2 HTML5 Text Columns3Core
Asteroids HTML5 Canvas 2D And JavaScript2Observation
Psychedelic Browsing2Core
Hardware Acceleration Stress Test2Dated
Mozilla WebGL FishIE5Core
WebGL Solar System5Observation
Efficiency Benchmarks (5)
Memory Usage: Single Tab3Core
Memory Usage: 40 Tabs3Core
Memory Management: -39 Tabs3Core
Memory Management: -39 Tabs (extra 2 minutes)3Core
Reliability Benchmarks (1)
Proper Page Loads3Core
Conformance Benchmarks (3)
Ecma test2621Core
Peacekeeper 2.0 HTML5 Capabilities1Core
HTML5Test.com1Core

Detailed individual methodologies are described on the pages corresponding to each benchmark.

TOPICS
  • wheredahoodat
    "Both Opera and Chrome feltmuch smoother on our old PC than Firefox"

    I do kinda feel the difference with Firefox's responsive going from my main modern desktop to my older labtop that has regulated to a makeshift HTC. I believe Firefox XUL interface is the culprit; it was a big enough problem for Firefox mobile to abandon it in favor of native Android GUI, but who knows at this point. I guess might actually give Opera a chance.
    Reply
  • agnickolov
    How come only a single reader requested numerical composite scoring, that's the most logical way of scoring after all! With that said, I'd have liked if you didn't use the rankings but the raw scores after a more intelligent transformation as the input for weighted averaging...
    For example, for each category you could subtract the lowest-placed score from all scores and then normalize in the range by dividing all adjusted scores by the topmost adjusted score. This way the top perfomer always has 1 and the worst performer always has 0 modified score (you'd need to invert them for tests where lower is better of course, e.g. subtract these from 1). Then apply your ranks to these scores and you get the composite score. It's not a perfect transformation, but it certainly has more fairly distributed weight (pun intended) than what you have used here.
    Reply
  • gwiz1987
    Why is IE8 being benched and not IE9?
    Reply
  • aznjoka
    Thats my Opera, for those who have never tried Opera. It's an amazing piece of software, it does the job, and it does it better then most.
    Reply
  • confish21
    Interesting move to make this article. Well done! Don't waste your time on a vista run though... Im so close my release date. xD
    Reply
  • csbeer
    aznjokaThats my Opera, for those who have never tried Opera. It's an amazing piece of software, it does the job, and it does it better then most.
    XP can't run 9. Need to upgrade OS in order to get higher IE.
    Reply
  • dameon51
    gwiz1987Why is IE8 being benched and not IE9?XP doesn't support 9, only 8.
    Reply
  • mayankleoboy1
    excellent review!
    some points:

    1.A lot of corporates still use IE7. maybe you should include that too in your benchmarks

    2.if you remove HTML5 (with and without H/W acceleration), i think Opera's victory margin will be quite huge.

    3.Regarding smoothness, i beleive FF is quite poor in this. But the developers know about it and are very activle working on it. I thik FF13 will be the release when smoothness will improve. look at "Firefox Snappy".

    4. i would like to have a subjective recommendation at the end of the article, something you subjectively felt was the best amongst all the browsers, even though it may be trailing in numbers.
    Reply
  • mayankleoboy1
    Why did you use the AGP? I bet 99.99% of those Pentium4 era computers use the onboard Intel IGP.
    Also that would definitely disable the H/W acceleration of browsers.

    Reply
  • Anyone who is still stuck using Windows 2000, Opera supports you.

    Reply