Web Browser Grand Prix: Chrome 18, Firefox 11, Windows XP

by
61 Comments

WBGP10 Test Suite And Methodology

We restart the computer and allow it to idle for a few minutes before benchmarking each browser. Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variance. 

All tests are placed into one of four groups: core, observation, dated, and quarantine. Core tests are considered current. These tests are usually trusted industry standards or our own creations, and they make up the foundation of the WBGP suite. Tests that are either generally unknown, mostly untested, or just too bleeding-edge are placed under observation. Tests classified as dated are either outdated, losing relevance, or otherwise need replacing. We are actively seeking community feedback and contributions regarding alternatives to these benchmarks. The final group is for quarantined benchmarks. Benchmarks find their way into quarantine by delivering dubious results or by being gamed. Whenever benchmarks that test the same thing yield conflicting results, more weight is given to tests with a better rating when creating the analysis tables.

The table below lists all 51 of the tests currently in our suite (along with a version number and link, where applicable), number of iterations performed, and current rating:

Tom's Hardware Web Browser Grand Prix Test Suite v9.0
Test Name
Iterations
Rating
Performance Tests (44)
Cold Startup Time: Single Tab
3
Core
Cold Startup Time: Eight Tabs3
Core
Hot Startup Time: Single Tab3
Core
Hot Startup Time: Eight Tabs
3
Core
Uncached Page Load Times (9 Test Pages)
5
Core
Cached Page Load Times (9 Test Pages)
5
Core
Kraken v1.1
2
Core
Google SunSpider v0.91 Mod2
Core
FutureMark Peacekeeper 2.02
Core
Dromaeo DOM2
Core
Maze Solver5
Core
GUIMark2 Flash Vector Charting3
Core
GUIMark2 Flash Bitmap Gaming3
Core
GUIMark2 Flash Text Columns3
Core
Flash Benchmark 2008 v1.09.1
2
Core
GUIMark Java3
Dated
Encog Silverlight3
Dated
Facebook JSGameBench v4.12
Core
GUIMark 2 HTML5 Vector Charting (1 pixel variant)
3
Core
GUIMark 2 HTML5 Bitmap Gaming
3
Core
GUIMark 2 HTML5 Text Columns
3
Core
Asteroids HTML5 Canvas 2D And JavaScript2
Observation
Psychedelic Browsing
2
Core
Hardware Acceleration Stress Test
2
Dated
Mozilla WebGL FishIE
5
Core
WebGL Solar System5
Observation
Efficiency Benchmarks (5)
Memory Usage: Single Tab
3
Core
Memory Usage: 40 Tabs
3
Core
Memory Management: -39 Tabs
3
Core
Memory Management: -39 Tabs (extra 2 minutes)
3
Core
Reliability Benchmarks (1)
Proper Page Loads
3
Core
Conformance Benchmarks (3)
Ecma test2621
Core
Peacekeeper 2.0 HTML5 Capabilities1
Core
HTML5Test.com1
Core


Detailed individual methodologies are described on the pages corresponding to each benchmark.

Previous Next
Summary
  1. Tom's Hardware's Tenth Web Browser Grand Prix
  2. Chrome, Firefox, Internet Explorer, Opera, And Safari
  3. Test System Specs And Software Setup
  4. WBGP10 Test Suite And Methodology
  5. Startup Time Performance Benchmarks
  6. Page Load Time Performance Benchmarks
  7. JavaScript Performance Benchmarks
  8. DOM And CSS Performance Benchmarks
  9. Flash Performance Benchmarks
  10. Java And Silverlight Performance Benchmarks
  11. HTML5 Performance Benchmarks
  12. Hardware Acceleration And WebGL
  13. Memory Efficiency Benchmarks
  14. Proper Page Load Reliability
  15. Standards Conformance Benchmarks
  16. Benchmark Analysis
  17. Crowning A Windows XP Champion
About the author
Adam Overa

Adam Overa was the Managing Editor of Tom's Hardware US from 2012-2017

Read more
This thread is closed for comments
61 comments
Comment from the forums
    Your comment
  • wheredahoodat
    "Both Opera and Chrome feltmuch smoother on our old PC than Firefox"

    I do kinda feel the difference with Firefox's responsive going from my main modern desktop to my older labtop that has regulated to a makeshift HTC. I believe Firefox XUL interface is the culprit; it was a big enough problem for Firefox mobile to abandon it in favor of native Android GUI, but who knows at this point. I guess might actually give Opera a chance.
  • agnickolov
    How come only a single reader requested numerical composite scoring, that's the most logical way of scoring after all! With that said, I'd have liked if you didn't use the rankings but the raw scores after a more intelligent transformation as the input for weighted averaging...
    For example, for each category you could subtract the lowest-placed score from all scores and then normalize in the range [0-1] by dividing all adjusted scores by the topmost adjusted score. This way the top perfomer always has 1 and the worst performer always has 0 modified score (you'd need to invert them for tests where lower is better of course, e.g. subtract these from 1). Then apply your ranks to these scores and you get the composite score. It's not a perfect transformation, but it certainly has more fairly distributed weight (pun intended) than what you have used here.
  • gwiz1987
    Why is IE8 being benched and not IE9?
  • aznjoka
    Thats my Opera, for those who have never tried Opera. It's an amazing piece of software, it does the job, and it does it better then most.
  • confish21
    Interesting move to make this article. Well done! Don't waste your time on a vista run though... Im so close my release date. xD
  • csbeer
    aznjokaThats my Opera, for those who have never tried Opera. It's an amazing piece of software, it does the job, and it does it better then most.


    XP can't run 9. Need to upgrade OS in order to get higher IE.
  • dameon51
    gwiz1987Why is IE8 being benched and not IE9?

    XP doesn't support 9, only 8.
  • mayankleoboy1
    excellent review!
    some points:

    1.A lot of corporates still use IE7. maybe you should include that too in your benchmarks

    2.if you remove HTML5 (with and without H/W acceleration), i think Opera's victory margin will be quite huge.

    3.Regarding smoothness, i beleive FF is quite poor in this. But the developers know about it and are very activle working on it. I thik FF13 will be the release when smoothness will improve. look at "Firefox Snappy".

    4. i would like to have a subjective recommendation at the end of the article, something you subjectively felt was the best amongst all the browsers, even though it may be trailing in numbers.
  • mayankleoboy1
    Why did you use the AGP? I bet 99.99% of those Pentium4 era computers use the onboard Intel IGP.
    Also that would definitely disable the H/W acceleration of browsers.
  • Anyone who is still stuck using Windows 2000, Opera supports you.

    http://www.opera.com/browser/download/requirements/
  • bunnywannyAnyone who is still stuck using Windows 2000, Opera supports you.

    Toms, the "add an url" in the comment toolbar doesn't work. Here is the link:
    http://www.opera.com/browser/download/requirements/
  • ronch79
    I don't know about you guys, but I've been a fan of Opera for a few years now, until recently. I've noticed that the then-latest version, 11.61, took so long to load pages. At first, I thought there was something wrong with my internet connection. For some reason I installed Chrome. I noticed right away that browsing with Chrome was faster (pages loaded quickly). My first suspicion was that it was just plain luck; that the bandwidth simply was faster coincidentally when I was using Chrome. I did a side-by-side comparison with both browsers open, and yes, Opera did load pages sluggishly. I was stumped. For the record, I'm NOT bashing Opera here, folks. I just don't know why this is happening. Heck, I'd switch back to Opera in a flash. This all happened with Opera 11.61. I think I'll give Opera 12 a shot.
  • straatkat
    The html5 ranking is surprising. A score north of 300 for HTML5 support is, I would say, about par. In the end, HTML5 adoption in the wild is a good as the lowest score, because you want people to have access to your site, you are not going to build a site that locks out a substantial portion of the internet. So you are not going to target a HTML5 feature that is specific to a browser.
  • hellfire24
    man i still love XP!!!!
  • for me 0.5 ponts is noting and for me chrome is the winner here
  • straatkat
    Can you state which version of Windows XP you were using? Circa 2003 you had Windows XP service pack 1 only.
  • assassin123
    I love firefox
  • Cryio
    muhsi44even though the OSes are still in beta.


    What OSes?
  • Hypertraxx
    CHROMEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
  • sheol
    Out of the box, no browser that I know of comes close to Opera when it comes to usability(mouse gestures, tab manangement, "Closed Tabs " - a recycle bin of tabs if you will, an awesome feature that keeps your closed tabs for easy retrieval for when you might need them.
    This is why I believe Opera uses so much memory after 39 tabs have been closed.

    And reliability, who could complain? Routinely i have 50+ tabs open for days, and I have not had a single crash because of it.
  • Tamz_msc
    Yes, Opera is the best browser out-of-the-box, but I've seen that sometimes it takes a long time to load a page and there is no way to tell if its done or not. On complex websites it keeps showing Elements 87/88 and it seems like its stuck at that point.

    Anyways, Firefox is my primary browser and I use Opera as a backup.
  • trumpeter1994
    I use firefox still because in my opinion it's like a swiss army knife among browsers. I have chrome and use it on the rare occasion but I will definetely be getting opera now, because I'm very impressed with it's performance even on an outdated system.
    I do think its unfair though to compare IE8 to the newer versions of the other browsers, but theres not much you can do about that on this test system so I'll just disregard it.
  • LaloFG
    This is a better approach qualifying browsers, at least now you're mention the feeling in responsiveness.
    Opera deserve this win (and others that it lost for bad testing in tomshardware), maybe someday will caught a bigger quota.
  • tipoo
    I'm glad you included the responsiveness results, to me it never mattered if Firefox or other browsers met or surpassed Chrome and Opera in benchmarks, the fact was that Chrome and Opera just stay responsive no matter what you throw at them which no other browser has yet come close to.
Display more comments
Most Popular
  1. Killer Wireless-N 1103 Review: Can Qualcomm Take On Centrino?
  2. 16-year-old Makes Working Scientific Calculator in Minecraft
  3. The Top 15 Best-Selling PC Games Of All Time
The Latest On Tom's Hardware
Edition
Subscribe to our newsletter
Company
Resources
Other Purch sites
  • © 2019 Purch All Rights Reserved.