Sign in with
Sign up | Sign in

Test Suite And Methodology

Web Browser Grand Prix: Firefox 15, Safari 6, OS X Mountain Lion
By

Changes

The Start and Page Load test pages are updated to current versions, with the exception of the Tom's Hardware Wikipedia page, which has not changed since the last update. We say goodbye to YouTube, eBay, and The Huffington Post. In their place is an About.com page on barbeque, a randomly-selected popular question on Ask.com, and my own LinkedIn profile page. The Google homepage is replaced with the search results page for "Tom's Hardware", and craigslist is now the "free stuff" results page for Los Angeles. Amazon remains as the Computer Parts & Components page, though we update it. And finally, the Yahoo homepage is updated and now serves as the single-tab test page in place of the old Google homepage.

We're introducing the remaining RIABench JavaScript, Flash, Java, and Silverlight tests. RIABench JavaScript consists of eight tests, Java has seven tests, while Flash and Silverlight have all ten tests. The final RIABench scores are now the geometric mean of individual tests, instead of simple averages.

CSS Stress Testing & Performance Profiling is now performed on the CSS version of the CSS3 Speed Test demo page. Microsoft's Maze Solver CSS3 benchmark is being retired in favor of the bookmarklet applied to the CCS3 version of the very same CSS3 Speed Test demo page.

We're also adding The CSS3 Test to our standards conformance tests. It replaces the HTML5 Capabilities section of Futuremark Peacekeeper 2.0. HTML5Test.com is now the sole HTML5 conformance test in the standards conformance composite, providing an even split between JavaScript, HTML5, and CSS3 in our final standards conformance grade.

Last but not least, the JavaScript Composite score is also being switched to geometric mean instead of the inverse averages we used in the previous installment.

Web Browser Grand Prix Test Suite v12

The table below lists all 34 benchmarks (consisting of 77 individual tests) currently in our suite, along with the version number and link (where applicable), and the number of iterations performed.

Benchmark Name
Iterations Performed
Performance Benchmarks (24 Benchmarks, 67 Tests)
Cold Startup Time: Single Tab
3
Hot Startup Time: Single Tab3
Cold Startup Time: Eight Tabs3
Hot Startup Time: Eight Tabs
3
Uncached Page Load Times (Eight Test Pages)
5
Cached Page Load Times (Eight Test Pages)
5
RIABench JavaScript (Eight Tests)
3
Mozilla Kraken v1.1
2
Google SunSpider v0.9.1 Mod2
FutureMark Peacekeeper 2.02
Dromaeo DOM Core
1
CSS Stress Test and Performance Profiling - CSS Speed Test2
CSS Stress Test and Performance Profiling - CSS3 Speed Test
2
GUIMark 2 HTML5 (3 Tests)
3
Asteroids HTML5 Canvas 2D And JavaScript2
HTML5 Canvas Performance Test
2
Facebook JSGameBench v4.12
Psychedelic Browsing2
WebVizBench
2
Mozilla WebGL FishIE2
WebGL Solar System2
RIABench Flash (10 Tests)
3
RIABench Java (7 Tests)
3
RIABench Silverlight (10 Tests)
3
Efficiency Benchmarks (Four Benchmarks/Tests)
Memory Usage: Single Tab
3
Memory Usage: 40 Tabs
3
Memory Management: -39 Tabs
3
Memory Management: -39 Tabs (extra 2 minutes)
3
Reliability Benchmarks (One Test)
Proper Page Loads
3
Responsiveness Benchmarks (One Test)
General Responsiveness Under Load
3
Security Benchmarks (One Test)
BrowserScope Security
1
Conformance Benchmarks (Three Benchmarks/Tests)
Ecma Language test2621
HTML5Test.com1
The CSS3 Test
1


Methodology

We restart the computer and allow it to idle before benchmarking. Most individual benchmark final scores are an average of several iterations. More iterations are run for tests that have short durations, lower scales, and/or higher variance. Any obvious outliers (usually network hiccups) are removed and retested.

We switched most of the composite scores from arithmetic mean (average) to geometric mean in order to ensure that every test in each category is given equal weight, regardless of absolute value. The exceptions are the Standards Conformance grade and Memory Efficiency score, which are achieved differently, as well as the Reliability, Responsiveness, and Security composites, each of which only contains a single test.

Individual detailed methodologies and information regarding composite scoring is described on the corresponding benchmark pages.

Display all 57 comments.
This thread is closed for comments
Top Comments
  • 21 Hide
    Eggrenade , September 7, 2012 4:58 AM
    It would be nice if I could view the additional charts with only one click, and not in a separate window.
  • 16 Hide
    adamovera , September 7, 2012 5:50 AM
    JOSHSKORNHow about 64-bit Internet Explorer 9 vs Waterfox 15.0?

    When we have more [official] stable 64-bit browsers, I'll definitely do a 64-bit WBGP - including versus their 32-bit counterparts.
  • 14 Hide
    adamovera , September 7, 2012 7:43 AM
    nitriumSo OSX is really just a LOT slower than Win7 generally for web browsing on the identical hardware. Is that right?

    Nearly every performance benchmark there is points in that direction. This probably has a lot to do with how much time developers spend optimizing for Windows - after all, Windows holds 90+% of the desktop user base. However, it is interesting that the rift between Windows and OS X is far greater than between Windows and Linux for the core stuff like JS, CSS, DOM, page loads, etc. Plug-ins are another story, they're always much better on Windows than the other two platforms.
Other Comments
  • 21 Hide
    Eggrenade , September 7, 2012 4:58 AM
    It would be nice if I could view the additional charts with only one click, and not in a separate window.
  • 7 Hide
    lahawzel , September 7, 2012 4:59 AM
    It's nice to see Chrome performing so well, but I'm still waiting on the Chrome equivalents of all the plugins I use in FF before I think about switching. The web just doesn't feel the same without them.

    (The nice popular ones like ABP, Lazarus, Greasemonkey all have equivalents; some lesser-used plugins like Rikaichan also have ports by now. Only a matter of time!)
  • 5 Hide
    bennaye , September 7, 2012 5:03 AM
    chrome is absolutely deserving of the award. say what you will about the frequent patch releases touted as upgrades, chrome is a very good browser, as shown by this month's article. even on OSX there is only a small margin separating chrome and safari. but the one qualm i do have with chrome is the lack of add-ons compared to firefox. and i a lot of people share this concern. the add-ons do make the experience that much better.

    as always, a great read.
  • 12 Hide
    Anonymous , September 7, 2012 5:15 AM
    Would like to see this again after IE10 is released.
  • 11 Hide
    JOSHSKORN , September 7, 2012 5:42 AM
    How about 64-bit Internet Explorer 9 vs Waterfox 15.0?
  • 5 Hide
    adamovera , September 7, 2012 5:47 AM
    bennayechrome is absolutely deserving of the award. say what you will about the frequent patch releases touted as upgrades, chrome is a very good browser, as shown by this month's article. even on OSX there is only a small margin separating chrome and safari. but the one qualm i do have with chrome is the lack of add-ons compared to firefox. and i a lot of people share this concern. the add-ons do make the experience that much better.as always, a great read.

    All versions of Chrome hold up incredibly well cross-platform, if you look back at the two Linux WBGPs, it won there, too. Thanks for reading!
  • 12 Hide
    adamovera , September 7, 2012 5:49 AM
    AdamsTaiwanWould like to see this again after IE10 is released.

    Absolutely, a Windows 8-based WBGP is already in the cards for October.
  • 16 Hide
    adamovera , September 7, 2012 5:50 AM
    JOSHSKORNHow about 64-bit Internet Explorer 9 vs Waterfox 15.0?

    When we have more [official] stable 64-bit browsers, I'll definitely do a 64-bit WBGP - including versus their 32-bit counterparts.
  • 8 Hide
    Anonymous , September 7, 2012 6:22 AM
    I wish Tom's would fiddle around with the settings of these browsers for these tests. In every System Builder Marathon you overclock the builds, why not try and crank the most speed while ensuring better memory management out of the browser as well?

    Testing these browsers at stock doesn't reveal even an eighth of the picture.
  • 14 Hide
    nitrium , September 7, 2012 7:19 AM
    So OSX is really just a LOT slower than Win7 generally for web browsing on the identical hardware. Is that right?
  • 4 Hide
    shahrooz , September 7, 2012 7:20 AM
    it would be nice to have unreleased(Beta/Aurora/etc) versions of the browsers in benchmarks and scoring but only stable releases in the crowning so it will be fair. many users opt for beta releases of browsers including myself, I use Firefox nightly(18.0a1)

    btw great work adamovera keep it up man
  • 13 Hide
    adamovera , September 7, 2012 7:29 AM
    boopyI wish Tom's would fiddle around with the settings of these browsers for these tests. In every System Builder Marathon you overclock the builds, why not try and crank the most speed while ensuring better memory management out of the browser as well?Testing these browsers at stock doesn't reveal even an eighth of the picture.

    Interesting idea, so basically a tweaked-out edition of the WBGP, where we use all the tools available to each browser for performance gains... That could work, but I gotta warn you that the next three WBGPs are already decided, so it would probably be real late in the year, or even next year before I could get to it.
  • 14 Hide
    adamovera , September 7, 2012 7:43 AM
    nitriumSo OSX is really just a LOT slower than Win7 generally for web browsing on the identical hardware. Is that right?

    Nearly every performance benchmark there is points in that direction. This probably has a lot to do with how much time developers spend optimizing for Windows - after all, Windows holds 90+% of the desktop user base. However, it is interesting that the rift between Windows and OS X is far greater than between Windows and Linux for the core stuff like JS, CSS, DOM, page loads, etc. Plug-ins are another story, they're always much better on Windows than the other two platforms.
  • 11 Hide
    merikafyeah , September 7, 2012 7:44 AM
    All things considered, IE9 does pretty well for a browser that's not even in the double-digits.
  • 7 Hide
    adamovera , September 7, 2012 7:58 AM
    shahroozit would be nice to have unreleased(Beta/Aurora/etc) versions of the browsers in benchmarks and scoring but only stable releases in the crowning so it will be fair. many users opt for beta releases of browsers including myself, I use Firefox nightly(18.0a1)btw great work adamovera keep it up man

    The big problem with including the dev channel browsers is the amount of time it takes to produce the article (testing/charts/writing/editing/translating), combined with the tendency of the dev channel to constantly update. Before testing is even completed it's certain that something will update. TBH, the stable channels of Chrome and Firefox are a handful as it is. For example, for this article I had to test 8 browsers (4 on each OS), but I ended up testing 18+ due to OS X, Chrome, Firefox, Opera, Flash, and Java updates. Sorry, but I'm just not sure it's even doable in this format. Thanks for reading!
  • 1 Hide
    shahrooz , September 7, 2012 9:16 AM
    adamoveraThe big problem with including the dev channel browsers is the amount of time it takes to produce the article (testing/charts/writing/editing/translating), combined with the tendency of the dev channel to constantly update. Before testing is even completed it's certain that something will update. TBH, the stable channels of Chrome and Firefox are a handful as it is. For example, for this article I had to test 8 browsers (4 on each OS), but I ended up testing 18+ due to OS X, Chrome, Firefox, Opera, Flash, and Java updates. Sorry, but I'm just not sure it's even doable in this format. Thanks for reading!

    well I wanted to include it in my comment myself but I forgot I wanted to say if the timing allows :) 
  • 2 Hide
    Anonymous , September 7, 2012 9:31 AM
    Meaningless tests designed to generate clicks. All browsers are mature enough at this point in terms of performance. Even the lowly IE is OK. What matters is who is the browser working for, the user, or the mothership? And that's where FF is king. It's about treacherous, gimmicky commercial software made to push standards lock-in and data mining your information, versus free open source software that serves YOUR needs. And it's not only about the code being open either (Chrome). Get savvy on this stuff. Performance metrics are for FPS nerd teenagers.
  • 2 Hide
    JoeMomma , September 7, 2012 10:42 AM
    Call me Old Skool but for me functionality and ease of use far outweigh speed.
    My computer is fast enough that it does not really mater what browser I choose.
    In my case, ease of use means that I can see what is going on.
    I decry the trend towards dumbing down the UI on every program I use.
    (I also refuse to call software 'Apps', to me an app is a mini-program on a phone.)
    I always turn on all menus, buttons and labels in WaterFox.

    BTW: Good point.
    Why don't you include WaterFox in your testing?
    It is the 64 bit version of FireFox and I am sure that in your speed tests it may do a little better.
  • 0 Hide
    azathoth , September 7, 2012 10:47 AM
    Internet Explorer is starting to look pretty good in terms of speed.
Display more comments