Sign in with
Sign up | Sign in

Test Analysis

Web Browser Grand Prix: Firefox 15, Safari 6, OS X Mountain Lion
By

Finishes

Each category of testing has four columns: Winner, Strong, Average, and Weak. The Winner is obviously the browser that achieves the highest scores in that category. The Strong column is for browsers exhibiting superior performance, but not achieving a first-place victory. Average is for browsers that perform adequately or in-line with a majority of their competitors. A Weak finish is assigned to browsers that perform poorly, or substantially lower than their competitors.

Brackets

In order to reflect how each category of testing affects the average end-user Web browsing experience, we need to create brackets (or levels of importance) to place the different categories of testing into.

Essential
CSS, DOM, JavaScript, Reliability, Standards Conformance
Important
Flash, HTML5, Memory Efficiency, Page Load Time, Responsiveness, Security, Startup Time
Nonessential
Java, Silverlight
Unimportant
HTML5 Hardware Acceleration, WebGL


The Essential bracket contains those categories of testing that are indispensable to rendering the vast majority of Web pages online today. The Important bracket is for categories not quite essential to browsing the Web, yet still affect the user experience to a great degree. The Nonessential bracket contains the popular plug-ins Java and Silverlight. While these plug-in technologies are nowhere near as ubiquitous as Flash, certain applications like corporate intranet apps and Netflix simply will not work without them. Finally, the Unimportant bracket is for emerging technologies, such as HTML5 Hardware Acceleration and WebGL, which still don't really exist outside of testing/demo sites.

Points

Now that the brackets are all sorted out, we can apply a numerical point system to the finishes of each bracket.


Winner
Strong
Average
Weak
Essential
2.5
2
1.5
-2
Important
2
1.5
1
-1.5
Nonessential
1.5
1
0.5
-1
Unimportant
1
0.5
0
-0.5


As you can see, we decided to apply negative point values to the Weak finishes and start the Average performances at zero for the Unimportant bracket. The Winner has also been de-emphasized over Strong finishes, with just a small tie-breaking bonus going to Winner.

OS X 10.8 Analysis Table


Winner
Strong
Average
Weak
Essential
CSS
Firefox

Chrome, Safari
Opera
DOM
Firefox

Chrome, Safari
Opera
JavaScript
Chrome
Safari
Firefox, Opera

Reliability
Opera

Firefox, Safari
Chrome
Standards Conformance
Chrome

Firefox, Opera, Safari

Important
Flash
Firefox, Opera, Safari
Chrome


HTML5
Safari

Chrome
Firefox, Opera
Memory Efficiency
Chrome

Firefox, Opera, Safari

Page Load Time
Chrome

Firefox, Opera, Safari

Responsiveness
Opera
Firefox, Safari
Chrome

Security
Chrome
Safari
Firefox
Opera
Startup Time
Chrome
Safari
Firefox
Opera
Nonessential
Java
Chrome, Firefox, Opera, Safari



Silverlight
Chrome, Firefox, Opera, Safari


Unimportant
HTML5 Hardware Acceleration
Safari

Chrome
Firefox, Opera
WebGL
Firefox

Chrome
Opera, Safari


Now, let's see how the Windows 7 Web browsers compare to each other.

Windows 7 Analysis Table


Winner
Strong
Average
Weak
Essential
CSS
Firefox

Chrome, IE
Opera
DOM
Chrome

Firefox, IE
Opera
JavaScript
Chrome

Firefox, Opera
IE
Reliability
Opera

Chrome, Firefox, IE

Standards Conformance
Chrome

Firefox, Opera
IE
Important
Flash
Firefox, IE, Opera
Chrome


HTML5
IE

Chrome, Firefox
Opera
Memory Efficiency
Chrome
IE, Firefox
Opera

Page Load Time
IE

Chrome, Firefox, Opera

Responsiveness
Opera
Firefox
Chrome
IE
Security
Chrome

Firefox, IE
Opera
Startup Time
Chrome
Opera
Firefox, IE

Nonessential
Java
Chrome, Firefox, IE, Opera



Silverlight
Firefox
Chrome, IE, Opera


Unimportant
HTML5 Hardware Acceleration
Firefox
IE
Chrome
Opera
WebGL
Firefox

Chrome
IE, Opera


And the winners are...

Ask a Category Expert

Create a new thread in the Reviews comments forum about this subject

Example: Notebook, Android, SSD hard drive

Display all 57 comments.
This thread is closed for comments
Top Comments
  • 21 Hide
    Eggrenade , September 7, 2012 4:58 AM
    It would be nice if I could view the additional charts with only one click, and not in a separate window.
  • 16 Hide
    adamovera , September 7, 2012 5:50 AM
    JOSHSKORNHow about 64-bit Internet Explorer 9 vs Waterfox 15.0?

    When we have more [official] stable 64-bit browsers, I'll definitely do a 64-bit WBGP - including versus their 32-bit counterparts.
  • 14 Hide
    adamovera , September 7, 2012 7:43 AM
    nitriumSo OSX is really just a LOT slower than Win7 generally for web browsing on the identical hardware. Is that right?

    Nearly every performance benchmark there is points in that direction. This probably has a lot to do with how much time developers spend optimizing for Windows - after all, Windows holds 90+% of the desktop user base. However, it is interesting that the rift between Windows and OS X is far greater than between Windows and Linux for the core stuff like JS, CSS, DOM, page loads, etc. Plug-ins are another story, they're always much better on Windows than the other two platforms.
Other Comments
  • 21 Hide
    Eggrenade , September 7, 2012 4:58 AM
    It would be nice if I could view the additional charts with only one click, and not in a separate window.
  • 7 Hide
    lahawzel , September 7, 2012 4:59 AM
    It's nice to see Chrome performing so well, but I'm still waiting on the Chrome equivalents of all the plugins I use in FF before I think about switching. The web just doesn't feel the same without them.

    (The nice popular ones like ABP, Lazarus, Greasemonkey all have equivalents; some lesser-used plugins like Rikaichan also have ports by now. Only a matter of time!)
  • 5 Hide
    bennaye , September 7, 2012 5:03 AM
    chrome is absolutely deserving of the award. say what you will about the frequent patch releases touted as upgrades, chrome is a very good browser, as shown by this month's article. even on OSX there is only a small margin separating chrome and safari. but the one qualm i do have with chrome is the lack of add-ons compared to firefox. and i a lot of people share this concern. the add-ons do make the experience that much better.

    as always, a great read.
  • 12 Hide
    Anonymous , September 7, 2012 5:15 AM
    Would like to see this again after IE10 is released.
  • 11 Hide
    JOSHSKORN , September 7, 2012 5:42 AM
    How about 64-bit Internet Explorer 9 vs Waterfox 15.0?
  • 5 Hide
    adamovera , September 7, 2012 5:47 AM
    bennayechrome is absolutely deserving of the award. say what you will about the frequent patch releases touted as upgrades, chrome is a very good browser, as shown by this month's article. even on OSX there is only a small margin separating chrome and safari. but the one qualm i do have with chrome is the lack of add-ons compared to firefox. and i a lot of people share this concern. the add-ons do make the experience that much better.as always, a great read.

    All versions of Chrome hold up incredibly well cross-platform, if you look back at the two Linux WBGPs, it won there, too. Thanks for reading!
  • 12 Hide
    adamovera , September 7, 2012 5:49 AM
    AdamsTaiwanWould like to see this again after IE10 is released.

    Absolutely, a Windows 8-based WBGP is already in the cards for October.
  • 16 Hide
    adamovera , September 7, 2012 5:50 AM
    JOSHSKORNHow about 64-bit Internet Explorer 9 vs Waterfox 15.0?

    When we have more [official] stable 64-bit browsers, I'll definitely do a 64-bit WBGP - including versus their 32-bit counterparts.
  • 8 Hide
    Anonymous , September 7, 2012 6:22 AM
    I wish Tom's would fiddle around with the settings of these browsers for these tests. In every System Builder Marathon you overclock the builds, why not try and crank the most speed while ensuring better memory management out of the browser as well?

    Testing these browsers at stock doesn't reveal even an eighth of the picture.
  • 14 Hide
    nitrium , September 7, 2012 7:19 AM
    So OSX is really just a LOT slower than Win7 generally for web browsing on the identical hardware. Is that right?
  • 4 Hide
    shahrooz , September 7, 2012 7:20 AM
    it would be nice to have unreleased(Beta/Aurora/etc) versions of the browsers in benchmarks and scoring but only stable releases in the crowning so it will be fair. many users opt for beta releases of browsers including myself, I use Firefox nightly(18.0a1)

    btw great work adamovera keep it up man
  • 13 Hide
    adamovera , September 7, 2012 7:29 AM
    boopyI wish Tom's would fiddle around with the settings of these browsers for these tests. In every System Builder Marathon you overclock the builds, why not try and crank the most speed while ensuring better memory management out of the browser as well?Testing these browsers at stock doesn't reveal even an eighth of the picture.

    Interesting idea, so basically a tweaked-out edition of the WBGP, where we use all the tools available to each browser for performance gains... That could work, but I gotta warn you that the next three WBGPs are already decided, so it would probably be real late in the year, or even next year before I could get to it.
  • 14 Hide
    adamovera , September 7, 2012 7:43 AM
    nitriumSo OSX is really just a LOT slower than Win7 generally for web browsing on the identical hardware. Is that right?

    Nearly every performance benchmark there is points in that direction. This probably has a lot to do with how much time developers spend optimizing for Windows - after all, Windows holds 90+% of the desktop user base. However, it is interesting that the rift between Windows and OS X is far greater than between Windows and Linux for the core stuff like JS, CSS, DOM, page loads, etc. Plug-ins are another story, they're always much better on Windows than the other two platforms.
  • 11 Hide
    merikafyeah , September 7, 2012 7:44 AM
    All things considered, IE9 does pretty well for a browser that's not even in the double-digits.
  • 7 Hide
    adamovera , September 7, 2012 7:58 AM
    shahroozit would be nice to have unreleased(Beta/Aurora/etc) versions of the browsers in benchmarks and scoring but only stable releases in the crowning so it will be fair. many users opt for beta releases of browsers including myself, I use Firefox nightly(18.0a1)btw great work adamovera keep it up man

    The big problem with including the dev channel browsers is the amount of time it takes to produce the article (testing/charts/writing/editing/translating), combined with the tendency of the dev channel to constantly update. Before testing is even completed it's certain that something will update. TBH, the stable channels of Chrome and Firefox are a handful as it is. For example, for this article I had to test 8 browsers (4 on each OS), but I ended up testing 18+ due to OS X, Chrome, Firefox, Opera, Flash, and Java updates. Sorry, but I'm just not sure it's even doable in this format. Thanks for reading!
  • 1 Hide
    shahrooz , September 7, 2012 9:16 AM
    adamoveraThe big problem with including the dev channel browsers is the amount of time it takes to produce the article (testing/charts/writing/editing/translating), combined with the tendency of the dev channel to constantly update. Before testing is even completed it's certain that something will update. TBH, the stable channels of Chrome and Firefox are a handful as it is. For example, for this article I had to test 8 browsers (4 on each OS), but I ended up testing 18+ due to OS X, Chrome, Firefox, Opera, Flash, and Java updates. Sorry, but I'm just not sure it's even doable in this format. Thanks for reading!

    well I wanted to include it in my comment myself but I forgot I wanted to say if the timing allows :) 
  • 2 Hide
    Anonymous , September 7, 2012 9:31 AM
    Meaningless tests designed to generate clicks. All browsers are mature enough at this point in terms of performance. Even the lowly IE is OK. What matters is who is the browser working for, the user, or the mothership? And that's where FF is king. It's about treacherous, gimmicky commercial software made to push standards lock-in and data mining your information, versus free open source software that serves YOUR needs. And it's not only about the code being open either (Chrome). Get savvy on this stuff. Performance metrics are for FPS nerd teenagers.
  • 2 Hide
    JoeMomma , September 7, 2012 10:42 AM
    Call me Old Skool but for me functionality and ease of use far outweigh speed.
    My computer is fast enough that it does not really mater what browser I choose.
    In my case, ease of use means that I can see what is going on.
    I decry the trend towards dumbing down the UI on every program I use.
    (I also refuse to call software 'Apps', to me an app is a mini-program on a phone.)
    I always turn on all menus, buttons and labels in WaterFox.

    BTW: Good point.
    Why don't you include WaterFox in your testing?
    It is the 64 bit version of FireFox and I am sure that in your speed tests it may do a little better.
  • 0 Hide
    azathoth , September 7, 2012 10:47 AM
    Internet Explorer is starting to look pretty good in terms of speed.
Display more comments