Finishes
Each category of testing has four columns: Winner, Strong, Average, and Weak. The Winner is obviously the browser that achieves the highest scores in that category. The Strong column is for browsers exhibiting superior performance, but not achieving a first-place victory. Average is for browsers that perform adequately or in-line with a majority of their competitors. A Weak finish is assigned to browsers that perform poorly, or substantially lower than their competitors.
Brackets
In order to reflect how each category of testing affects the average end-user Web browsing experience, we need to create brackets (or levels of importance) to place the different categories of testing into.
| Essential | CSS, DOM, JavaScript, Reliability, Standards Conformance |
|---|---|
| Important | Flash, HTML5, Memory Efficiency, Page Load Time, Responsiveness, Security, Startup Time |
| Nonessential | Java, Silverlight |
| Unimportant | HTML5 Hardware Acceleration, WebGL |
The Essential bracket contains those categories of testing that are indispensable to rendering the vast majority of Web pages online today. The Important bracket is for categories not quite essential to browsing the Web, yet still affect the user experience to a great degree. The Nonessential bracket contains the popular plug-ins Java and Silverlight. While these plug-in technologies are nowhere near as ubiquitous as Flash, certain applications like corporate intranet apps and Netflix simply will not work without them. Finally, the Unimportant bracket is for emerging technologies, such as HTML5 Hardware Acceleration and WebGL, which still don't really exist outside of testing/demo sites.
Points
Now that the brackets are all sorted out, we can apply a numerical point system to the finishes of each bracket.
| Winner | Strong | Average | Weak | |
|---|---|---|---|---|
| Essential | 2.5 | 2 | 1.5 | -2 |
| Important | 2 | 1.5 | 1 | -1.5 |
| Nonessential | 1.5 | 1 | 0.5 | -1 |
| Unimportant | 1 | 0.5 | 0 | -0.5 |
As you can see, we decided to apply negative point values to the Weak finishes and start the Average performances at zero for the Unimportant bracket. The Winner has also been de-emphasized over Strong finishes, with just a small tie-breaking bonus going to Winner.
OS X 10.8 Analysis Table
| Winner | Strong | Average | Weak | |
|---|---|---|---|---|
| Essential | ||||
| CSS | Firefox | Chrome, Safari | Opera | |
| DOM | Firefox | Chrome, Safari | Opera | |
| JavaScript | Chrome | Safari | Firefox, Opera | |
| Reliability | Opera | Firefox, Safari | Chrome | |
| Standards Conformance | Chrome | Firefox, Opera, Safari | ||
| Important | ||||
| Flash | Firefox, Opera, Safari | Chrome | ||
| HTML5 | Safari | Chrome | Firefox, Opera | |
| Memory Efficiency | Chrome | Firefox, Opera, Safari | ||
| Page Load Time | Chrome | Firefox, Opera, Safari | ||
| Responsiveness | Opera | Firefox, Safari | Chrome | |
| Security | Chrome | Safari | Firefox | Opera |
| Startup Time | Chrome | Safari | Firefox | Opera |
| Nonessential | ||||
| Java | Chrome, Firefox, Opera, Safari | |||
| Silverlight | Chrome, Firefox, Opera, Safari | |||
| Unimportant | ||||
| HTML5 Hardware Acceleration | Safari | Chrome | Firefox, Opera | |
| WebGL | Firefox | Chrome | Opera, Safari | |
Now, let's see how the Windows 7 Web browsers compare to each other.
Windows 7 Analysis Table
| Winner | Strong | Average | Weak | |
|---|---|---|---|---|
| Essential | ||||
| CSS | Firefox | Chrome, IE | Opera | |
| DOM | Chrome | Firefox, IE | Opera | |
| JavaScript | Chrome | Firefox, Opera | IE | |
| Reliability | Opera | Chrome, Firefox, IE | ||
| Standards Conformance | Chrome | Firefox, Opera | IE | |
| Important | ||||
| Flash | Firefox, IE, Opera | Chrome | ||
| HTML5 | IE | Chrome, Firefox | Opera | |
| Memory Efficiency | Chrome | IE, Firefox | Opera | |
| Page Load Time | IE | Chrome, Firefox, Opera | ||
| Responsiveness | Opera | Firefox | Chrome | IE |
| Security | Chrome | Firefox, IE | Opera | |
| Startup Time | Chrome | Opera | Firefox, IE | |
| Nonessential | ||||
| Java | Chrome, Firefox, IE, Opera | |||
| Silverlight | Firefox | Chrome, IE, Opera | ||
| Unimportant | ||||
| HTML5 Hardware Acceleration | Firefox | IE | Chrome | Opera |
| WebGL | Firefox | Chrome | IE, Opera | |
And the winners are...
- The Top Four Browsers, Tested And Ranked
- Chrome, Firefox, IE9, Opera, Safari
- Test System Specs And Software Setup
- Test Suite And Methodology
- Start Time
- Page Load Time
- JavaScript Performance
- DOM And CSS Performance
- HTML5 Performance
- Hardware Acceleration Performance
- Plug-In Performance: Flash, Java, Silverlight
- Memory Efficiency
- Reliability, Responsiveness, And Security
- Standards Conformance
- Test Analysis
- OS X And Windows 7 Winners' Circle
When we have more [official] stable 64-bit browsers, I'll definitely do a 64-bit WBGP - including versus their 32-bit counterparts.
Nearly every performance benchmark there is points in that direction. This probably has a lot to do with how much time developers spend optimizing for Windows - after all, Windows holds 90+% of the desktop user base. However, it is interesting that the rift between Windows and OS X is far greater than between Windows and Linux for the core stuff like JS, CSS, DOM, page loads, etc. Plug-ins are another story, they're always much better on Windows than the other two platforms.
(The nice popular ones like ABP, Lazarus, Greasemonkey all have equivalents; some lesser-used plugins like Rikaichan also have ports by now. Only a matter of time!)
as always, a great read.
All versions of Chrome hold up incredibly well cross-platform, if you look back at the two Linux WBGPs, it won there, too. Thanks for reading!
Absolutely, a Windows 8-based WBGP is already in the cards for October.
When we have more [official] stable 64-bit browsers, I'll definitely do a 64-bit WBGP - including versus their 32-bit counterparts.
Testing these browsers at stock doesn't reveal even an eighth of the picture.
btw great work adamovera keep it up man
Interesting idea, so basically a tweaked-out edition of the WBGP, where we use all the tools available to each browser for performance gains... That could work, but I gotta warn you that the next three WBGPs are already decided, so it would probably be real late in the year, or even next year before I could get to it.
Nearly every performance benchmark there is points in that direction. This probably has a lot to do with how much time developers spend optimizing for Windows - after all, Windows holds 90+% of the desktop user base. However, it is interesting that the rift between Windows and OS X is far greater than between Windows and Linux for the core stuff like JS, CSS, DOM, page loads, etc. Plug-ins are another story, they're always much better on Windows than the other two platforms.
The big problem with including the dev channel browsers is the amount of time it takes to produce the article (testing/charts/writing/editing/translating), combined with the tendency of the dev channel to constantly update. Before testing is even completed it's certain that something will update. TBH, the stable channels of Chrome and Firefox are a handful as it is. For example, for this article I had to test 8 browsers (4 on each OS), but I ended up testing 18+ due to OS X, Chrome, Firefox, Opera, Flash, and Java updates. Sorry, but I'm just not sure it's even doable in this format. Thanks for reading!
well I wanted to include it in my comment myself but I forgot I wanted to say if the timing allows
My computer is fast enough that it does not really mater what browser I choose.
In my case, ease of use means that I can see what is going on.
I decry the trend towards dumbing down the UI on every program I use.
(I also refuse to call software 'Apps', to me an app is a mini-program on a phone.)
I always turn on all menus, buttons and labels in WaterFox.
BTW: Good point.
Why don't you include WaterFox in your testing?
It is the 64 bit version of FireFox and I am sure that in your speed tests it may do a little better.