Sign in with
Sign up | Sign in

Benchmark Analysis

Web Browser Grand Prix: Chrome 18, Firefox 11, Windows XP
By

For the past year or so, one reader's request has gnawed at us: come up with a weighted numerical scoring system for determining the winner of the Web Browser Grand Prix. Today we're transitioning to such a system.

First, we need to modify the existing analysis table. We dropped the distinction between Performance, Efficiency, Reliability, and Standards Conformance, though the benchmarks are still arranged that way in the article. We also condensed the categories of testing into major categories. For example, Startup Time is no longer split between light and heavy, Page Load Time is no longer split between cached and uncached, all the memory tests now fall under Memory Efficiency, and the same applies to Standards Conformance.

Finishes

As before, each category of testing has four columns: winner, strong, average (previously, acceptable), and weak. Winner is obviously the browser that achieves the highest scores in that category. The Strong column is for browsers exhibiting superior performance, but not achieving a first-place victory. Average is for browsers that perform adequately or in-line with a majority of their competitors. The Weak column is for browsers that perform poorly, or substantially lower than their competitors.

Brackets

In order to reflect how each category of testing affects the average end-user Web browsing experience, we need to create brackets (or levels of importance) to place the different categories of testing into.

Essential
CSS, DOM, JavaScript, Page Load Reliability, Standards Conformance
Important
Flash, HTML5, Memory Efficiency, Page Load Time, Startup Time
Nonessential
Java, Silverlight
Unimportant
HTML5 Hardware Acceleration, WebGL


We received some great feedback from readers on the brackets, and consequently made a few changes to our original grouping. Page Load Time was demoted from Essential to Important due to the minuscule scale of those tests. Memory Efficiency and Startup Time were promoted from Nonessential to Important due to the large number of folks who still browse the Web on much older hardware, such as our XP-based test system.

The Essential bracket contains those categories of testing that are indispensable to rendering the vast majority of Web pages online today. The Important bracket is for categories not quite essential to browsing the Web, yet still affect the user experience to a great degree. The Nonessential bracket contains the popular plug-ins Java and Silverlight. While these plug-in technologies are nowhere near as ubiquitous as Flash, certain applications like corporate intranet apps and Netflix simply will not work without them. Finally, the Unimportant bracket is for emerging technologies, such as HTML5 Hardware Acceleration and WebGL, that simply do not exist outside of testing/demo sites.

Points

Now that the brackets are all sorted out, we can apply a numerical point system to the finishes of each bracket.


Winner
Strong
Average
Weak
Essential
2.5
2
1.5
-2
Important
2
1.5
1
-1.5
Nonessential
1.5
1
0.5
-1
Unimportant
1
0.5
0
-0.5


As you can see, we decided to apply negative point values to the Weak finishes and start the Average performances at zero for the Unimportant bracket. The Winner has also been de-emphasized over strong finishes, with just a small tie-breaking bonus going to the winner.

Web Browser Grand Prix XP Analysis Table


Winner
Strong
Average
Weak
Essential
CSS
Chrome
Safari
Opera
Firefox, Internet Explorer 8
DOM
Opera

Chrome, Firefox, Safari
Internet Explorer 8
JavaScript
Chrome
Firefox, Opera
Safari
Internet Explorer 8
Page Load Reliability
Opera
Internet Explorer 8
Firefox
Chrome, Safari
Standards Conformance
Chrome
Opera, Firefox
Safari
Internet Explorer 8
Important
Flash
Internet Explorer 8, Opera, Safari
Chrome, Firefox


HTML5
Chrome
Opera, Safari
Firefox
Internet Explorer 8
Memory Efficiency
Chrome
Firefox
Opera, Internet Explorer 8, Safari

Page Load Time
Opera
Chrome, Safari
Firefox, Internet Explorer 8

Startup Time
Opera
Chrome, Firefox

Internet Explorer 8, Safari
Nonessential
Java
Chrome, Internet Explorer 8, Opera
Firefox
Safari

Silverlight
Chrome
Firefox, Internet Explorer 8
Opera, Safari

Unimportant
HTML5 Hardware Acceleration
Chrome
Opera
Firefox, Safari
Internet Explorer 8
WebGL
Chrome


Firefox, Internet Explorer 8, Opera, Safari


Now, let's total up the points so we can see the final standings and crown a champion for Windows XP.

Display all 61 comments.
This thread is closed for comments
Top Comments
  • 21 Hide
    dameon51 , April 18, 2012 5:26 AM
    gwiz1987Why is IE8 being benched and not IE9?

    XP doesn't support 9, only 8.
  • 11 Hide
    wheredahoodat , April 18, 2012 4:47 AM
    "Both Opera and Chrome feltmuch smoother on our old PC than Firefox"

    I do kinda feel the difference with Firefox's responsive going from my main modern desktop to my older labtop that has regulated to a makeshift HTC. I believe Firefox XUL interface is the culprit; it was a big enough problem for Firefox mobile to abandon it in favor of native Android GUI, but who knows at this point. I guess might actually give Opera a chance.
Other Comments
  • 11 Hide
    wheredahoodat , April 18, 2012 4:47 AM
    "Both Opera and Chrome feltmuch smoother on our old PC than Firefox"

    I do kinda feel the difference with Firefox's responsive going from my main modern desktop to my older labtop that has regulated to a makeshift HTC. I believe Firefox XUL interface is the culprit; it was a big enough problem for Firefox mobile to abandon it in favor of native Android GUI, but who knows at this point. I guess might actually give Opera a chance.
  • 7 Hide
    agnickolov , April 18, 2012 5:14 AM
    How come only a single reader requested numerical composite scoring, that's the most logical way of scoring after all! With that said, I'd have liked if you didn't use the rankings but the raw scores after a more intelligent transformation as the input for weighted averaging...
    For example, for each category you could subtract the lowest-placed score from all scores and then normalize in the range [0-1] by dividing all adjusted scores by the topmost adjusted score. This way the top perfomer always has 1 and the worst performer always has 0 modified score (you'd need to invert them for tests where lower is better of course, e.g. subtract these from 1). Then apply your ranks to these scores and you get the composite score. It's not a perfect transformation, but it certainly has more fairly distributed weight (pun intended) than what you have used here.
  • 8 Hide
    aznjoka , April 18, 2012 5:19 AM
    Thats my Opera, for those who have never tried Opera. It's an amazing piece of software, it does the job, and it does it better then most.
  • 2 Hide
    confish21 , April 18, 2012 5:23 AM
    Interesting move to make this article. Well done! Don't waste your time on a vista run though... Im so close my release date. xD
  • 8 Hide
    csbeer , April 18, 2012 5:25 AM
    aznjokaThats my Opera, for those who have never tried Opera. It's an amazing piece of software, it does the job, and it does it better then most.


    XP can't run 9. Need to upgrade OS in order to get higher IE.
  • 21 Hide
    dameon51 , April 18, 2012 5:26 AM
    gwiz1987Why is IE8 being benched and not IE9?

    XP doesn't support 9, only 8.
  • 4 Hide
    mayankleoboy1 , April 18, 2012 5:57 AM
    excellent review!
    some points:

    1.A lot of corporates still use IE7. maybe you should include that too in your benchmarks

    2.if you remove HTML5 (with and without H/W acceleration), i think Opera's victory margin will be quite huge.

    3.Regarding smoothness, i beleive FF is quite poor in this. But the developers know about it and are very activle working on it. I thik FF13 will be the release when smoothness will improve. look at "Firefox Snappy".

    4. i would like to have a subjective recommendation at the end of the article, something you subjectively felt was the best amongst all the browsers, even though it may be trailing in numbers.
  • 7 Hide
    mayankleoboy1 , April 18, 2012 6:01 AM
    Why did you use the AGP? I bet 99.99% of those Pentium4 era computers use the onboard Intel IGP.
    Also that would definitely disable the H/W acceleration of browsers.

  • 6 Hide
    Anonymous , April 18, 2012 6:16 AM
    Anyone who is still stuck using Windows 2000, Opera supports you.

  • 1 Hide
    Anonymous , April 18, 2012 6:20 AM
    bunnywannyAnyone who is still stuck using Windows 2000, Opera supports you.

    Toms, the "add an url" in the comment toolbar doesn't work. Here is the link:
    http://www.opera.com/browser/download/requirements/
  • 3 Hide
    ronch79 , April 18, 2012 6:48 AM
    I don't know about you guys, but I've been a fan of Opera for a few years now, until recently. I've noticed that the then-latest version, 11.61, took so long to load pages. At first, I thought there was something wrong with my internet connection. For some reason I installed Chrome. I noticed right away that browsing with Chrome was faster (pages loaded quickly). My first suspicion was that it was just plain luck; that the bandwidth simply was faster coincidentally when I was using Chrome. I did a side-by-side comparison with both browsers open, and yes, Opera did load pages sluggishly. I was stumped. For the record, I'm NOT bashing Opera here, folks. I just don't know why this is happening. Heck, I'd switch back to Opera in a flash. This all happened with Opera 11.61. I think I'll give Opera 12 a shot.
  • 2 Hide
    straatkat , April 18, 2012 6:52 AM
    The html5 ranking is surprising. A score north of 300 for HTML5 support is, I would say, about par. In the end, HTML5 adoption in the wild is a good as the lowest score, because you want people to have access to your site, you are not going to build a site that locks out a substantial portion of the internet. So you are not going to target a HTML5 feature that is specific to a browser.
  • -6 Hide
    hellfire24 , April 18, 2012 8:01 AM
    man i still love XP!!!!
  • 4 Hide
    straatkat , April 18, 2012 9:23 AM
    Can you state which version of Windows XP you were using? Circa 2003 you had Windows XP service pack 1 only.
  • -4 Hide
    assassin123 , April 18, 2012 10:05 AM
    I love firefox
  • -2 Hide
    Cryio , April 18, 2012 10:40 AM
    muhsi44even though the OSes are still in beta.


    What OSes?
  • -6 Hide
    Hypertraxx , April 18, 2012 10:56 AM
    CHROMEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE
  • 5 Hide
    sheol , April 18, 2012 12:17 PM
    Out of the box, no browser that I know of comes close to Opera when it comes to usability(mouse gestures, tab manangement, "Closed Tabs " - a recycle bin of tabs if you will, an awesome feature that keeps your closed tabs for easy retrieval for when you might need them.
    This is why I believe Opera uses so much memory after 39 tabs have been closed.

    And reliability, who could complain? Routinely i have 50+ tabs open for days, and I have not had a single crash because of it.
Display more comments