Sign in with
Sign up | Sign in

Test Suite And Methodology

Web Browser Grand Prix: Chrome 20, Opera 12, Firefox 13
By

Big Changes

We replaced GUIMark 2 Flash, GUIMark Java, and Encog Silverlight with five tests from RIABench Flash, Java, and Silverlight. We also added RIABench JavaScript to our JS testing, a long overdue replacement for JSBenchmark and V8. While RIABench consists of ten separate tests, two of the tests aren't available for all four technologies, one is currently spitting up a 404 error, Safari has a problem with one, and IE9 has a problem with another.

We had to temporarily replace Dromaeo DOM with Acid3 due to the former having issues with the WebKit-based browsers under Windows.

Another CSS test was added to balance out the Maze Solver CSS3 benchmark.

The Mozilla Hardware Acceleration Stress Test was replaced with WebVizBench, which has no highest score limit.

We've added a general responsiveness observation to the 40-Tab Memory Usage test.

And finally, we found a security test that is still relevant and not passed by any of the contenders.

Our tests are no longer placed into the core, observation, dated, and quarantine groups. With the massive refresh to the benchmark suite and the introduction of composite scores to cover every category of testing, this is simply no longer necessary.

Web Browser Grand Prix Test Suite v11

The table below lists all 34 benchmarks (consisting of 66 individual tests) currently in our suite, along with the version number and link (where applicable), and the number of iterations performed.

Benchmark Name
Iterations Performed
Performance Benchmarks (24 Benchmarks, 56 Tests)
Cold Startup Time: Single Tab
3
Hot Startup Time: Single Tab3
Cold Startup Time: Eight Tabs3
Hot Startup Time: Eight Tabs
3
Uncached Page Load Times (8 Test Pages)
5
Cached Page Load Times (8 Test Pages)
5
RIABench JavaScript (5 Tests)
3
Mozilla Kraken v1.1
2
Google SunSpider v0.9.1 Mod2
FutureMark Peacekeeper 2.02
Acid3
5
Maze Solver5
CSS Stress Test and Performance Profiling - Tom's Hardware
2
GUIMark 2 HTML5 (3 Tests)
3
Asteroids HTML5 Canvas 2D And JavaScript2
HTML5 Canvas Performance Test
2
Facebook JSGameBench v4.12
Psychedelic Browsing2
WebVizBench
2
Mozilla WebGL FishIE2
WebGL Solar System2
RIABench Flash (5 Tests)
3
RIABench Java (5 Tests)
3
RIABench Silverlight (5 Tests)
3
Efficiency Benchmarks (4 Benchmarks/Tests)
Memory Usage: Single Tab
3
Memory Usage: 40 Tabs
3
Memory Management: -39 Tabs
3
Memory Management: -39 Tabs (extra 2 minutes)
3
Reliability Benchmarks (1 Test)
Proper Page Loads
3
Responsiveness Benchmarks (1 Test)
General Responsiveness Under Load
3
Security Benchmarks (1 Test)
BrowserScope Security
1
Conformance Benchmarks (3 Benchmarks/Tests)
Ecma test2621
Peacekeeper 2.0 HTML5 Capabilities1
HTML5Test.com1


Methodology

We restart the computer and allow it to idle before benchmarking. Most of our final scores are an average of several iterations. More iterations are run for tests that have short durations, lower scales, and/or higher variance. Any obvious outliers (usually network hiccups) are removed and retested.

Individual detailed methodologies and information regarding composite scoring is described on the corresponding benchmark pages.

Display all 110 comments.
This thread is closed for comments
Top Comments
  • 24 Hide
    mayankleoboy1 , July 5, 2012 5:10 AM
    1.what the benchmarks dont show is that in Firefox , if a tab has a heavy page with a lot of CPU intensive workload, the complete browser UI starts stuttering. That means the browser UI is on the same thread as the page loading.

    2. in the 40 tab test, try working in a tab during the loading of the 40 tabs. you will find lots of difference between browsers. FF hangs, Opera and Chrome remain fluid.

    3. how about a test where a browser is using 1GB+ RAM and you are trying to open/close tabs. Then see the UI responsiveness. most browsers can easily handle 800MB RAM. but which browser easily handles 1.2GB+ RAM ?
  • 22 Hide
    mayankleoboy1 , July 5, 2012 5:17 AM
    How many firefox users dont use ADblockPlus ? very very less.Also ABP developer is a regular contributor to the Firefox source code.
    maybe you should do a few memory benchmarks with ABP installed just to realistically judge what 99.99% of FF users go through.
  • 21 Hide
    ben850 , July 5, 2012 5:22 AM
    WTF Chrome is already on 20?? It ninja updated to 19 just a few days ago..
Other Comments
  • 24 Hide
    mayankleoboy1 , July 5, 2012 5:10 AM
    1.what the benchmarks dont show is that in Firefox , if a tab has a heavy page with a lot of CPU intensive workload, the complete browser UI starts stuttering. That means the browser UI is on the same thread as the page loading.

    2. in the 40 tab test, try working in a tab during the loading of the 40 tabs. you will find lots of difference between browsers. FF hangs, Opera and Chrome remain fluid.

    3. how about a test where a browser is using 1GB+ RAM and you are trying to open/close tabs. Then see the UI responsiveness. most browsers can easily handle 800MB RAM. but which browser easily handles 1.2GB+ RAM ?
  • 13 Hide
    mayankleoboy1 , July 5, 2012 5:13 AM
    IE9 does so good on HTML5 HWA accelerated benchmarks because its able to offload more of the processing to the GPU.

    i tested this and found that during a HTML5 benchmark, IE9 had the least CPU usage, and most GPU usage amongst all the browsers.
  • 22 Hide
    mayankleoboy1 , July 5, 2012 5:17 AM
    How many firefox users dont use ADblockPlus ? very very less.Also ABP developer is a regular contributor to the Firefox source code.
    maybe you should do a few memory benchmarks with ABP installed just to realistically judge what 99.99% of FF users go through.
  • 21 Hide
    ben850 , July 5, 2012 5:22 AM
    WTF Chrome is already on 20?? It ninja updated to 19 just a few days ago..
  • 20 Hide
    lethalsam , July 5, 2012 5:35 AM
    i won't ever use a browser a browser WITHOUT AD BLOCK Plus. (ABP)

    ABP works wonderful on Firefox, i RARELY see any ad. While I have used ABP on Chrome BUT its doesn't block half the ads.
    I know its Not Google's fault, its just that ABP developers are putting more effort with Firefox.

    So for me, Firefox > Chrome.
  • 9 Hide
    adamovera , July 5, 2012 5:42 AM
    @mayankleoboy1: 1+2) Interesting, I'll be looking for that next time 3) That would require a different workload for each browser.
    IE9 does so good on HTML5 HWA accelerated benchmarks because its able to offload more of the processing to the GPU. i tested this and found that during a HTML5 benchmark, IE9 had the least CPU usage, and most GPU usage amongst all the browsers.

    Really interesting, what utility do you use for measuring GPU usage?
    How many firefox users dont use ADblockPlus ? very very less.Also ABP developer is a regular contributor to the Firefox source code.maybe you should do a few memory benchmarks with ABP installed just to realistically judge what 99.99% of FF users go through.

    I'd estimate ABP usage on FF at around 5% or less based on ABP and FF usage statistics. Besides, that would give FF an unfair advantage.
  • 6 Hide
    mayankleoboy1 , July 5, 2012 5:51 AM
    Quote:
    Really interesting, what utility do you use for measuring GPU usage?


    MSI afterburner for GPU. windows task manager for CPU.

    Quote:
    @mayankleoboy1: 1+2) Interesting, I'll be looking for that next time


    i sent a mail regarding this to Chris. but maybe i sent it too late for this article...
  • 6 Hide
    mayankleoboy1 , July 5, 2012 5:56 AM
    if you open multiple tabs together in chrome, it can use each CPU core for each tab. so if you have a quad core, and you open 4+ tabs together, the CPU usage will be 100% (using all 4 cores) during the tab loading time.
    but if you run 4 instance of dromaeo in 4 tabs, the CPU usage is still 25% (using only 1 core).
    so chrome is not completely multiprocessing.

    in IE10 beta, if you run 4 instances of dromaeo benchmark in 4 tabs, it uses all the for cores. so we can expect better multiprocessing from IE10 and win8 :) 
  • 3 Hide
    adamovera , July 5, 2012 6:08 AM
    @mayankleoboy1: I got that email yesterday or the day before, this article was completed a few days before that. Sorry, my inbox usually gets a few pages deep after a doing long benchmark-heavy article.
    Is Dromaeo (the DOM portion) working in Chrome for you? I could not get it to finish in Chrome or Safari on any of my Windows machines.
  • 1 Hide
    mayankleoboy1 , July 5, 2012 6:22 AM
    i ran the javascript benchmark that ran fine. Didnt run the DOM benches.

    BTW, i run chrome dev version. so that could make a difference.
  • 9 Hide
    Regor245 , July 5, 2012 6:40 AM
    I have Both Chrome and Firefox.

    I still prefer Firefox since it has more features and i like it's features.

    I disabled Smooth Scrolling to make it more responsive.

    I just hate random freezes/stutter sometimes and some problem on Youtube while watching,
    when you scroll up/down, the youtube screen is messed (Glitch).

    By the way, ABP user here too.
  • 4 Hide
    mayankleoboy1 , July 5, 2012 6:44 AM
    Quote:
    when you scroll up/down, the youtube screen is messed (Glitch).

    thats a flash issue. disable the protected mode of falsh.
  • 6 Hide
    Memnarchon , July 5, 2012 7:41 AM
    mayankleoboy1IE9 does so good on HTML5 HWA accelerated benchmarks because its able to offload more of the processing to the GPU.

    i tested this and found that during a HTML5 benchmark, IE9 had the least CPU usage, and most GPU usage amongst all the browsers.


    +1.
    IE9 uses also a lot more GPU Memory than Chrome. I am listening to internet radio when I am playing games and I found that a single tab from IE9 (opened for more than an hour) uses 150MB VRAM oO. Now this amount is not significant for some users but for most users that have 1024MB VRAM and doing the same thing like me (or have open tabs and alt-tabing during games) while playing at 1920x1080/1200 might cause fps drops.
  • -1 Hide
    s3anister , July 5, 2012 7:48 AM
    lethalsami won't ever use a browser a browser WITHOUT AD BLOCK Plus. (ABP)ABP works wonderful on Firefox, i RARELY see any ad. While I have used ABP on Chrome BUT its doesn't block half the ads.I know its Not Google's fault, its just that ABP developers are putting more effort with Firefox.So for me, Firefox > Chrome.

    Chrome has Adblock, just to let you know. But personally I prefer Firefox Aurora with Adblock Plus, anyway. I couldn't imagine using a browser that didn't have any version of Adblock on it, though; it'd be torture.
  • 2 Hide
    PreferLinux , July 5, 2012 7:50 AM
    Has anyone ever investigated why the MS Maze Solver is so slow on Firefox? It almost has to be a FF bug that hasn't been fixed rather than just that FF is slow at CSS3 when the difference between FF and everything else is so huge, and also that FF does very well in CSS2.
  • 9 Hide
    s3anister , July 5, 2012 7:52 AM
    princesiddiquiFire Fox 14.8 has been announced, version 13 is old version.

    jerm1027Why is Firefox 13 on here? Firefox 14 has been out for a little while (even on Android), and it's support for SPDY may very affect the results. Just saying.

    13.0.1 is the latest stable release. I'm using version 15.0a2 of Firefox, but it's a moot point in any browser competition because it's not a release version.
  • 7 Hide
    PreferLinux , July 5, 2012 7:53 AM
    jerm1027Why is Firefox 13 on here? Firefox 14 has been out for a little while (even on Android), and it's support for SPDY may very affect the results. Just saying.

    Where? Not here: http://www.mozilla.org/en-US/firefox/all.html
  • 9 Hide
    jerm1027 , July 5, 2012 7:59 AM
    s3anister13.0.1 is the latest stable release. I'm using version 15.0a2 of Firefox, but it's a moot point in any browser competition because it's not a release version.

    i stand corrected. I'm not sure how i got mixed up.

    However, I'm not sure how Silverlight takes importance over WebGL or HTML5 in this test. >~
Display more comments