Web Browser Grand Prix: Chrome 20, Opera 12, Firefox 13

Features
By published

How do the latest Windows 7-based Web browsers stack up? Chrome and Firefox each have three wins under this operating system; can one become our champion? Or will Opera clean house here, as it did under Windows XP? We set up a new test system to find out.

Test Suite And Methodology

Big Changes

We replaced GUIMark 2 Flash, GUIMark Java, and Encog Silverlight with five tests from RIABench Flash, Java, and Silverlight. We also added RIABench JavaScript to our JS testing, a long overdue replacement for JSBenchmark and V8. While RIABench consists of ten separate tests, two of the tests aren't available for all four technologies, one is currently spitting up a 404 error, Safari has a problem with one, and IE9 has a problem with another.

We had to temporarily replace Dromaeo DOM with Acid3 due to the former having issues with the WebKit-based browsers under Windows.

Another CSS test was added to balance out the Maze Solver CSS3 benchmark.

The Mozilla Hardware Acceleration Stress Test was replaced with WebVizBench, which has no highest score limit.

We've added a general responsiveness observation to the 40-Tab Memory Usage test.

And finally, we found a security test that is still relevant and not passed by any of the contenders.

Our tests are no longer placed into the core, observation, dated, and quarantine groups. With the massive refresh to the benchmark suite and the introduction of composite scores to cover every category of testing, this is simply no longer necessary.

Web Browser Grand Prix Test Suite v11

The table below lists all 34 benchmarks (consisting of 66 individual tests) currently in our suite, along with the version number and link (where applicable), and the number of iterations performed.

Swipe to scroll horizontally
Benchmark NameIterations Performed
Performance Benchmarks (24 Benchmarks, 56 Tests)
Cold Startup Time: Single Tab3
Hot Startup Time: Single Tab3
Cold Startup Time: Eight Tabs3
Hot Startup Time: Eight Tabs3
Uncached Page Load Times (8 Test Pages)5
Cached Page Load Times (8 Test Pages)5
RIABench JavaScript (5 Tests)3
Mozilla Kraken v1.12
Google SunSpider v0.9.1 Mod2
FutureMark Peacekeeper 2.02
Acid35
Maze Solver5
CSS Stress Test and Performance Profiling - Tom's Hardware2
GUIMark 2 HTML5 (3 Tests)3
Asteroids HTML5 Canvas 2D And JavaScript2
HTML5 Canvas Performance Test2
Facebook JSGameBench v4.12
Psychedelic Browsing2
WebVizBench2
Mozilla WebGL FishIE2
WebGL Solar System2
RIABench Flash (5 Tests)3
RIABench Java (5 Tests)3
RIABench Silverlight (5 Tests)3
Efficiency Benchmarks (4 Benchmarks/Tests)
Memory Usage: Single Tab3
Memory Usage: 40 Tabs3
Memory Management: -39 Tabs3
Memory Management: -39 Tabs (extra 2 minutes)3
Reliability Benchmarks (1 Test)
Proper Page Loads3
Responsiveness Benchmarks (1 Test)
General Responsiveness Under Load3
Security Benchmarks (1 Test)
BrowserScope Security1
Conformance Benchmarks (3 Benchmarks/Tests)
Ecma test2621
Peacekeeper 2.0 HTML5 Capabilities1
HTML5Test.com1

Methodology

We restart the computer and allow it to idle before benchmarking. Most of our final scores are an average of several iterations. More iterations are run for tests that have short durations, lower scales, and/or higher variance. Any obvious outliers (usually network hiccups) are removed and retested.

Individual detailed methodologies and information regarding composite scoring is described on the corresponding benchmark pages.