Sign in with
Sign up | Sign in

Test Setup And Methodology

Web Browser Grand Prix 7: Firefox 7, Chrome 14, Opera 11.51
By

Hardware Setup

Test System Specs
Operating System
Microsoft Windows 7 Ultimate (64-bit)
Processor
Intel Core i5-750 (Lynnfield) @ 2.8 GHz, Quad-Core
Motherboard
Gigabyte GA-P55A-UD7, LGA 1156, P55 Express, F7 BIOS
Memory
8 GB Crucial DDR3 @ 1333 MT/s (2 x 4 GB)
Graphics
AMD Radeon HD 4870 Reference Boards 512 GB GDDR5 (PCI-e 2.0)
Storage
Seagate Barracuda 7200.12 500 GB SATA 3Gb/s, 7200 RPM, 16 MB Cache
Optical
Asus DRW-24B1ST/BLK/B/AS
Power Supply
Corsair TX750W (750 Watt Max)
Chassis
Zalman MS1000-HS2
CPU Cooler
Scythe Mugen 2 Revision B


The WBGP7 Test InstallationThe WBGP7 Test Installation

The following table holds the system specs of the local Web server used for our new startup time tests, the page load time tests, and JSGameBench.

Local Web Server Specs
Operating System
Ubuntu 10.04 LTS Server Edition "Lucid Lynx" (32-bit)
Processor
AMD Athlon @ 1150 MHz
Motherboard
Soyo Dragon Platinum
Memory
512 MB DDR
Graphics
AMD Radeon 9550, 256 MB GDDR
Storage
40 GB Western Digital HDD WD400BB
Optical
Samsung DVD-ROM SD-616T
Extra Packages
Apache2, MySQL Client, MySQL Server, PHP5, PHP-GD, PHP5-MySQL, PHPMyAdmin, SSH


The table below holds additional information on the test network.

Network Specs
ISP Service
Cox Premium (28 Mb/s down, 5 Mb/s up)
Modem
Motorola SURFboard SBS101U
Router
Linksys WRT54G2 V1


Software Setup

Our Windows 7 Ultimate test installation was freshly installed and fully updated as of midnight on September 25th. Power management and automatic updating is disabled before testing. The Web browsers and additional software, along with the exact version numbers tested, are listed in the table below.

Software
Version
Chrome14.0.835.186 m
Firefox
7.0
Internet Explorer
9.0.8112.16421
Opera
11.51 (build 1087)
Safari
5.1 (7534.50)
AMD Driver
8.850.0.0
Adobe Flash
10.3.183.10
Microsoft Silverlight
4.0.60531.0
Oracle Java
6 Update 27 (build 1.6.0_27-b07)


Test Setup


We restart the computer and allow it to idle for a few minutes before benchmarking the next browser. Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variations.

All tests are placed into one of four groups: core, observation, dated, and quarantine. Core tests are considered current. These tests are usually trusted industry standards or our own creations, and they make up the core of the WBGP suite. Tests that are either generally unknown, mostly untested, or just too bleeding-edge are placed under observation. Tests classified as dated are either outdated, losing relevance, or otherwise need replacing. We are actively seeking community feedback and contributions regarding alternatives to these benchmarks. The final group is for quarantined benchmarks. Benchmarks find their way into quarantine by delivering dubious results or by being gamed. Whenever benchmarks that test the same thing yield conflicting results, more weight is given to tests with a better rating when creating the analysis tables.

The table below lists all 41 of the tests currently in our test suite (along with a version number and link, where applicable), number of iterations performed, and current rating in the Web Browser Grand Prix:

Web Browser Grand Prix 7 Test Suite
Test Name
Iterations
Rating
Performance Tests (33)
Startup Time: Single Tab
5
Core
Startup Time: Eight Tabs
5
Core
Page Load Time: Google
5
Core
Page Load Time: YouTube
5
Core
Page Load Time: Yahoo!
5
Core
Page Load Time: Amazon
5
Core
Page Load Time: Wikipedia
5
Core
Page Load Time: eBay
5
Core
Page Load Time: craigslist
5
Core
Page Load Time: The Huffington Post
5
Core
FutureMark Peacekeeper
3
Core
Kraken v1.1
3
Core
Google Kraken v1.1 Mod
3
Observation
SunSpider v0.9.1
3
Quarantine
Google SunSpider v0.91 Mod
3
Observation
Dromaeo DOM
2
Core
Maze Solver
3
Core
JSGameBench v4.1
2
Core
Asteroids HTML5 Canvas 2D And JavaScript
3
Observation
GUIMark 2 HTML5 Vector Charting (1 pixel variant)
3
Core
GUIMark 2 HTML5 Bitmap Gaming
3
Core
GUIMark 2 HTML5 Text Columns
3
Core
Psychedelic Browsing
2
Core
Hardware Acceleration Stress Test
2
Dated
WebGL FishIE
5
Observation
WebGL Solar System
5
Observation
ThoughtsInComputation Particles
5
Observation
GUIMark2 Flash Vector Charting
3
Core
GUIMark2 Flash Bitmap Gaming
3
Core
GUIMark2 Flash Text Columns
3
Core
Flash Benchmark 2008
3
Core
GUIMark Java
3
Dated
Encog Silverlight
5
Dated
Efficiency Benchmarks (4)
Memory Usage: Single Tab
3
Core
Memory Usage: 40 Tabs
3
Core
Memory Management: -39 Tabs
3
Core
Memory Management: -39 Tabs (extra 5 minutes)
3
Core
Reliability Benchmarks (1)
Proper Page Loads
3
Core
Conformance Benchmarks (3)
HTML5Test.com
1
Core
Ecma test262
1
Observation
Acid3
1
Dated


You've seen the line-up and toured the track. Now it's off to the races.

React To This Article