Test Setup And Methodology
Hardware Setup
| Test System Specs | |
|---|---|
| Operating System | Microsoft Windows 7 Ultimate (64-bit) |
| Processor | Intel Core i5-750 (Lynnfield) @ 2.8 GHz, Quad-Core |
| Motherboard | Gigabyte GA-P55A-UD7, LGA 1156, P55 Express, F7 BIOS |
| Memory | 8 GB Crucial DDR3 @ 1333 MT/s (2 x 4 GB) |
| Graphics | AMD Radeon HD 4870 Reference Boards 512 GB GDDR5 (PCI-e 2.0) |
| Storage | Seagate Barracuda 7200.12 500 GB SATA 3Gb/s, 7200 RPM, 16 MB Cache |
| Optical | Asus DRW-24B1ST/BLK/B/AS |
| Power Supply | Corsair TX750W (750 Watt Max) |
| Chassis | Zalman MS1000-HS2 |
| CPU Cooler | Scythe Mugen 2 Revision B |
The following table holds the system specs of the local Web server used for our new startup time tests, the page load time tests, and JSGameBench.
| Local Web Server Specs | |
|---|---|
| Operating System | Ubuntu 10.04 LTS Server Edition "Lucid Lynx" (32-bit) |
| Processor | AMD Athlon @ 1150 MHz |
| Motherboard | Soyo Dragon Platinum |
| Memory | 512 MB DDR |
| Graphics | AMD Radeon 9550, 256 MB GDDR |
| Storage | 40 GB Western Digital HDD WD400BB |
| Optical | Samsung DVD-ROM SD-616T |
| Extra Packages | Apache2, MySQL Client, MySQL Server, PHP5, PHP-GD, PHP5-MySQL, PHPMyAdmin, SSH |
The table below holds additional information on the test network.
| Network Specs | |
|---|---|
| ISP Service | Cox Premium (28 Mb/s down, 5 Mb/s up) |
| Modem | Motorola SURFboard SBS101U |
| Router | Linksys WRT54G2 V1 |
Software Setup
Our Windows 7 Ultimate test installation was freshly installed and fully updated as of midnight on September 25th. Power management and automatic updating is disabled before testing. The Web browsers and additional software, along with the exact version numbers tested, are listed in the table below.
| Software | Version |
|---|---|
| Chrome | 14.0.835.186 m |
| Firefox | 7.0 |
| Internet Explorer | 9.0.8112.16421 |
| Opera | 11.51 (build 1087) |
| Safari | 5.1 (7534.50) |
| AMD Driver | 8.850.0.0 |
| Adobe Flash | 10.3.183.10 |
| Microsoft Silverlight | 4.0.60531.0 |
| Oracle Java | 6 Update 27 (build 1.6.0_27-b07) |
Test Setup
We restart the computer and allow it to idle for a few minutes before benchmarking the next browser. Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variations.
All tests are placed into one of four groups: core, observation, dated, and quarantine. Core tests are considered current. These tests are usually trusted industry standards or our own creations, and they make up the core of the WBGP suite. Tests that are either generally unknown, mostly untested, or just too bleeding-edge are placed under observation. Tests classified as dated are either outdated, losing relevance, or otherwise need replacing. We are actively seeking community feedback and contributions regarding alternatives to these benchmarks. The final group is for quarantined benchmarks. Benchmarks find their way into quarantine by delivering dubious results or by being gamed. Whenever benchmarks that test the same thing yield conflicting results, more weight is given to tests with a better rating when creating the analysis tables.
The table below lists all 41 of the tests currently in our test suite (along with a version number and link, where applicable), number of iterations performed, and current rating in the Web Browser Grand Prix:
| Web Browser Grand Prix 7 Test Suite | ||
|---|---|---|
| Test Name | Iterations | Rating |
| Performance Tests (33) | ||
| Startup Time: Single Tab | 5 | Core |
| Startup Time: Eight Tabs | 5 | Core |
| Page Load Time: Google | 5 | Core |
| Page Load Time: YouTube | 5 | Core |
| Page Load Time: Yahoo! | 5 | Core |
| Page Load Time: Amazon | 5 | Core |
| Page Load Time: Wikipedia | 5 | Core |
| Page Load Time: eBay | 5 | Core |
| Page Load Time: craigslist | 5 | Core |
| Page Load Time: The Huffington Post | 5 | Core |
| FutureMark Peacekeeper | 3 | Core |
| Kraken v1.1 | 3 | Core |
| Google Kraken v1.1 Mod | 3 | Observation |
| SunSpider v0.9.1 | 3 | Quarantine |
| Google SunSpider v0.91 Mod | 3 | Observation |
| Dromaeo DOM | 2 | Core |
| Maze Solver | 3 | Core |
| JSGameBench v4.1 | 2 | Core |
| Asteroids HTML5 Canvas 2D And JavaScript | 3 | Observation |
| GUIMark 2 HTML5 Vector Charting (1 pixel variant) | 3 | Core |
| GUIMark 2 HTML5 Bitmap Gaming | 3 | Core |
| GUIMark 2 HTML5 Text Columns | 3 | Core |
| Psychedelic Browsing | 2 | Core |
| Hardware Acceleration Stress Test | 2 | Dated |
| WebGL FishIE | 5 | Observation |
| WebGL Solar System | 5 | Observation |
| ThoughtsInComputation Particles | 5 | Observation |
| GUIMark2 Flash Vector Charting | 3 | Core |
| GUIMark2 Flash Bitmap Gaming | 3 | Core |
| GUIMark2 Flash Text Columns | 3 | Core |
| Flash Benchmark 2008 | 3 | Core |
| GUIMark Java | 3 | Dated |
| Encog Silverlight | 5 | Dated |
| Efficiency Benchmarks (4) | ||
| Memory Usage: Single Tab | 3 | Core |
| Memory Usage: 40 Tabs | 3 | Core |
| Memory Management: -39 Tabs | 3 | Core |
| Memory Management: -39 Tabs (extra 5 minutes) | 3 | Core |
| Reliability Benchmarks (1) | ||
| Proper Page Loads | 3 | Core |
| Conformance Benchmarks (3) | ||
| HTML5Test.com | 1 | Core |
| Ecma test262 | 1 | Observation |
| Acid3 | 1 | Dated |
You've seen the line-up and toured the track. Now it's off to the races.
