Hardware Setup
| Test System Specs | |
|---|---|
| Operating System 1 | Microsoft Windows 7 Ultimate (64-bit) |
| Operating System 2 | Apple Mac OS X Lion (64-bit) |
| Processor | Intel Core i5-750 (Lynnfield) @ 2.8 GHz, Quad-Core |
| Motherboard | Gigabyte GA-P55A-UD7, LGA 1156, P55 Express, F7 BIOS |
| Memory | 8 GB Crucial DDR3 @ 1333 MT/s (2 x 4 GB) |
| Graphics | AMD Radeon HD 4870 Reference Boards 512 GB GDDR5 (PCI-e 2.0) |
| Storage | Seagate Barracuda 7200.12 500 GB SATA 3Gb/s, 7200 RPM, 16 MB Cache |
| Optical | Asus DRW-24B1ST/BLK/B/AS |
| Power Supply | Corsair TX750W (750 Watt Max) |
| Chassis | Zalman MS1000-HS2 |
| CPU Cooler | Scythe Mugen 2 Revision B |
The Windows 7 Test Installation
The OS X Lion Test Installation
| Local Web Server Specs | |
|---|---|
| Operating System | Ubuntu 10.04 LTS Server Edition "Lucid Lynx" (32-bit) |
| Processor | AMD Athlon @ 1150 MHz |
| Motherboard | Soyo Dragon Platinum |
| Memory | 512 MB DDR |
| Graphics | AMD Radeon 9550, 256 MB GDDR |
| Storage | 40 GB Western Digital HDD WD400BB |
| Optical | Samsung DVD-ROM SD-616T |
| Extra Packages | Apache2, MySQL Client, MySQL Server, PHP5, PHP-GD, PHP5-MySQL, PHPMyAdmin, SSH |
As requested by readers in our previous stories, the table below hosts additional information on the test network.
| Network Specs | |
|---|---|
| ISP Service | Cox Premium (28 Mb/s down, 5 Mb/s up) |
| Modem | Motorola SURFboard SBS101U |
| Router | Linksys WRT54G2 V1 |
Hackintosh Setup
The table below lists what we used to get the Lion-based Hackintosh up and running.
| Installer | xMove |
|---|---|
| Bootloader | iBoot |
| DSDT | GA-P55A-UD7 |
While these tests are conducted on Mac OS X, this test system is not an Apple-branded Mac. We had to use a different bootloader than what you'd find on a genuine Mac, and real Mac systems use EFI instead of BIOS. Therefore, performance may vary from the same tests conducted on an Apple-branded system.
Also keep in mind that this use of OS X is completely unauthorized, and anyone doing this at home should have no expectations of receiving any help or support from Apple.
Software Setup
Both our Windows 7 Ultimate and Mac OS X Lion installations were freshly installed and fully updated as of midnight on August 15th. Power management and automatic updates were disabled before testing. The Web browsers and additional software, along with the exact version numbers tested, are listed in the table below.
| Software | Windows Version | Mac Version |
|---|---|---|
| Chrome | 13.0.782.215 | 13.0.782.215 |
| Firefox | 6.0 | 6.0 |
| Internet Explorer | 9.0.8112.16421 | N/A |
| Opera | 11.50 (1074) | 11.50 (1074) |
| Safari | 5.1 (7534.50) | 5.1 (7534.48.3) |
| AMD Driver | 8.850.0.0 | ATI Radeon HD 4800 series |
| Adobe Flash | 10.3.183.5 | 10.3.183.5 |
| Microsoft Silverlight | 4.0.60531.0 | 4.0.60531.0 |
| Oracle Java | 6.0.260 | 14.0.3 |
Test Setup
We restart the computer and allow it to idle for a while before benchmarking the next browser. Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variations.
This time around, we're rating the benchmarks themselves. The tests are placed into one of four groups: core, observation, dated, and quarantine.
Core tests are considered current. These tests are usually trusted industry standards or our own creations, and they make up the core of the WBGP suite. Examples include our own page-load time tests, FutureMark Peacekeeper, and Mozilla Dromaeo DOM.
Tests that are either generally unknown, mostly untested, or just too bleeding-edge are placed under observation. The WebGL tests and Ecma test262 are examples.
Tests classified as dated are either outdated, losing relevance, or otherwise need replacing. We are actively seeking community feedback and contributions regarding alternatives to these benchmarks. Examples of this group include Acid3 and GUIMark Java.
The final group is for quarantined benchmarks. Benchmarks find their way into quarantine by delivering dubious results or by being gamed. Examples include previous suite dropouts Google V8, Dromaeo JavaScript, and now SunSpider as well.
Whenever the results of benchmarks that test the same thing conflict, more weight is given to tests with a better rating when creating the analysis tables.
The table below lists all 41 of the tests currently in our test suite (along with a version number, where applicable), current rating in the Web Browser Grand Prix, and number of iterations performed:
| Web Browser Grand Prix Test Suite 6.0 | ||
|---|---|---|
| Test Name | Iterations | Rating |
| Performance Tests (32) | ||
| Startup Time: Single Tab | 5 | Core |
| Startup Time: Eight Tabs | 5 | Core |
| Page Load Time: Google | 5 | Core |
| Page Load Time: YouTube | 5 | Core |
| Page Load Time: Yahoo! | 5 | Core |
| Page Load Time: Amazon | 5 | Core |
| Page Load Time: Wikipedia | 5 | Core |
| Page Load Time: eBay | 5 | Core |
| Page Load Time: craigslist | 5 | Core |
| Page Load Time: The Huffington Post | 5 | Core |
| Page Load Time: Tom's Hardware | 5 | Core |
| Peacekeeper | 3 | Core |
| Kraken v1.1 | 3 | Core |
| SunSpider v0.9.1 | 3 | Quarantine |
| Dromaeo DOM | 3 | Core |
| Maze Solver | 5 | Core |
| JSGameBench v4.1 | 3 | Core |
| Asteroids HTML5 Canvas 2D And JavaScript | 3 | Observation |
| GUIMark 2 HTML5 Vector Charting (1 pixel variant) | 5 | Core |
| GUIMark 2 HTML5 Bitmap Gaming | 5 | Core |
| GUIMark 2 HTML5 Text Columns | 5 | Core |
| Psychedelic Browsing | 3 | Core |
| Hardware Acceleration Stress Test | 3 | Dated |
| WebGL FishIE | 5 | Observation |
| WebGL Solar System | 5 | Observation |
| ThoughtsInComputation Particles | 5 | Observation |
| GUIMark2 Flash Vector Charting | 5 | Core |
| GUIMark2 Flash Bitmap Gaming | 5 | Core |
| GUIMark2 Flash Text Columns | 5 | Core |
| Flash Benchmark 2008 | 3 | Core |
| GUIMark Java | 5 | Dated |
| Encog Silverlight | 5 | Dated |
| Efficiency Benchmarks (4) | ||
| Memory Usage: Single Tab | 3 | Core |
| Memory Usage: 40 Tabs | 3 | Core |
| Memory Management: -39 Tabs | 3 | Core |
| Memory Management: -39 Tabs (extra 5 minutes) | 3 | Core |
| Reliability Benchmarks (1) | ||
| Proper Page Loads | 3 | Core |
| Conformance Benchmarks (4) | ||
| HTML5Test.com | 1 | Core |
| CSS3 Selectors Test | 1 | Core |
| Ecma test262 | 1 | Observation |
| Acid3 | 1 | Dated |
You've seen the lineup and toured the track. Now it's off to the races.
- Crowning A Web-Browsing King In Windows 7 And OS X
- The Contenders
- A Spotlight On Lion's Safari
- Hardware And Test Setup
- Performance Benchmarks: Startup Time
- Performance Benchmarks: Page Load Time
- Performance Benchmarks: JavaScript, DOM, And CSS
- Performance Benchmarks: Flash, Java, And Silverlight
- Performance Benchmarks: HTML5
- Performance Benchmarks: HTML5 Hardware Acceleration And WebGL
- Efficiency Benchmarks: Memory Usage
- Efficiency Benchmarks: Memory Management
- Reliability Benchmarks: Proper Page Loads
- Conformance Benchmarks: HTML5, CSS3, JavaScript, And DOM
- Placing Tables
- Analysis Tables
- Two Winners: One In Windows 7, One in OS X
thank you, workin' on it
chrome13 completely obliterats it.
and firefox 8/9 are still a memory hog.
not really surprised by poor show of ie9. moat updates it gets are "security updates".
Yeah? And exactly what principle would that be?
Bring back the Google Dictionary, otherwise I will use Bing Search, Firefox and Facebook instead of Google Search, Chrome and G+.
According to the graphic on "Reliability Benchmarks: Proper Page Loads" on MacOS Firefox is actually second, not third.
thank you, workin' on it
These "browser" GP are getting more and more complete and the're always very interesting.
I have to say, I am a bit surprised to see FF being so close to Chrome now: kudos to Mozilla.
I have been using FF since 1.0 and only recently coupled it with Chrome (it is just convenient for me to have 2 completely different setups).
FF 7.0 should have a significant boost in memory efficiency: if everything else stays the same, we´ll have a new champion ...
But if anythin is clear from these reviews, is that nothing stays the same for very long in the browser´s domain (well, except IE).
Looking forward to GP7, whenever that will be.
You should've put more emphasis on the actual scores and performances in tests rather than count the times when certain browsers placed 1st. Thus a browser that had a small advantage in more and minor tests and at the same time severe handicaps in more important but fewer tests would seem better, when technically it is not. Suggestion: tie all the candidates when the differences between them in a certain test are less than a single digit percent. Good article anyway.
And to think Apple hates Flash...
There are no points in the analysis tables. They simply list how each browser rates per category of testing. The 'Strong' part of the table was added a long time ago and it basically means that it's right up there with the winner in terms of performance. When we get a solid point-based scoring system figured out 'Winner' will only receive a minor boost above 'Strong', whereas 'Strong' will receive a significant boost above 'Acceptable', and 'Acceptable' above 'Weak'. We're not there yet, but we're getting closer with every WBGP. The composite tests are a BIG step in that direction, and the new benchmark rankings further lay the groundwork for a fair scoring system which accurately reflects scale.
The analysis tables were created to balance the raw placing tables. The problem with what you're saying is that you would have to decide which categories are more important than others. Is JavaScript more important than CSS? Is HTML5 more important than Flash? This is going to depend on who you ask. People who only watch Netflix with an HTPC will put mega emphasis on Silverlight perf, whereas the chronic YouTuber will be more concerned with Flash, and devs are going to gravitate towards standards conformance. Ranking benchmarks based on the importance of what they test isn't a one-size-fits-all type of thing with Web browsers. As far as your other suggestion, dealing with practical ties, this is something we definitely want to look into moving forward. Thanks!