Sign in with
Sign up | Sign in

WBGP VIII Test Setup

Web Browser Grand Prix VIII: Chrome 16, Firefox 9, And Mac OS X
By

Hardware Setup

Test System Specs
Operating System 1
Microsoft Windows 7 Ultimate (64-bit)
Operating System 2
Apple Mac OS X Lion (64-bit)
Model
Apple MacBook Air 11.6-inch (late 2011)
Processor
Intel Core i7 @ 1.8 GHz (dual-core)
Memory
4 GB DDR3 @ 1333 MT/s
Graphics
Intel HD Graphics 3000 384 MB DDR3 SDRAM
Storage
128 GB SSD


Behold The Glory!Behold The Glory!

Software Setup

Our test installations are freshly installed and fully updated as of midnight on December 20th. Power management and automatic updating is disabled before testing. The Web browsers and additional software, along with the exact version numbers tested, are listed in the table below.

Software
Version
Chrome16.0.912.63 m
Firefox
9.0.1
Internet Explorer
9.0.8112.16421
Opera
11.60 (build 1185)
Safari
5.1.2 (7534.5.2.7)
Adobe Flash11.1.102.55 (64-bit)
Microsoft Silverlight5.0.61118.0
Oracle Java6.0.30


The Windows 7 Test InstallationThe Windows 7 Test Installation

The Mac OS X Lion Test InstallationThe Mac OS X Lion Test Installation

Network Setup

The following table contains the system specs of the local Web server used for our startup time tests, the page load time tests, and JSGameBench.

Local Web Server Specs
Operating System
Ubuntu 10.04 LTS Server Edition "Lucid Lynx" (32-bit)
Processor
AMD Athlon @ 1150 MHz
Motherboard
Soyo Dragon Platinum
Memory
512 MB DDR
Graphics
AMD Radeon 9550, 256 MB GDDR
Storage
40 GB Western Digital HDD WD400BB
Optical
Samsung DVD-ROM SD-616T
Extra Packages
Apache2, MySQL Client, MySQL Server, PHP5, PHP-GD, PHP5-MySQL, PHPMyAdmin, SSH


The table below holds additional information on the test network.

Network Specs
ISP Service
Cox Premium (28 Mb/s down, 5 Mb/s up)
Modem
Motorola SURFboard SBS101U
Router
Linksys WRT54G2 V1

 
While we typically use a hard-wired Ethernet connection, due to the MacBook Air being the test system, we're debuting the use of Wi-Fi in the Web Browser Grand Prix.

Test Setup

We restart the computer and allow it to idle for a few minutes before benchmarking the next browser. Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variance.

All tests are placed into one of four groups: core, observation, dated, and quarantine. Core tests are considered current. These tests are usually trusted industry standards or our own creations, and they make up the core of the WBGP suite. Tests that are either generally unknown, mostly untested, or just too bleeding-edge are placed under observation. Tests classified as dated are either outdated, losing relevance, or otherwise need replacing. We are actively seeking community feedback and contributions regarding alternatives to these benchmarks. The final group is for quarantined benchmarks. Benchmarks find their way into quarantine by delivering dubious results or by being gamed. Whenever benchmarks that test the same thing yield conflicting results, more weight is given to tests with a better rating when creating the analysis tables.

The table below lists all 53 of the tests currently in our suite (along with a version number and link, where applicable), number of iterations performed, and current rating:

Tom's Hardware Web Browser Grand Prix Test Suite v8.0
Test Name
Iterations
Rating
Performance Tests (44)
Cold Startup Time: Single Tab
3
Core
Cold Startup Time: Eight Tabs3
Core
Hot Startup Time: Single Tab3
Core
Hot Startup Time: Eight Tabs
3
Core
Uncached Page Load Times (9 Test Pages)
5
Core
Cached Page Load Times (9 Test Pages)
5
Core
Kraken v1.1
2
Core
Google Kraken v1.1 Mod2
Observation
SunSpider v0.9.12
Observation
Google SunSpider v0.91 Mod2
Observation
FutureMark Peacekeeper 2.02
Core
Dromaeo DOM2
Core
Maze Solver5
Core
GUIMark2 Flash Vector Charting3
Core
GUIMark2 Flash Bitmap Gaming3
Core
GUIMark2 Flash Text Columns3
Core
Flash Benchmark 2008 v1.09.1
2
Core
GUIMark Java3
Dated
Encog Silverlight3
Dated
Facebook JSGameBench v4.12
Core
GUIMark 2 HTML5 Vector Charting (1 pixel variant)
3
Core
GUIMark 2 HTML5 Bitmap Gaming
3
Core
GUIMark 2 HTML5 Text Columns
3
Core
Asteroids HTML5 Canvas 2D And JavaScript2
Core
Psychedelic Browsing
2
Observation
Hardware Acceleration Stress Test
2
Dated
Mozilla WebGL FishIE
5
Observation
WebGL Solar System5
Observation
Efficiency Benchmarks (5)
Memory Usage: Single Tab
3
Core
Memory Usage: 40 Tabs
3
Core
Memory Management: -39 Tabs
3
Core
Memory Management: -39 Tabs (extra 2 minutes)
3
Core
Reliability Benchmarks (1)
Proper Page Loads
3
Core
Conformance Benchmarks (3)
Ecma test2621
Core
Peacekeeper 2.0 HTML5 Capabilities1
Core
HTML5Test.com1
Core


Legend

Throughout this article there are a ton of charts, many of which house data from both Windows 7 and Mac OS X. The data in these charts is sorted by the Windows 7 score, with the best performer on top and the worst at the bottom. Windows 7 scores are represented by blue bars and OS X scores are in green. If an OS X browser outperforms all of the Windows 7 browsers, we'll indicate this occurrence by changing the color of the winning OS X browser from green to red.

Now, let's fire off the starting pistol and let the race begin!

Ask a Category Expert

Create a new thread in the Reviews comments forum about this subject

Example: Notebook, Android, SSD hard drive

Display all 152 comments.
This thread is closed for comments
Top Comments
  • 21 Hide
    twztechman , January 6, 2012 3:58 AM
    Been using Firefox for years - it works best for me.
  • 10 Hide
    Anonymous , January 6, 2012 5:42 AM
    Nobody else seems to have noticed that this article tries to rate browsers by speed differences that are far less than the blink of an eye. A browser is a tool - just use the ones you want and stop bothering people about their own choices. (I keep two or three loaded on my machine in case one of them has an issue with a particular webpage)
Other Comments
  • 21 Hide
    twztechman , January 6, 2012 3:58 AM
    Been using Firefox for years - it works best for me.
  • -1 Hide
    shiftmx112 , January 6, 2012 4:24 AM
    This makes it worth putting up with the constant updates on Aurora. :) 
  • 9 Hide
    Anonymous , January 6, 2012 4:30 AM
    The best part is I'm quite sure that this is using an out of the box build. Using a PGO compiled nighlty build, with about:config properly configured, and addons like Adblock/NoScript blocking things from ever loading Firefox is significantly faster than these benchmarks state.
  • 2 Hide
    frost_fenix , January 6, 2012 4:40 AM
    I have use firefox and chrome interchangeably for a few years now. I enjoy chromes streamlined design but have recently discovered the noscript addon for Firefox and have since favored Firefox. I have also found Firefox to be more compatable with school webpages and application pages. Still either firefox or chrome is better than IE.
  • 4 Hide
    pharoahhalfdead , January 6, 2012 4:44 AM
    Good point Stoof. I have IE9 and the newest FF, and with the FF add ons, it blows IE out of the water. The majority of IE pages like yahoo video links, boxingscene etc take 6 or more seconds to load, whereas FF is only a fraction of the time.

    I think add ons are much easier to find with FF, and there seems to be a wider variety. Then again I do realize this article wasn't about browsers with add ons.
  • 4 Hide
    hardcore_gamer , January 6, 2012 4:50 AM
    The only one thing I hate about firefox is that it takes a lot of time to launch.
  • 6 Hide
    adamovera , January 6, 2012 5:02 AM
    stoofThe best part is I'm quite sure that this is using an out of the box build. Using a PGO compiled nighlty build, with about:config properly configured, and addons like Adblock/NoScript blocking things from ever loading Firefox is significantly faster than these benchmarks state.

    Yes, we're using everything stock. There is no one-size-fits-all combination of plug-ins to standardize on, and every browser might not have the exact same plugins available. So that throws out a fair comparison between browsers - wouldn't work for the WBGP. Perhaps an article concentrating specifically on Firefox (or another Web browser) with and without various plug-ins would clear that up?
  • 6 Hide
    Anonymous , January 6, 2012 5:09 AM
    Please use Firefox's latest logo, the one with the shiny orb in Mozilla's press kit! The one they're using now is the old one. http://www.mozilla.org/en-US/firefox/brand/identity/
  • -2 Hide
    nevertell , January 6, 2012 5:21 AM
    Chrome is the easiest to use if you've got lots of tabs open. Scrolling through them with mouse is a breeze and tab management is just excellent.
  • -9 Hide
    soccerdocks , January 6, 2012 5:28 AM
    frost_fenix. I enjoy chromes streamlined design but have recently discovered the noscript addon for Firefox and have since favored Firefox.


    Why do people seem to forget Chrome has this built in. All you have to do is go into the options menu and disable JavaScript.
  • 0 Hide
    de5_Roy , January 6, 2012 5:32 AM
    Firefox!!! oh yeah. i use ff and the new one kinda feels snappier. the addons are awesome.
    i know i know, chrome is faster, has market share, ie 9/10 are coming up, blah blah. but ff can still fight. google's benevolent (read: to antitrust-pacifier) fund injection should help ff. besides, chrome is a sneakware bundled with numerous softwares. ff has scriptblockers that block statcounter. :D 
  • 10 Hide
    Anonymous , January 6, 2012 5:42 AM
    Nobody else seems to have noticed that this article tries to rate browsers by speed differences that are far less than the blink of an eye. A browser is a tool - just use the ones you want and stop bothering people about their own choices. (I keep two or three loaded on my machine in case one of them has an issue with a particular webpage)
  • 2 Hide
    adamovera , January 6, 2012 6:04 AM
    Mozilla-fanPlease use Firefox's latest logo, the one with the shiny orb in Mozilla's press kit! The one they're using now is the old one. http://www.mozilla.org/en-US/firefox/brand/identity/

    Thanks for the feedback, and good catch. I must have goofed and started making the graphics with an older file when I already had the newer one. Doh! It's all fixed now, and it should update momentarily.
  • -4 Hide
    arcus_doom , January 6, 2012 6:23 AM
    Google Chrome basically sucked after they removed the Side Tabs option.
  • 2 Hide
    arunloveshacking , January 6, 2012 6:24 AM
    Have you ever noticed that Firefox hangs up whenever you switch tabs while attaching a file? Or is it only for me?
  • 6 Hide
    Anonymous , January 6, 2012 6:30 AM
    Dear Adam, you might want to remove the html5.com test as it represents nothing but a checklist. In no way does it test the correct implementation or functioning of html5 features.
  • 2 Hide
    bodyknight , January 6, 2012 7:16 AM
    I haven't read the name Soyo in a review since a decade ago...
  • 1 Hide
    MCstrick , January 6, 2012 7:52 AM
    nevertellChrome is the easiest to use if you've got lots of tabs open. Scrolling through them with mouse is a breeze and tab management is just excellent.


    Firefox can do the same with tab mix plus. I couldn't live without scrolling though my tabs.
  • 2 Hide
    freggo , January 6, 2012 8:33 AM
    Firefox wins, yet Chrome is #1 in usage.
    Just like VHS vs Beta, NTSC vs PAL or Gasoline vs Electric... just because the public likes something does not mean it is the best solution.
  • 1 Hide
    mayne92 , January 6, 2012 10:00 AM
    Great review Adam...as usual! Awesome work. A good read with my morning coffee! :) 
Display more comments