Web Browser Grand Prix VI: Firefox 6, Chrome 13, Mac OS X Lion

Hardware And Test Setup

Hardware Setup

Test System Specs
Operating System 1
Microsoft Windows 7 Ultimate (64-bit)
Operating System 2
Apple Mac OS X Lion (64-bit)
Processor
Intel Core i5-750 (Lynnfield) @ 2.8 GHz, Quad-Core
Motherboard
Gigabyte GA-P55A-UD7, LGA 1156, P55 Express, F7 BIOS
Memory
8 GB Crucial DDR3 @ 1333 MT/s (2 x 4 GB)
Graphics
AMD Radeon HD 4870 Reference Boards 512 GB GDDR5 (PCI-e 2.0)
Storage
Seagate Barracuda 7200.12 500 GB SATA 3Gb/s, 7200 RPM, 16 MB Cache
Optical
Asus DRW-24B1ST/BLK/B/AS
Power Supply
Corsair TX750W (750 Watt Max)
Chassis
Zalman MS1000-HS2
CPU Cooler
Scythe Mugen 2 Revision B


The Windows 7 Test InstallationThe Windows 7 Test InstallationThe OS X Lion Test InstallationThe OS X Lion Test Installation

Local Web Server Specs
Operating System
Ubuntu 10.04 LTS Server Edition "Lucid Lynx" (32-bit)
Processor
AMD Athlon @ 1150 MHz
Motherboard
Soyo Dragon Platinum
Memory
512 MB DDR
Graphics
AMD Radeon 9550, 256 MB GDDR
Storage
40 GB Western Digital HDD WD400BB
Optical
Samsung DVD-ROM SD-616T
Extra Packages
Apache2, MySQL Client, MySQL Server, PHP5, PHP-GD, PHP5-MySQL, PHPMyAdmin, SSH


As requested by readers in our previous stories, the table below hosts additional information on the test network.

Network Specs
ISP Service
Cox Premium (28 Mb/s down, 5 Mb/s up)
Modem
Motorola SURFboard SBS101U
Router
Linksys WRT54G2 V1


Hackintosh Setup

The table below lists what we used to get the Lion-based Hackintosh up and running.

Installer
xMove
Bootloader
iBoot
DSDT
GA-P55A-UD7


While these tests are conducted on Mac OS X, this test system is not an Apple-branded Mac. We had to use a different bootloader than what you'd find on a genuine Mac, and real Mac systems use EFI instead of BIOS. Therefore, performance may vary from the same tests conducted on an Apple-branded system.

Also keep in mind that this use of OS X is completely unauthorized, and anyone doing this at home should have no expectations of receiving any help or support from Apple.

Software Setup

Both our Windows 7 Ultimate and Mac OS X Lion installations were freshly installed and fully updated as of midnight on August 15th. Power management and automatic updates were disabled before testing. The Web browsers and additional software, along with the exact version numbers tested, are listed in the table below.

Software
Windows Version
Mac Version
Chrome
13.0.782.215
13.0.782.215
Firefox
6.0
6.0
Internet Explorer
9.0.8112.16421
N/A
Opera
11.50 (1074)
11.50 (1074)
Safari
5.1 (7534.50)
5.1 (7534.48.3)
AMD Driver
8.850.0.0
ATI Radeon HD 4800 series
Adobe Flash
10.3.183.5
10.3.183.5
Microsoft Silverlight
4.0.60531.0
4.0.60531.0
Oracle Java
6.0.260
14.0.3


Test Setup

We restart the computer and allow it to idle for a while before benchmarking the next browser. Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variations.

This time around, we're rating the benchmarks themselves. The tests are placed into one of four groups: core, observation, dated, and quarantine.

Core tests are considered current. These tests are usually trusted industry standards or our own creations, and they make up the core of the WBGP suite. Examples include our own page-load time tests, FutureMark Peacekeeper, and Mozilla Dromaeo DOM.

Tests that are either generally unknown, mostly untested, or just too bleeding-edge are placed under observation. The WebGL tests and Ecma test262 are examples.

Tests classified as dated are either outdated, losing relevance, or otherwise need replacing. We are actively seeking community feedback and contributions regarding alternatives to these benchmarks. Examples of this group include Acid3 and GUIMark Java.

The final group is for quarantined benchmarks. Benchmarks find their way into quarantine by delivering dubious results or by being gamed. Examples include previous suite dropouts Google V8, Dromaeo JavaScript, and now SunSpider as well.

Whenever the results of benchmarks that test the same thing conflict, more weight is given to tests with a better rating when creating the analysis tables.

The table below lists all 41 of the tests currently in our test suite (along with a version number, where applicable), current rating in the Web Browser Grand Prix, and number of iterations performed:

Web Browser Grand Prix Test Suite 6.0
Test Name
Iterations
Rating
Performance Tests (32)
Startup Time: Single Tab
5
Core
Startup Time: Eight Tabs
5
Core
Page Load Time: Google
5
Core
Page Load Time: YouTube
5
Core
Page Load Time: Yahoo!
5
Core
Page Load Time: Amazon
5
Core
Page Load Time: Wikipedia
5
Core
Page Load Time: eBay
5
Core
Page Load Time: craigslist
5
Core
Page Load Time: The Huffington Post
5
Core
Page Load Time: Tom's Hardware
5
Core
Peacekeeper
3
Core
Kraken v1.1
3
Core
SunSpider v0.9.1
3
Quarantine
Dromaeo DOM
3
Core
Maze Solver
5
Core
JSGameBench v4.1
3
Core
Asteroids HTML5 Canvas 2D And JavaScript3
Observation
GUIMark 2 HTML5 Vector Charting (1 pixel variant)5
Core
GUIMark 2 HTML5 Bitmap Gaming5
Core
GUIMark 2 HTML5 Text Columns5
Core
Psychedelic Browsing
3
Core
Hardware Acceleration Stress Test
3
Dated
WebGL FishIE
5
Observation
WebGL Solar System
5
Observation
ThoughtsInComputation Particles
5
Observation
GUIMark2 Flash Vector Charting
5
Core
GUIMark2 Flash Bitmap Gaming
5
Core
GUIMark2 Flash Text Columns
5
Core
Flash Benchmark 2008
3
Core
GUIMark Java
5
Dated
Encog Silverlight
5
Dated
Efficiency Benchmarks (4)
Memory Usage: Single Tab
3
Core
Memory Usage: 40 Tabs
3
Core
Memory Management: -39 Tabs
3
Core
Memory Management: -39 Tabs (extra 5 minutes)
3
Core
Reliability Benchmarks (1)
Proper Page Loads
3
Core
Conformance Benchmarks (4)
HTML5Test.com
1
Core
CSS3 Selectors Test
1
Core
Ecma test262
1
Observation
Acid3
1
Dated


You've seen the lineup and toured the track. Now it's off to the races.

Create a new thread in the US Reviews comments forum about this subject
This thread is closed for comments
92 comments
    Your comment
  • ne0nguy
    The first chart says "higher is better" for the load time
    5
  • adamovera
    ne0nguyThe first chart says "higher is better" for the load time

    thank you, workin' on it
    2
  • SteelCity1981
    Chrome is the best browser out there right now. While FireFox maybe more popular then Chrome is, Chrome has shown why it is the best browser out today. If you haven't used Chrome yet it's def worth a look.
    8
  • soccerdocks
    The reader function in safari actually looks really nice. Although I'd never use Safari on principle. I hope other browsers implement a similar function.
    0
  • mayankleoboy1
    why does firefox(6/8/9) performa so horribly on the IE9 maze solover test?
    chrome13 completely obliterats it.

    and firefox 8/9 are still a memory hog.
    not really surprised by poor show of ie9. moat updates it gets are "security updates".
    0
  • tofu2go
    Being on a Macbook with only 3GB of memory, memory is the most important factor for me. I open a LOT of tabs and I keep them open for long periods. For awhile I used Chrome, but recently switched to Firefox 6 and saw my memory utilization drop by well over 1GB. Granted with Firefox I was able to do something I am not able to do in any other browser, I could group my tabs into tab groups. I believe this allows for more efficient memory management, i.e. only the current group uses much memory. Not having done any tests, this is pure speculation. All I know is that I'm seeing MUCH lower memory usage with Firefox on OSX. Despite what this article would suggest.
    -5
  • Anonymous
    @soccerdocks

    Yeah? And exactly what principle would that be?
    2
  • andy5174
    @Google:
    Bring back the Google Dictionary, otherwise I will use Bing Search, Firefox and Facebook instead of Google Search, Chrome and G+.
    -7
  • kartu
    Quote:
    Firefox 6 comes in third for both OSes, representing a major drop from Firefox 5.

    According to the graphic on "Reliability Benchmarks: Proper Page Loads" on MacOS Firefox is actually second, not third.
    0
  • LaloFG
    I keep Opera, more memory used and time to load pages is nothing when it load pages correctly; and the feeling in its interface is the greater.
    2
  • noob2222
    while these articles are entertaining, giving straight points skews the results a bit IMO. I think it would give more insight to give percentages in the analysis tables rather than just ranking them. After all, giving 1 pt for 5% better result out of the 5 is 20%, kinda throws off actual performance.
    0
  • adamovera
    kartuAccording to the graphic on "Reliability Benchmarks: Proper Page Loads" on MacOS Firefox is actually second, not third.

    thank you, workin' on it
    1
  • Anonymous
    On OSX browser 'vendors' are denied access to certain os hooks that would make their browsers better than they are.
    7
  • yankeeDDL
    Nice overview: thank you.
    These "browser" GP are getting more and more complete and the're always very interesting.
    I have to say, I am a bit surprised to see FF being so close to Chrome now: kudos to Mozilla.
    I have been using FF since 1.0 and only recently coupled it with Chrome (it is just convenient for me to have 2 completely different setups).
    FF 7.0 should have a significant boost in memory efficiency: if everything else stays the same, we´ll have a new champion ...
    But if anythin is clear from these reviews, is that nothing stays the same for very long in the browser´s domain (well, except IE).
    Looking forward to GP7, whenever that will be.
    1
  • Anonymous
    Adam, you should have mentioned in the end that even if Safari won on OSX, the victory is a pyrrhic one as OSX lacks in Java and Silverlight plugin performance; OSX Lion is also very poor at system memory management and reliability.

    You should've put more emphasis on the actual scores and performances in tests rather than count the times when certain browsers placed 1st. Thus a browser that had a small advantage in more and minor tests and at the same time severe handicaps in more important but fewer tests would seem better, when technically it is not. Suggestion: tie all the candidates when the differences between them in a certain test are less than a single digit percent. Good article anyway.
    5
  • cookoy
    Quote:
    Mac OS X is capable of providing better results than Windows 7 in Flash, HTML5, WebGL, and the ever-important page load times.


    And to think Apple hates Flash...
    9
  • damasvara
    Tried Chrome, but somehow it doesn't behave the way I wanted. Browsing pages is faster with Firefox on 384 kbps internet. Makes me wonder...
    0
  • adamovera
    noob2222while these articles are entertaining, giving straight points skews the results a bit IMO. I think it would give more insight to give percentages in the analysis tables rather than just ranking them. After all, giving 1 pt for 5% better result out of the 5 is 20%, kinda throws off actual performance.

    There are no points in the analysis tables. They simply list how each browser rates per category of testing. The 'Strong' part of the table was added a long time ago and it basically means that it's right up there with the winner in terms of performance. When we get a solid point-based scoring system figured out 'Winner' will only receive a minor boost above 'Strong', whereas 'Strong' will receive a significant boost above 'Acceptable', and 'Acceptable' above 'Weak'. We're not there yet, but we're getting closer with every WBGP. The composite tests are a BIG step in that direction, and the new benchmark rankings further lay the groundwork for a fair scoring system which accurately reflects scale.
    0
  • adamovera
    tgreaderAdam, you should have mentioned in the end that even if Safari won on OSX, the victory is a pyrrhic one as OSX lacks in Java and Silverlight plugin performance; OSX Lion is also very poor at system memory management and reliability.You should've put more emphasis on the actual scores and performances in tests rather than count the times when certain browsers placed 1st. Thus a browser that had a small advantage in more and minor tests and at the same time severe handicaps in more important but fewer tests would seem better, when technically it is not. Suggestion: tie all the candidates when the differences between them in a certain test are less than a single digit percent. Good article anyway.

    The analysis tables were created to balance the raw placing tables. The problem with what you're saying is that you would have to decide which categories are more important than others. Is JavaScript more important than CSS? Is HTML5 more important than Flash? This is going to depend on who you ask. People who only watch Netflix with an HTPC will put mega emphasis on Silverlight perf, whereas the chronic YouTuber will be more concerned with Flash, and devs are going to gravitate towards standards conformance. Ranking benchmarks based on the importance of what they test isn't a one-size-fits-all type of thing with Web browsers. As far as your other suggestion, dealing with practical ties, this is something we definitely want to look into moving forward. Thanks!
    0
  • Anonymous
    I had to switch to Chrome, FF was crashing like crazy here, and i only have Firebug add-on installed.
    0