Sign in with
Sign up | Sign in

Test Setup And Benchmark Suite

Web Browser Grand Prix: Chrome 25, Firefox 19, And IE10
By

While our test hardware has not changed since Which Web Browser Is Best Under Windows 8?, our benchmark suite underwent substantial changes. First, the startup times are now taken using a stopwatch, and the test pages are hosted from our local Web server. Using a stopwatch includes the time it takes for the actual applications (Web browsers) to open, as opposed to just timing how long it takes the browsers to load their tabs. By hosting the test pages from the local Web server, we cut out the instability and variation that comes with testing live pages.

As you may gave guessed, since we ditched our startup timers, we also ditched the page load timers that they're based on. We now use EEMBC's BrowsingBench to gauge page load times. This new test addresses several of the shortcomings that plagued our old page load timers; for instance, it tests both desktop and mobile pages, multiple pages on the same site, foreign sites, and it runs multiple iterations per test run.

Moving on to JavaScript, we finally retired SunSpider from the test suite. This benchmark hasn't been updated in several years, and there are far better JavaScript benchmarks available today (such as Kraken, which we're also retiring in favor of an even better test: Rightware Browsermark). Octane will not appear in the WBGP because the V8 portion still skews the results far in Chrome's favor. RIABench and Peacekeeper remain in the JS portion of the test suite. In order to minimize the effect of Dromaeo DOM in the final scoring, that test is now counted equally with the three JS tests in the final JavaScript/DOM composite score.

The HTML5 section also received several cuts, including GUIMark 2, Asteroids, and Smashcat. Only Impact remains. Joining it is the first consumer preview of Principled Technologies' WebXPRT, an HTML5 applications benchmark that frames the performance testing around common office Web apps, such as an image editor, stock tracker, and notes.

The final massive change to the benchmark lineup is in WebGL, both WebGL Solar System and Mozilla's WebGL FishIE Tank were replaced. Airtight Interactive's WebGL Demo and Scirra's WebGL Performance Test now make up that portion of the test suite.

Test Setup And Benchmark Suite

Test System Specs
Operating System 1
Microsoft Windows 7 Ultimate SP1 (64-bit)Microsoft Windows 7 Ultimate SP1 (64-bit)
Operating System 2
Microsoft Windows 8 (64-bit)Microsoft Windows 8 (64-bit)
Processor
Intel Core i5-2500K @ 3.3 GHz (quad-core)
Motherboard
Gigabyte GA-Z68XP-UD3 (F10 BIOS)
Memory
8 GB Crucial DDR3 @ 1333 MT/s (2 x 4 GB)
Graphics
Asus GeForce GTX 560 Ti 1 GB GDDR5 (PCIe 2.0 x16)
Storage
Seagate Barracuda 7200.12 500 GB SATA 3Gb/s, 7200 RPM, 16 MB Cache
Optical
Asus DRW-24B1ST/BLK/B/AS
Power Supply
Corsair TX750W (750 W max)
Case
Zalman MS-1000 HS2
CPU Cooler
Scythe Mugen 2 Revision B
Monitor
AOC E2752Vh 27-inch LED (1920x1080)
Keyboard
Logitech Wireless Keyboard K320
Mouse
Logitech Wireless Trackball M570
Local Web Server Specs
Operating System
Ubuntu 12.04.1 LTS Server Edition "Precise Pangolin" (32-bit)
Processor
Intel Pentium 4 @ 2.41 GHz
Motherboard
Biostar P4M80-M4
Memory
768 MB DDR @ 333 MT/s
Storage
Western Digital Caviar SE WD1600AAJD, 160 GB EIDE, 7200 RPM
Extra Packages
Apache2, MySQL Client, MySQL Server, PHP5, PHP-GD, PHP5-MySQL, PHPMyAdmin, SSH, Node.js, NPM
Network Specs
ISP Service
Cox Preferred (18 Mb/s down, 2 Mb/s up)
Modem
Arris Touchstone Telephony Modem TM502G
Router
Linksys WRT54G2 V1
Benchmark Suite
Startup Time
Cold Start Time (Google SERP, Cached)
Hot Start Time (Google SERP, Cached)
Cold Start Time (Eight Tabs, Cached)
Hot Start Time (Eight Tabs, Cached)
Page Load Time
EEMBC BrowsingBench
JavaScriptRIABench JavaScript (Eight Tests)
Futuremark Peacekeeper v2.0
Rightware Browsermark v2.0
DOMMozilla Dromaeo DOM (Core)
HTML5
Principled Technologies WebXPRT CP1
Impact HTML5 Benchmark
Hardware Acceleration
Facebook JSGameBench v0.4.1
HTML HWA
WebVizBench
Psychedelic Browsing
WebGLAirtight Interactive WebGL Demo
Scirra WebGL Performance Test
Memory Efficiency
Memory Usage (Single Tab)
Memory Usage (40 Tabs)
Memory Management (-39 Tabs)
Memory Management (-39 Tabs, Two Additional Minutes)
Reliability
Proper Page Loads
SecurityBrowserscope Security
Standards ConformanceHTML5Test.com
The CSS3 Test
Ecmascript Language test262


While applicable links are included in the table above, we also have a public delicious account dedicated to Web Browser Grand Prix benchmark links.

Detailed methodologies are explained on the individual benchmark pages.

Display all 84 comments.
This thread is closed for comments
Top Comments
  • 14 Hide
    alidan , March 13, 2013 10:40 PM
    i wish we could get some older browsers in here to to see the difference newerones make.

    i personally am still on ff10, memory is really the reason i use fire fox, along with chrome. i would love to see how older versions stack up to newer ones.
Other Comments
  • 1 Hide
    mayankleoboy1 , March 13, 2013 10:31 PM
    1. I am assuming that IE9 was included just to show improvements in IE10, and IE9 will be retired in the next WBGP

    2. " Opera jumps on-board the Chromium bandwagon" is false. Opera is using Webkit rendering engine for rendering. We dont know what the Javascript engine and JIT compiler is going to be.

    3. On page4, in "the average wait time" , you are taking the Geometric Mean. I think that using values less than 1 in a geometric mean is skewing the results. Can you take a Arithmetic Mean, and check please ?

    4. Memory usage reporting of IE9 and IE10 is completely bogus. For some workloads, in TaskManager, IE10 is seen using about 300MB memory, while it is actually using around 3GB RAM ! :o  . My 4GB RAM is completely used up, and disk-paging starts occuring.

    5. I am completely mystified why quite a few browsers do better in JS benchmarks on Windows8 . I can understand getting better scores in WebGL or HW acceleration tests (because of potential better Dx) , but JavaScript execution does not use Dx. Any ideas ?
  • 14 Hide
    alidan , March 13, 2013 10:40 PM
    i wish we could get some older browsers in here to to see the difference newerones make.

    i personally am still on ff10, memory is really the reason i use fire fox, along with chrome. i would love to see how older versions stack up to newer ones.
  • 5 Hide
    bison88 , March 13, 2013 11:43 PM
    Opera is definitely lagging, but a I love it to death. Hope the switch to the Webkit platform gives them the motivation to start leading some of the boards again as it's still a great browser. The memory problem can get absurd sometimes and builds up fast. I have to say IE 10 has shocked me just toying around with it on Windows 7. The thing is smooth and fast, something I haven't seen since the days when it dethroned Netscape.
  • 4 Hide
    adamovera , March 14, 2013 12:01 AM
    mayankleoboy11. I am assuming that IE9 was included just to show improvements in IE10, and IE9 will be retired in the next WBGP2. " Opera jumps on-board the Chromium bandwagon" is false. Opera is using Webkit rendering engine for rendering. We dont know what the Javascript engine and JIT compiler is going to be.3. On page4, in "the average wait time" , you are taking the Geometric Mean. I think that using values less than 1 in a geometric mean is skewing the results. Can you take a Arithmetic Mean, and check please ?4. Memory usage reporting of IE9 and IE10 is completely bogus. For some workloads, in TaskManager, IE10 is seen using about 300MB memory, while it is actually using around 3GB RAM ! . My 4GB RAM is completely used up, and disk-paging starts occuring.5. I am completely mystified why quite a few browsers do better in JS benchmarks on Windows8 . I can understand getting better scores in WebGL or HW acceleration tests (because of potential better Dx) , but JavaScript execution does not use Dx. Any ideas ?

    1. Yup
    2. The press release mentioned "Chromium", so I'm assuming WebKit/V8.
    3. Yes, you're right! The last timers went by milliseconds, so that wasn't an issue - the replacement charts should appear soon. Good catch!
    4. It seems pretty reasonable to me, basically mirrors Chrome in this regard.
    5. Nope
  • 4 Hide
    caspy7 , March 14, 2013 12:15 AM
    @mayankleoboy1
    #2, incorrect, they have said they're going with Chromium's V8.

    @alidan
    Mozilla saw the err of their ways and got after memory. The most recent version of Firefox should beat version 10 in both memory and performance. The last benchmarks I saw had it beating all the other browsers in memory usage as well. (In this article you can see a snapshot of this in the "40 tabs" graphic. Like many techies, this is more what my browsing looks like.)
  • -1 Hide
    mayankleoboy1 , March 14, 2013 12:30 AM
    adamovera1. Yup2. The press release mentioned "Chromium", so I'm assuming WebKit/V8.3. Yes, you're right! The last timers went by milliseconds, so that wasn't an issue - the replacement charts should appear soon. Good catch!4. It seems pretty reasonable to me, basically mirrors Chrome in this regard.5. Nope


    #2 : I must have forgotten the release. Thanks for the correction.
    #4 : I am not saying that IE10 uses excessive memory. I am saying that the total memory used by all the ieexplorer processes in the task manager is incorrect. For some image heavy pages, total memory usage of IE as reported by task manager is about 300MB, but total system memory usage gets around 3GB!. So if you close IE10, system memory goes from 90% full to 50% full.
  • 1 Hide
    slomo4sho , March 14, 2013 12:32 AM
    I currently use Firefox, Chrome, and IE 10. They each have their own advantages. Thanks for the write up. When can be expect the Android browser round up?
  • 3 Hide
    xintox , March 14, 2013 12:51 AM
    Really? I thought you were better than that at Toms.
    The composite hardware acceleration scores is most likely the main reason why IE9/10 is so far behind Firefox and Chrome on performane. Yet, from what I can tell, this composite score is heavily influenced by the WebGL scores, which is exclusive to Chrome and Firefox.
    In that respect, MS has at some point stated that they do not even wish to support WebGL, as it represents a significant security risk, as it gives the browser close access to the computer hardware.
    Long story short, your methods of calculating performance scores heavily favors Chrome and Firefox as they are the only ones to implement support for WebGL.

    Additionally, I wish you would make it more clear how you arrive at your composite scores and of course the final Performance Index. How do you add numbers that are so varied in nature, without some method of normalizing the numbers?
  • 5 Hide
    srap , March 14, 2013 12:54 AM
    @adamovera
    I heard that Futuremark Peacekeeper is unreliable, it used to miscalculate it's own benchmarks and it's still a black box so it may still be buggy. Do you know anything specific about it?
  • 5 Hide
    abbadon_34 , March 14, 2013 1:10 AM
    Nice to see IE9 AND IE10, as well as Win 7 AND Win8. But I am curious how the final scores were calculated, seeing how WebGL was only supported on 2 browsers, as well as how the other incompatible tests were scored. A 0 vs not including can make a good deal of difference.
  • 2 Hide
    Anukul Bodile , March 14, 2013 2:22 AM
    i prefer opera over chrome because of its intuitive interface
  • 3 Hide
    Octiva , March 14, 2013 2:58 AM
    CHROME STEAL ALL YOUR PROCCESOR POWER, use it to send infomation MOUNTVIEW IP
  • 1 Hide
    Octiva , March 14, 2013 2:59 AM
    Make test on packet in IDLE MODE , 1 tab open (empty)
  • -7 Hide
    merikafyeah , March 14, 2013 3:02 AM
    You call it a Grand Prix, but you use stock browsers. Not very exciting. You don't watch Formula 1 for stock cars. Tom's needs an "Enthusiast Grand Prix" with experts tweaking and modding each browser to fully maximize its potential, then have it compete against other tricked out browsers, the way it should be. No enthusiast uses an out-of-box vanilla browser.
  • 2 Hide
    srap , March 14, 2013 3:55 AM
    merikafyeahYou call it a Grand Prix, but you use stock browsers. Not very exciting. You don't watch Formula 1 for stock cars. Tom's needs an "Enthusiast Grand Prix" with experts tweaking and modding each browser to fully maximize its potential, then have it compete against other tricked out browsers, the way it should be. No enthusiast uses an out-of-box vanilla browser.

    There is a reason why he uses the default install settings, and he explained this before in the former WBGP. The number of enthusiasts are far smaller than the 1-bit users, thus the default. Also, most enthusiasts actually know what is to be expected from the mods/tweaks, but normal users hardly know anything about browsers (if they even know the term: internet browser).
  • -6 Hide
    belardo , March 14, 2013 3:57 AM
    Wait a minute... firefox 19? It was little over a year ago that version 4.0 came out?! So its really version 4.18? right?
  • 1 Hide
    belardo , March 14, 2013 4:02 AM
    "Before Opera becomes yet another Webkit browser and we're down to the top three," What does that mean? That its taken off the list because it'll be a webkit? Chrome is already a webkit.
  • 1 Hide
    Onus , March 14, 2013 4:06 AM
    I use IE at work because I have to. I use FF (with ABP & NoScript) at home for everything except those few things (like some router config pages) that only work right in IE. I don't trust Chrome not to unnecessarily report my every move to Google, and what small speed "advantage" it may have is just not relevant, like someone bragging about getting 110 FPS in a game when I'm getting "only" 104.
  • 0 Hide
    srap , March 14, 2013 4:16 AM
    belardo"Before Opera becomes yet another Webkit browser and we're down to the top three," What does that mean? That its taken off the list because it'll be a webkit? Chrome is already a webkit.

    Because if it switches to webkit, all of it's test results will be identical to Chrome's, or it will hardly differ.

    caspy7The most recent version of Firefox should beat version 10 in both memory and performance.
    Firefox 15: HueyFix, details here.

    @alidan: FF17 ESR would be more safe than FF10, and even better with memory.

    Edit: I hate how it keeps deleting the links.
  • 1 Hide
    belardo , March 14, 2013 4:23 AM
    Oh... well, as long as Opera retains its UI/functions... it wouldn't bother me much. I use both Chrome and Opera. I prefer Opera, but use Chrome when a site doesnt like Opera.

    Same for my Android phone.
Display more comments