WBGP9 Test Suite And Methodology
We restart the computer and allow it to idle for a few minutes before benchmarking each browser. Other than the conformance benchmarks, all of our final scores are an average of several iterations. More iterations are run on tests that have short durations, lower scales, and/or higher variance.
All tests are placed into one of four groups: core, observation, dated, and quarantine. Core tests are considered current. These tests are usually trusted industry standards or our own creations, and they make up the foundation of the WBGP suite. Tests that are either generally unknown, mostly untested, or just too bleeding-edge are placed under observation. Tests classified as dated are either outdated, losing relevance, or otherwise need replacing. We are actively seeking community feedback and contributions regarding alternatives to these benchmarks. The final group is for quarantined benchmarks. Benchmarks find their way into quarantine by delivering dubious results or by being gamed. Whenever benchmarks that test the same thing yield conflicting results, more weight is given to tests with a better rating when creating the analysis tables.
The table below lists all 51 of the tests currently in our suite (along with a version number and link, where applicable), number of iterations performed, and current rating:
| Tom's Hardware Web Browser Grand Prix Test Suite v9.0 | ||
|---|---|---|
| Test Name | Iterations | Rating |
| Performance Tests (44) | ||
| Cold Startup Time: Single Tab | 3 | Core |
| Cold Startup Time: Eight Tabs | 3 | Core |
| Hot Startup Time: Single Tab | 3 | Core |
| Hot Startup Time: Eight Tabs | 3 | Core |
| Uncached Page Load Times (9 Test Pages) | 5 | Core |
| Cached Page Load Times (9 Test Pages) | 5 | Core |
| Kraken v1.1 | 2 | Core |
| Google SunSpider v0.91 Mod | 2 | Core |
| FutureMark Peacekeeper 2.0 | 2 | Core |
| Dromaeo DOM | 2 | Core |
| Maze Solver | 5 | Core |
| GUIMark2 Flash Vector Charting | 3 | Core |
| GUIMark2 Flash Bitmap Gaming | 3 | Core |
| GUIMark2 Flash Text Columns | 3 | Core |
| Flash Benchmark 2008 v1.09.1 | 2 | Core |
| GUIMark Java | 3 | Dated |
| Encog Silverlight | 3 | Dated |
| Facebook JSGameBench v4.1 | 2 | Core |
| GUIMark 2 HTML5 Vector Charting (1 pixel variant) | 3 | Core |
| GUIMark 2 HTML5 Bitmap Gaming | 3 | Core |
| GUIMark 2 HTML5 Text Columns | 3 | Core |
| Asteroids HTML5 Canvas 2D And JavaScript | 2 | Observation |
| Psychedelic Browsing | 2 | Core |
| Hardware Acceleration Stress Test | 2 | Dated |
| Mozilla WebGL FishIE | 5 | Core |
| WebGL Solar System | 5 | Observation |
| Efficiency Benchmarks (5) | ||
| Memory Usage: Single Tab | 3 | Core |
| Memory Usage: 40 Tabs | 3 | Core |
| Memory Management: -39 Tabs | 3 | Core |
| Memory Management: -39 Tabs (extra 2 minutes) | 3 | Core |
| Reliability Benchmarks (1) | ||
| Proper Page Loads | 3 | Core |
| Conformance Benchmarks (3) | ||
| Ecma test262 | 1 | Core |
| Peacekeeper 2.0 HTML5 Capabilities | 1 | Core |
| HTML5Test.com | 1 | Core |
Detailed individual methodologies are described on the pages corresponding to each benchmark.
Legend
There are a ton of charts throughout this article, many of which house data from both Windows 7 and Ubuntu 11.10. Data is sorted according to the Windows 7 score, with the best performer on top and the worst at the bottom. Windows 7 scores are represented by blue bars and Ubuntu 11.10 scores are in green. If an Ubuntu-based browser outperforms all of the Windows 7 browsers, we indicate this by changing the color of the winning Ubuntu browser from green to red.
Related Stories :
- Web Browser Grand Prix 9: Chrome 17, Firefox 10, And Ubuntu
- Web Browser Grand Prix: Chrome 25, Firefox 19, And IE10
- Firefox 18 Posts Speed Gains Over Chrome, Mozilla Says
- Web Browser Grand Prix: Chrome 20, Opera 12, Firefox 13
- Web Browser Grand Prix: Chrome 18, Firefox 11, Windows XP
- Web Browser Grand Prix VIII: Chrome 16, Firefox 9, And Mac OS X
- Google's Done It: Chrome Passes Firefox in Market Share
- Chrome to Finally Pull Ahead of Firefox in November
- Mozilla Dev on How to Convert Chrome User to Firefox
- Chrome Climbed Above Firefox in Market Share Ranking
just wondering if use of a DX11 capable GPU will change scores in some HTML5 and other benchmarks as the browsers use DX11 assisted rendering.
Also, AMD driver support in linux is poor compared to Nvidia.
For future Linux articles, can you use a Dx11 based Nvidia GPU?
IMO, Firefox is concentrating more on HTML5, ignoring CSS and JavaScript.
It does well in HTML5 benches but 99% of the websites use primarily CSS and JS and HTML3, in which Firefox does poorly.
Waiting for OPERA12. It keeps impressing me.
Even without hardware acceleration, it keeps up with the competition,
When that beast launches, it will kill FF/IE and most probably chrome too.
Who wants to guess that the poor Linux Flash and WebGL results were because Flash and WebGL don't use hardware acceleration with that graphics card and driver? I would be thinking so.
Firefox performance took a dive starting with version 4, where all hardware acceleration was disabled: before then, in version 3.6, XRENDER was used when available (it was 4/5th as fast as IE9 on the same PC) while it is now really slow - it's all software.
Moreover, the only driver enabled for hardware acceleration on Linux is the Nvidia driver: according to Mozilla (and verified by yours truly on AMD and Intel hardware), most display drivers in Linux suck when it comes to 2D rendering - ouch. Note that Mozilla and Google could add shims to circumvent those bugs, but they don't -not worth the effort, especially when driver makers could fix their bugs rather easily, leaving the browsers broken yet again.
I use Chrome (19.0.1041.0 dev presently) the most on Linux (Ubuntu) and empirically I felt Chrome works very well. Now your tests confirm it.
I find Opera 12 really nice too. It can run with Opera 11.61. Opera 12 has a silver icon & 11.61 has its classic red. I like Firefox & Epiphany too.
Its a shame Safari and IE are not truly cross-platform.
how many of those top 40 sites use HTML5?
i think that the HTML5 scores should be weighed by a factor of the percent of top40 sites that use HTML5.
This way actual importance of HTML5 can be judged in real world.
It's a shame Apple does not pay enough attention to the Windows market and optimize their browser! On Mac Safari is king of the hill - personal opinion of course!
On Windows I feel that IE9 works really well for me, although Chrome is the speed demon! FF 4+ lost their appeal for me.
The OSes that are used are 64 bits but the browsers are mostly (all?) 32bits on Windows, and probably 64bits on Linux.
Internet Explorer has 64bits builds on Win7, and Firefox has "almost" a 64bits browser on Windows too: Waterfox, which is a semi-official Firefox for 64bits Windows. Waterfox in particular claims huge improvements over base 32bits install, I would like to see how that translates into real-world.
Not sure about availability of 64bits editions of other browsers on Windows.
Here are my wishes:
-clearly mention if the 32bits or 64bits version of the browser is used
-where applicable and relevant, test with both 32bits and 64bits variants. I would like to see IE and FF split into 32 and 64 variants on Win for example.
I personally migrated from FF to WF on my machines 3 weeks ago and find it noticeably faster in everyday use. WF is now my main browser.
As long as phones keep using Android, Chrome will be the most popular browser for a long while. Google have got it all sorted.
One test that is missing in almost all browser test is memory leak over time
I find almost all browsers require more and more memory the longer they run
Example: Chrome 17: 8 new tabs =1500MB
This sentence makes no sense to me. how can it "win" where it "simply cannot compete"?
The OSes that are used are 64 bits but the browsers are mostly (all?) 32bits on Windows, and probably 64bits on Linux.
Internet Explorer has 64bits builds on Win7, and Firefox has "almost" a 64bits browser on Windows too: Waterfox, which is a semi-official Firefox for 64bits Windows. Waterfox in particular claims huge improvements over base 32bits install, I would like to see how that translates into real-world.
Not sure about availability of 64bits editions of other browsers on Windows.
Here are my wishes:
-clearly mention if the 32bits or 64bits version of the browser is used
-where applicable and relevant, test with both 32bits and 64bits variants. I would like to see IE and FF split into 32 and 64 variants on Win for example.
I personally migrated from FF to WF on my machines 3 weeks ago and find it noticeably faster in everyday use. WF is now my main browser.
IE9 64 bit performs very bad in comparison to the 32 bit builds.
For firefox/waterfox, on Windows, using 64 bit builds has the following
1. Native performance increase due to 64 bit.
2. Performance degradation due to the fact that the MSVC does not have the same memory optimizations for 64 bit as for 32 bit.
so overall the experience of 64 bit FF/WF is the same as 32 bit builds.
For 64 bit Ubuntu, you get the 64 bit FF by default..
For a really great optimised FF, use PALEMOON.
@AdamOvera : 32/64 bit should be clearly mentioned in the article.
One test that is missing in almost all browser test is memory leak over time I find almost all browsers require more and more memory the longer they runExample: Chrome 17: 8 new tabs =1500MB
Memory benchmarks are almost useless in WBGP. Browsers leak, and Firefox leaks ALOT. But that is not the only problem. Opera works ok even when it fills up RAM, but Firefox becomes close to useless when it gets RAM deprived.
Nothing works well when ram is full.
And bashing Opera for doing things different is a shame
in FF, when it begans to use 1GB+ memory, it becomes sluggish.
So it is eating RAM AND becoming slow. I dont mind it eating RAM but it has to be responsive then.
IMO, Firefox is concentrating more on HTML5, ignoring CSS and JavaScript.It does well in HTML5 benches but 99% of the websites use primarily CSS and JS and HTML3, in which Firefox does poorly.
indeed in your opinion.
Maze solver is only one test among hundreds of things CSS does. If you want to argue about what 99% of websites use. then remember 99.9999999999% of websites don't use maze solver. For 99% of the websites, I would argue firefox does excellent job on CSS.
Firefox has focused on js speed for years with dedicated team, and with current benchmark (overall 2nd), you still claim it perform "poorly"? Its hard to argue you don't have prejudice here.
99 % of websites use HTML3? please, you argument is like mixture of 1980s and 2020s, whatever way you can put down firefox.
If you dislike firefox, state it, no need to hide behind the fake data.
in FF, when it begans to use 1GB+ memory, it becomes sluggish.So it is eating RAM AND becoming slow. I dont mind it eating RAM but it has to be responsive then.
Yes, exactly! It's as if something brakes in Firefox when it gets over 1 GB and Opera is mostly unaffected. I have seen many reports of this, and across all FF versions. That was the main reason I was using Opera for a long time, but I can't stand some of the things they've been doing since late 10 versions. So I'm stuck with terrible Firefox performance, but at least it's customizable. It is its only saving grace.
These kind of tests and comparisons are mostly useless. Only using the browser over a couple of days with 100+ tabs is what really shows its strengths and weaknesses, usability, performance, reliability...