How We Test Smartphones And Tablets

Web Browsing Benchmarks

While it’s desirable to measure a device’s performance for a common task like Web browsing, collecting good data is difficult because of the browser software layer. In many cases, the choice of Web browser has a larger effect on performance than hardware or even CPU scaling frequencies. Since the browser is so important, which one should we use?

There are several options available for Android, but the two obvious choices are Google’s Chrome browser and the stock Android browser. As the most commonly used browsers, their performance is the most representative of what a user would experience using a particular device. Unfortunately, neither is well suited for benchmarking. Chrome’s frequent updates makes comparing scores from device-to-device or even week-to-week difficult. The stock browser is even worse, because every OEM makes its own modifications (including some benchmark cheating), making device-to-device comparisons impossible. To avoid these issues, we use a static version of the Chromium-based Opera browser. The advantage to this approach is consistency and the ability to compare hardware performance. The downside is that using an out-of-date, less-popular browser produces scores that are not representative of what a user would actually see.

Due to platform restrictions, Safari is the only choice for iOS-based devices, while Internet Explorer is the only game in town on Windows RT. This makes it impossible to compare hardware performance across different platforms. This also is not fair to Android devices since iOS and Windows devices get to use newer, and likely higher performing, browsers.

When running browser benchmarks, no additional browser tabs or pages are loaded. After running each benchmark, we close the page and force close the browser to make sure caches are cleared between runs.

Now that you have a better understanding of the complexities of browser-based benchmarks and the limitations of our method, let’s discuss the benchmarks we use. Browser benchmarks primarily test JavaScript performance which is CPU-dependent, usually scaling directly with CPU clock frequency.

Browsermark

Browsermark by Basemark Ltd. tests a number of common tasks, including CSS 2D/3D transforms and resizes, DOM (Document Object Model) searches and creation, HTML5 graphics (Canvas, WebGL, SVG), and a number of different JavaScript functions.

JSBench

JSBench was originally created at Purdue University. Unlike most JavaScript performance benchmarks, it could almost be considered real-world, since it utilizes actual snippets of JavaScript from Amazon, Google, Facebook, Twitter, and Yahoo.

Google Octane

Google Octane is a synthetic benchmark that tests JavaScript performance. Octane is interesting because it really pushes the JavaScript engine, testing many different aspects missed by other browser benchmarks. It tests everything from regular expressions and floating-point math to compiler and garbage collection latency. For a full listing of the included tests, see Google’s reference page.

Because some of the Octane test scores show a wider variance than other benchmarks, we run it three times and average the results (throwing away any outlier runs) instead of the usual two runs.

Peacekeeper

This benchmark by Futuremark also tests JavaScript performance, including text parsing, array processing, DOM operations, HTML5 Canvas graphics, and even a 2D HTML5 game.