Introduction
Alright folks. Today we have a special treat for you. The last two Web Browser Grand Prix stories received tons of reader-based feedback. But the comments in the first article resounded one question more loudly than the others: what about browser performance in Linux?
Ah, that's a good one.
Hot on the heels of Web Browser Grand Prix 2, we bring you The Linux Circuit. Using the exact same test system that we employed last time, we installed Ubuntu 10.04 LTS (Lucid Lynx) and the three major cross-platform Web browsers: Google Chrome, Mozilla Firefox, and Opera. Not only will be be declaring a winning Linux-based browser, but we'll also be comparing the results to those generated under Windows 7. Because the top three performers in Windows 7 happen to be Opera, Chrome, and Firefox, we can declare a winner that spans both platforms as well.
Dromaeo Drama
Over the past two articles we've had a little bit of controversy concerning Mozilla's Dromaeo JavaScript Benchmark. In the first Web Browser Grand Prix, Opera 10.50 scored far better than the other four Web browsers. Since it didn't affect Chrome's win either way, I decided to let Opera's highly dubious score persist. Chrome fans didn't seem to care, but Opera fans definitely let me know that the score is just swell, and I shouldn't be questioning it. In last week's sequel, Opera 10.60 again received an incredibly high score in the Dromaeo JavaScript test. That time, however, I disqualified Opera from the benchmark. Despite Opera 10.60 still handily taking the speed crown from Google Chrome, Opera fans really let me have it because of Chrome's slight edge in overall performance (including non-speed-related tests).
Now, just to set the record straight, the decision to remove Opera from the placing in the Dromaeo JavaScript benchmark did not affect the final outcome of the overall winner. If I had not disqualified Opera, making it the first-place finisher for that test, Chrome would still have been the overall performance champion. Though Chrome's win would have been by the absolute smallest possible margin, a quantifiable lead would have existed nonetheless. I don't like making these calls. I'd much prefer for everything to go smoothly. But stuff happens, and sometimes executive decisions need to be made.
Less than a week later, hindsight shows the decision to disqualify Opera as the right call. We have since contacted the three parties involved: Mozilla (author of the Dromaeo benchmarks), Opera, and Google. Below is an excerpt from a statement made by Mozilla's vice president of engineering, Mike Shaver:
Until recently, the Dromaeo benchmark, especially as regarded regular expression performance, was very susceptible to optimizations specifically targeted at those weaknesses. A JavaScript engine that was specifically altered to accommodate those tests could see astronomical score improvements by effectively subverting the work that is supposed to be done and timed.Google brought this bug to our attention and requested that we fix the test. We agreed to, since it was clearly not testing anything useful. That test fix was completed last week. (In the interim they added specific code to V8 to also "game" that aspect of the benchmark, for competitive reasons.
Well, there you have it. Whether or not Opera intentionally 'gamed' this benchmark is still up in the air. Without access to the source code, no one outside of Opera would know for sure. An Opera rep offered the initial idea that Profile-Guided Optimizations (PGO) may be to blame for the high Dromaeo scores seen in Windows. According to Opera, its OS X version (which doesn't yet have PGO enabled) does not achieve absurdly high scores in Dromaeo. The company had no results available for its Linux version, and the rep didn't know whether or not the penguin had PGO. Without any Linux data from Opera, we cannot further determine if PGO is truly the culprit. In the meantime, we are still waiting to hear back from the developers in Oslo for an official explanation for the high Dromaeo JavaScript scores.
As far as Google is concerned, thanks to Chromium, Mozilla has the smoking gun on the company's Dromaeo shenanigans. In the last article, I reported that Chrome 5.0.375.89 also yielded dramatically high Dromaeo JavaScript scores--much higher than the score that got Opera disqualified. Mozilla and Google both confirmed that the 5.0.375.70 and 5.0.375.86 scores are bunk. The final version of Chrome that was tested and published (5.0.375.99) scored 290.72. Google contends that a caching feature, added to 5.0.375.70 but then removed in 5.0.375.99, was to blame. The Mountain View company did nothing about the high Dromaeo scores back then, even though they were aware of it, because Opera hadn't either.
According to Mr. Shaver, the test fix was put into place shortly after our last piece was published. We have since re-run the Dromaeo JavaScript tests on both Opera 10.60 and Chrome 5.0.375.99 using our Windows 7 Ultimate 64-bit test system. Opera now gets a score of 211.5, while Chrome still earns around 290. Neither Mozilla nor Google have reason to doubt the Chrome 5.0.375.99 score. While the Opera disqualification from our previous piece stands, we'll update Opera's Windows 7 Dromeao JavaScript score to reflect the 'real' result in the charts for this article.
With this brouhaha behind us, and without further delay, let the benchmarking begin!
