IE9 Beta: Better Is Not Enough

I hate to crash Microsoft IE9 Beta party early, but it appears that the browser will miss its mark.

There is no doubt that IE9 will be a much better browser than IE8 and Microsoft is absolutely heading into the right direction, but IE9 does not deliver what I hoped it would. The performance we have seen so far is mediocre at best and the screenshots that were leaked make IE9 look like a half-baked product. Is this the best Microsoft can do or will the company surprise us on September 15 with a truly innovative browser?

Last week we finally were informed that Microsoft will be releasing its new IE9 Beta during a fancy September party in San Francisco, at which the company will pitch the IE9 story to a select group of bloggers, developers and journalists. It somewhat reminded me at the good old Microsoft times back in the mid-90s, when the company discovered just how appealing beta software is to users – Microsoft gave out 1000 free beta copies of Windows 95 through a prize drawing at the magazine I worked back then and threw a huge bash when the IE4 Beta was launched. If you were not around back then, IE4 was the browser Microsoft designed to bury Netscape and it was, conceivably, Microsoft’s big bet on proprietary web technologies as HTML 4 was introduced.

13 years later, we are looking at version 9 that is Microsoft’s first HTML 5 capable web browser. Its step up from IE8 is about as significant as the step from IE3 to IE4 and Microsoft is yet again in a scenario of a browser war. This time, however, it needs to defend its market leadership against the mighty Google, which is taking aim at Microsoft in most of its core markets. Also, today’s browser war is much more competitive than what we have seen in the mid- to late-90s – there is Mozilla, Apple and we should not forget Opera. Microsoft’s browsers have been bleeding market share for several years and have lost more than 35 points of share so far – from about 95% to about 60%. Microsoft recently gained some points back, but we should be realistic and see that this trend may be short-lived as it is due to an advertising campaign Microsoft has launched across multiple TV channels. As soon as the campaign stops, Microsoft’s share is likely to drop again.

But what about IE9? What if this next browser is as great as IE4 was? The enthusiasm is certainly there. Microsoft is bold again and has some great technologies in this new browser, a new JavaScript engine and perhaps a few more surprises we do not know about yet. If we go by Microsoft’s IE blog, then the browser has taken over the world already, almost. According to Microsoft’s benchmarks, the latest IE9 PP4 is already faster than Firefox (3.6.x and Firefox 4 Beta) and Safari 5. The chart indicates that the browser is almost as fast as Chrome 5, Chrome 6 and Opera. To me, such charts are always an invitation to run the benchmarks myself to check if those numbers are accurate. The claims are rather bold and I was wondering whether IE9 has made such huge progress or whether Microsoft’s IE9 team is pulling an Apple here and tries to get away with something that may not be exactly true. And, most importantly, we should get a good idea how good IE9 Beta may be when it is released next month. Let’s have a look.

Benchmarks: V8, Sunspider

Benchmarking browsers is always a difficult exercise. Browser performance depends on many variables we can’t stabilize and there are different benchmarks that play into different strengths. For example, V8 is a browser that caters to Google’s Chrome browser, Microsoft’s HTML5 tests let IE9 shine and Mozilla’s Dromaeo suite typically favors Firefox strengths. However, Webkit’s Sunspider is cited by all browser manufacturers as the most important performance indicator (JavaScript) these days and I personally like to run new browsers through Google’s V8 as it highlights performance changes between different browser versions. However, comparing different browsers to Chrome in V8 is a bit unfair.

Also, please note that the results are highly dependent on computer hardware. I ran the tests using a fairly antiquated Q6600 Intel quad-core CPU, 8 GB of memory, and an ATI HD 5570 graphics card.

Here is the result for Google’s V8 benchmark:

The Webkit browsers Chrome and Safari perform well here, but it is worth noting just how much Chrome is different from Safari and how much fine tuning has gone into Chrome. Also note Opera’s excellent result. As for IE9, Microsoft promised fine tuning or its JavaScript engine when it released IE9 PP1 and it certainly delivered as the browser nearly doubled its performance in V8. Notice that the latest IE9 PP4 is about 12x faster here than IE8. The same can be said about Firefox, however. The TraceMonkey (TM) JavaScript engine is significantly faster in Firefox 4.0 Beta 3 than it is in the current stable 3.6.8. The differences between Firefox and IE9 are almost negligible at this point, which is somewhat interesting as Microsoft pretty much forgets about Firefox in its benchmarks and blog posts already and has been focusing on Chrome instead.

I would expect IE9 Beta to gain more performance over PP4, but then we know that Mozilla is already preparing the launch of its new JaegerMonkey (JM) JavaScript engine as well, most likely with Firefox 4 Beta 5, which is due in early September. We have no exact performance numbers for JM at this point, but we hear that JM is about 15% faster than TM right now and Mozilla expects further gains. If we believe the developer team, JM’s V8 performance should be at about 1300 points, which puts it ahead of IE9 PP4.

Let’s look at Sunspider:

I have left out IE8 and its result of 4989.4 ms. The sluggish performance is widely know and somewhat irrelevant in this chart as it distorts the differences between the other browsers. What is truly remarkable is that Chrome 6 is close to be breaking through the 300 ms barrier. Chrome 5 was, by the way, the first browser that broke through 400 ms just about five months ago. The Chrome nightly builds (NB) are by far the fastest JavaScript browsers available right now, hands down. The first IE9 PP was faster than Firefox and IE9 has kept that lead until today. It is now substantially faster, but the distance to Firefox is about the same as it was in March of this year. The interesting part, however, is Mozilla’s current JM performance claim of 525 ms. Mozilla is shaving off milliseconds now virtually every day and we know that Mozilla wants to end up within 20% of Chrome 5 performance, which is somewhere around 350 ms these days. If Mozilla achieves its goal, it will end up just north of 400 ms and possibly in the range or slightly ahead of IE9 Beta. 

Microsoft has to be careful that it does not have to eat its own words and may see a déjà vu with IE9: Back in March 2008, it expected to be able to surpass Firefox 3’s performance only to see Mozilla pull ahead with its TM engine. Clearly, Firefox is not a done deal for IE9 yet. IE9 Beta has to deliver a stunning upgrade if it wants to beat Firefox’ JM engine.

So, what about HTML 5?

If you have been reading recent benchmark stories that were pitched by Microsoft, then you know that IE9 is the best HTML 5 browser around, thanks to its added HTML 5 support and  hardware accelerated rendering engine. You can actually check Microsoft’s graphics performance page and it is clear that IE9 is much better in these benchmarks than any other browser right now. Sorta … maybe ... maybe not …. ummm, no.

While Microsoft hardware (GPU) acceleration of certain content is truly stunning as multithreading web content in current browsers has not been possible so far, I have two main issues with Microsoft’s claims. First, Firefox is, on average, at least as good and occasionally better than IE9 in those benchmarks. Microsoft just did not activate the still developed hardware acceleration that has been available in Firefox since version 3.7a5. If you activate the feature, Firefox 4.0 breezes through most of Microsoft’s benchmarks just as fast as IE9 PP4.

Of course, Chrome, Safari and Opera do not support hardware acceleration yet. But I am just not sure how important this feature is right now. Seriously, how many HTML 5 web sites do you know that benefit from hardware acceleration? Sure, such websites will come, but we know that Google is working on hardware acceleration as well, Opera will have it in the not too distant future and I am sure Apple will have it as well, too, especially since Steve Jobs is so focused on killing Flash. The fact, however, is: This big IE9 advantage isn’t such a big advantage yet.

As for HTML 5 support, IE9 has still substantial gaps. PP4, for example, does not support the critical canvas element yet. According to Caniuse.com, IE9 only supports 81% of all HTML 5 requirements, while Chrome is at 88%, Firefox at 96%, Safari at 88% and Opera at 77%. In the html5test.com test run, IE9 scores 85 of 300 points. Chrome 6 Beta hits 227 points, Opera 10.61 166 points, Safari 5.0.1 214 points and Firefox 4 Beta 4-pre 199 points. IE9 may not be the best HTML 5 browser around – and the scenario is somewhat reminiscent of the introduction of IE8, when Microsoft said it would fully support web standards. Of course, we know that this was not exactly the case, as shown, for example, by the Acid 3 test: IE8 scored just 21/100 points. IE9 scores 83/100 in Acid 3, while Firefox is at 97/100 and all other major browsers at 100/100.

Conclusion

There is no doubt that IE9 will be a solid upgrade for loyal IE8 users, but not for Firefox or Chrome users. IE9 seems to be more nimble, but it isn’t quite as fast as it needs to be if it wants to play with the big boys. If the leaked screenshots in fact show the look and feel of the new IE9, then it will look old right out of the gate. Opera, Chrome, Safari and Firefox offer sleek new interfaces that represent a changed user behavior, while IE9 may look just like IE8 or IE7. It is a bit too early to draw a final conclusion already now, but there is a good chance that IE9 will not deliver on the expectation it has created. Any JavaScript performance below Firefox and the lack of HTML 5 support will leave Microsoft with a graphics hardware acceleration feature that is nice to haven but rather useless today – like a 600 hp super car that is mainly driven in commuter traffic.

Better isn’t good enough for IE9.