Yesterday I published a pretty entertaining story about how you can misrepresent characteristics of a product by skewing benchmarks (read the details here.) the article was based on a rather questionable browser benchmark procedure published by the W3C that aimed to indicate the HTML5 compliance of web browsers. The problem was that browser that cannot be compared and that there was, in particular, an IE version that was not available at the time of the test run.
The unanswered questions were: How could the W3C had access to the IE9 PP6 so early and who ran the test?
Late yesterday I got a call from Ian Jacobs, spokesman for the W3C, who apparently has been in France and did not have a chance to get back to me earlier. Jacobs conceded that the publication of the benchmark results has been unfortunate. "It was careless management on our part," he said. He noted that the W3C has been working on the HTML5 test and somehow was told that all browser vendors were in favor of releasing the results to the public, so the results were made public.
However, Jacobs said that a note that the test is work in progress (which was added one day later) should have been made right away and was simply not done. An oversight, if you will. I mentioned yesterday that the most likely explanation of the situation might be human error. On the W3Cs side, that was apparently the case.
Jacobs had, at the time, no knowledge who provided the test results for IE9 PP6, but mentioned that Microsoft, as well as some other parties in fact provided the HTML5 test suite and it appears to me that Microsoft is playing a major role in getting that test suite up and running. By itself, that isn't bad, as the W3C relies on such contributions, which can be made in the form of participation such as chairing an effort, editing standards or donating test. In Microsoft's case, the donations go a bit further, I might add, and include servers as well.
There is a good reason why Microsoft is making such big waves at the W3C and there is a good reason why Microsoft is stating that it wants to do more for the W3C. For years, the company has been bashed for ignoring web standards and cooking up whatever the company felt was best for the web community. IE9 is the very first browser that takes web standards very seriously and is the most web standard compliant web browser yet. Jacobs agreed that, of course, Microsoft wants to be heard and wants to make sure that we all know what work has been done. Microsoft wants us to know that IE9 is different than all the other browsers before. While all previous IE versions were taken down to the mat for their lack of compliance, it appears that Microsoft is now in need of some TLC.
As far as the tested browser version is concerned, Jacobs said that browser developers occasionally provide such non-public versions to show that some technology has been implemented. That makes sense and could have been an explanation, but Jacobs himself was unsure about how IE9 PP6 could have been tested before it was available. I mentioned that I felt it was just too much of a coincidence that a version would be tested early, that this version is an incomplete browser and that someone obviously knew that this version has more HTML5 support than the IE9 Beta. Perhaps he shared my concern.
Early this morning, I received an email from Jacobs confirming that it was Microsoft that submitted the IE9 PP6 test result. Yes, you could now create a conspiracy theory, which however, does not work out. Mozilla submitted its own result as well and, if you see it that way, screwed itself by being honest and using the most recent public version of its Beta browser. Opera, Google and Apple apparently did not care independent contributors submitted the results. Microsoft's participation is limited to guiding the listing of the benchmark results.
You could always make an allegation against Microsoft, especially because of the high visibility of the company in this case. Microsoft had the opportunity to push its IE9 ahead and it did. The Microsoft representative chose the browser because he knew of the improved HTML5 support and probably assumed that it was a fair game to use the newest version. The fact that the browser was tested before it was publicly available and you could assume some malicious intent is much more likely the result of a bad judgment. But then, of course, every developer had the same opportunity and the simply ignored it, with the exception of Mozilla.
What remains is the problem that benchmark results were published that should not have been published. I am filing the fact that results were provided by companies and not independent contributors under human error and lack of resources. There is no change that the results are just not valid. Today, the organization said that it would actually appreciate help with its HTML5 test and that the test could not be completed if the community does not contribute its knowledge and resources.
Microsoft's effort isn't seen in a very positive light in all of this, perhaps without good reason. However, Microsoft is walking on thin ice and can put the credibility of its messages on the line at any given moment. IE9 is the best browser the company had in a very long time. there is no need to push certain information, but the company needs to understand that it is under scrutiny and any information it posts needs to be backed up.
I'd wish Microsoft took a step back and left IE9 alone. In the end, users will discover that it is a much better browser than IE9.
