Sign in with
Sign up | Sign in

W3C: A Case of Careless Management

By - Source: Tom's Hardware US | B 20 comments

Part 2 of the W3C browser benchmarks expose.

Yesterday I published a pretty entertaining story about how you can misrepresent characteristics of a product by skewing benchmarks (read the details here.) the article was based on a rather questionable browser benchmark procedure published by the W3C that aimed to indicate the HTML5 compliance of web browsers. The problem was that browser that cannot be compared and that there was, in particular, an IE version that was not available at the time of the test run.

The unanswered questions were: How could the W3C had access to the IE9 PP6 so early and who ran the test?  

Late yesterday I got a call from Ian Jacobs, spokesman for the W3C, who apparently has been in France and did not have a chance to get back to me earlier. Jacobs conceded that the publication of the benchmark results has been unfortunate. "It was careless management on our part," he said. He noted that the W3C has been working on the HTML5 test and somehow was told that all browser vendors were in favor of releasing the results to the public, so the results were made public.

However, Jacobs said that a note that the test is work in progress (which was added one day later) should have been made right away and was simply not done. An oversight, if you will. I mentioned yesterday that the most likely explanation of the situation might be human error. On the W3Cs side, that was apparently the case.

Jacobs had, at the time, no knowledge who provided the test results for IE9 PP6, but mentioned that Microsoft, as well as some other parties in fact provided the HTML5 test suite and it appears to me that Microsoft is playing a major role in getting that test suite up and running. By itself, that isn't bad, as the W3C relies on such contributions, which can be made in the form of participation such as chairing an effort, editing standards or donating test. In Microsoft's case, the donations go a bit further, I might add, and include servers as well.  

There is a good reason why Microsoft is making such big waves at the W3C and there is a good reason why Microsoft is stating that it wants to do more for the W3C. For years, the company has been bashed for ignoring web standards and cooking up whatever the company felt was best for the web community. IE9 is the very first browser that takes web standards very seriously and is the most web standard compliant web browser yet. Jacobs agreed that, of course, Microsoft wants to be heard and wants to make sure that we all know what work has been done. Microsoft wants us to know that IE9 is different than all the other browsers before. While all previous IE versions were taken down to the mat for their lack of compliance, it appears that Microsoft is now in need of some TLC.      

As far as the tested browser version is concerned, Jacobs said that browser developers occasionally provide such non-public versions to show that some technology has been implemented. That makes sense and could have been an explanation, but Jacobs himself was unsure about how IE9 PP6 could have been tested before it was available. I mentioned that I felt it was just too much of a coincidence that a version would be tested early, that this version is an incomplete browser and that someone obviously knew that this version has more HTML5 support than the IE9 Beta. Perhaps he shared my concern.

Early this morning, I received an email from Jacobs confirming that it was Microsoft that submitted the IE9 PP6 test result. Yes, you could now create a conspiracy theory, which however, does not work out. Mozilla submitted its own result as well and, if you see it that way, screwed itself by being honest and using the most recent public version of its Beta browser. Opera, Google and Apple apparently did not care independent contributors submitted the results. Microsoft's participation is limited to guiding the listing of the benchmark results.     

You could always make an allegation against Microsoft, especially because of the high visibility of the company in this case. Microsoft had the opportunity to push its IE9 ahead and it did. The Microsoft representative chose the browser because he knew of the improved HTML5 support and probably assumed that it was a fair game to use the newest version. The fact that the browser was tested before it was publicly available and you could assume some malicious intent is much more likely the result of a bad judgment. But then, of course, every developer had the same opportunity and the simply ignored it, with the exception of Mozilla.

What remains is the problem that benchmark results were published that should not have been published. I am filing the fact that results were provided by companies and not independent contributors under human error and lack of resources. There is no change that the results are just not valid. Today, the organization said that it would actually appreciate help with its HTML5 test and that the test could not be completed if the community does not contribute its knowledge and resources.

Microsoft's effort isn't seen in a very positive light in all of this, perhaps without good reason. However, Microsoft is walking on thin ice and can put the credibility of its messages on the line at any given moment. IE9 is the best browser the company had in a very long time. there is no need to push certain information, but the company needs to understand that it is under scrutiny and any information it posts needs to be backed up.      

I'd wish Microsoft took a step back and left IE9 alone. In the end, users will discover that it is a much better browser than IE9.

Display 20 Comments.
This thread is closed for comments
Top Comments
  • 20 Hide
    rpgplayer , November 5, 2010 12:18 AM
    I'd wish Microsoft took a step back and left IE9 alone. In the end, users will discover that it is a much better browser than IE9.

    think that last spot should read IE8
  • 11 Hide
    Bolbi , November 5, 2010 12:17 AM
    Hm, not to be rude, but is this a draft article? A couple of first words of sentences aren't capitalized, a couple of sentences just don't make sense, there are some incomplete/improper verb forms, etc.
    As far as the content of the article, I can certainly understand Microsoft's actions. They saw an opportunity to push IE9, and they did. Since the other browser vendors were given the same opportunity, I say it's all fair. Kudos to Microsoft for trying to make IE9 standards compliant (though I remain a Firefox user, for now).
    And finally, good job Wolfgang for digging up this story and reporting it so completely!
Other Comments
  • 11 Hide
    Bolbi , November 5, 2010 12:17 AM
    Hm, not to be rude, but is this a draft article? A couple of first words of sentences aren't capitalized, a couple of sentences just don't make sense, there are some incomplete/improper verb forms, etc.
    As far as the content of the article, I can certainly understand Microsoft's actions. They saw an opportunity to push IE9, and they did. Since the other browser vendors were given the same opportunity, I say it's all fair. Kudos to Microsoft for trying to make IE9 standards compliant (though I remain a Firefox user, for now).
    And finally, good job Wolfgang for digging up this story and reporting it so completely!
  • 20 Hide
    rpgplayer , November 5, 2010 12:18 AM
    I'd wish Microsoft took a step back and left IE9 alone. In the end, users will discover that it is a much better browser than IE9.

    think that last spot should read IE8
  • 4 Hide
    Anonymous , November 5, 2010 12:27 AM
    what does this mean Yoda? IE9 is better than IE9?

    "I'd wish Microsoft took a step back and left IE9 alone. In the end, users will discover that it is a much better browser than IE9."
  • 3 Hide
    juncture , November 5, 2010 12:35 AM
    "I'd wish Microsoft took a step back and left IE9 alone. In the end, users will discover that it is a much better browser than IE9."

    What? My mind might be playing tricks on me.

    Love the beginning to the end of your investigation :D 
  • 4 Hide
    znegval , November 5, 2010 12:37 AM
    BolbiKudos to Microsoft for trying to make IE9 standards compliant


    I'm glad they embraced it. Let's just hope they don't go ahead with the extend and extinguish later on.
  • 0 Hide
    rpgplayer , November 5, 2010 12:39 AM
    imho if the W3C were going to test alpha and beta releases, they should have also tested the standard release along side it. that way it wouldn't nullify any of their findings, and wouldn't call them into question.
  • 4 Hide
    Prince_Porter , November 5, 2010 12:49 AM
    Do a quick edit, there's a lot of typos in there. Glad to read part 2 of this, great article overall.
  • 3 Hide
    Pyroflea , November 5, 2010 2:00 AM
    Flameoutbottom line is microsoft don't care about the end user. that's why their products will always be inferior


    That's not at all true. Look at Windows 7 after the Vista fiasco. They took input from their users, offered over a year of open-beta testing. Obviously they care about what their customers want. That's part of business whether or not people want to believe it. There's no point pumping out a product if your potential customers aren't in mind during the development stage, because nobody will use it.
  • -6 Hide
    dEAne , November 5, 2010 5:30 AM
    See IE9 is a crap.
  • 1 Hide
    alyoshka , November 5, 2010 6:56 AM
    Yeah.... Love the Yoda comment though........I've got my eye rolling all over..:) 
  • -3 Hide
    Benihana , November 5, 2010 7:03 AM
    PyrofleaThat's not at all true. Look at Windows 7 after the Vista fiasco. They took input from their users, offered over a year of open-beta testing. Obviously they care about what their customers want. That's part of business whether or not people want to believe it. There's no point pumping out a product if your potential customers aren't in mind during the development stage, because nobody will use it.

    I wish they didn't take advice from that guy in the green shirt. While snapping windows is pretty sweet, it seems they dropped a lot of other essential features. Maybe the guy in the green shirt told them to drop them?

    http://www.youtube.com/watch?v=rmiPzMY4nuE
  • -6 Hide
    Silmarunya , November 5, 2010 8:24 AM
    Wait... IE9 the very first browser designed to be compliant with web standards?

    The first IE for sure, but Opera has been complying with web standards for years, it was even co-founder of a few, including CSS. Chrome, even though it doesn't exist as long as Opera, has also been complying with all web standards. Firefox isn't fully compliant, but does a very good job too. Hell, even Safari, made by a company that usually ignores standards, is more compliant than IE 8.

    And to make matters worse, IE 9 isn't even fully compliant. In other news, it's no faster than FF, not extensible and lacking in features.
  • 0 Hide
    back_by_demand , November 5, 2010 9:03 AM
    Said it before, say it again - you jumped the gun Wolfgang
    I do like the style of your editorials, large, free-flowing and full of information. Maybe the subject matter isn't exactly "All The President's Men" and you sure aren't Robert Redford, but you do tend to go very indepth on a subject where other people will post soundbites.

    But on this matter, you sank to their level when you posted yesterday's article, just because some guy didn't get back to you in 1 day and posted up an insinuation that W3C had sold out to MS?

    Gutter journalism Wolfgang, we had come to expect so much more from you. Here's hoping it returns to it's previous quality soon.
  • 1 Hide
    bhaberle , November 5, 2010 10:40 AM
    Really this whole "issue" is blown out of proportion. It was clear that all the products were in Beta so the tests should always have been taken with a grain of sand. It is still useful because there are lots of people who use the beta builds of each browser.

    Yes they had a newer build of IE that was not publicly released yet. But guess what? Certain companies that review computer hardware get stuff to review before it comes out too. And yes I am aware that the tests are anywhere near completion. This still should be be a big deal. No where did they say those were the "final" version of those browsers to begin with.
  • -3 Hide
    bunz_of_steel , November 5, 2010 10:41 AM
    IE suks no matter how you carve it up, stats or whatev. Not using to stinking noob browser. I use opera or FF or google thingy. IE SUKS!
  • 1 Hide
    xero9200 , November 5, 2010 12:35 PM
    Is this really a news article? It seems to have the format of a whole article, and like someone said it appears to be a draft? Just wondering...
  • 0 Hide
    hebe , November 5, 2010 6:35 PM
    I have to say, this is one of the only compelling "news" articles posted recently to this site. I have about had it with the advertisement "news" that gets posted by most of the other "reporters" on Tom's.
  • 0 Hide
    soundping , November 5, 2010 6:36 PM
    Thank you Wolfgang Gruener for following up. ;) 
  • 0 Hide
    deltaranger509 , November 6, 2010 3:14 AM
    So you screwed the pooch yesterday by posting a premature "expose" on an issue that was a pretty cut a dried case of a reporter not understanding how the process works and crying foul before any of substance was actually _investigated_ at all and you follow it up with a poorly edited article correcting your mistakes but offering no appology or retraction?

    Well, I'm done with the gutter reporting here. I guess I'll stick to Ars for the real story. Taking this site off my rss feed.