RoboHornet Pro: Microsoft Snubs Google, Mozilla Concurs!

RoboHornet Redux: Alpha To Pro In 24 Hours

Holy train wreck, Batman! The end of September was one weird, wild week for Web browser news.

Despite RoboHornet being an independent GitHub project on paper, Microsoft and others aren't buying it. In possibly the fastest turnaround in development history, Microsoft has taken RoboHornet from first alpha release to "Pro" in less than 24 hours. In another surprising turn, Mozilla formally concurs with Redmond that RoboHornet's technical merits are questionable. Perhaps arch-rivals Microsoft and Mozilla finally realize that Google is their real enemy. After all, while the browser war veterans were busy battling each other, Chrome just waltzed through the front gates and usurped the throne.

Let's go through the wacky events of last week as they occurred:

Monday: We broke the RoboHornet story. The new Web browser performance test is actually a suite of micro-benchmarks in the areas of HTML, CSS, JavaScript, DOM, and SVG. In our testing, Internet Explorer 10 for Windows 8 claimed a decisive victory, more than doubling the average Web browser performance on the test creator's reference MacBook Pro (late 2011).

Tuesday: Microsoft says "Thanks, but no thanks" to RoboHornet, dismissing the new test as a meaningless micro-benchmark that does not reflect real-world performance. Below is an excerpt from the IE Blog, and from the language used in the opening sentence, you can tell that Microsoft isn't buying the "independent benchmark" part:

Yesterday Google released its latest micro-benchmark, RoboHornet, in which Internet Explorer 10 scores rather well. While we appreciate the gesture, members of our engineering team took a look at the benchmark and found that RoboHornet isn’t all that representative of the performance users might encounter on real-world sites. Like all micro-benchmarks, RoboHornet is a lab test that only focuses on specific aspects of browser performance.We decided to take the RoboHornet micro-benchmark and run it in the context of a real-world scenario. Using modern browser capabilities like CSS3 Animations, CSS3 Transforms, CSS3 Text Shadows, custom WOFF fonts, Unicode, Touch, and more, we created a site that looks a little bit like the familiar Matrix. We then ran the RoboHornet micro-benchmark in the context of this real website. While running both the Matrix and RoboHornet micro-benchmark at the same time, Chrome slows to a crawl and stops animating the screen, because it wasn’t designed to handle a benchmark load in the context of a real-world scenario. Meanwhile Internet Explorer 10 remains responsive, continues animating the screen, and finishes the test in less than half the time that Chrome does… We have made RoboHornet Pro available on IE Test Drive, so you can check it out for yourself.

Later that day Mozilla's Justin Lebar opened a bug on RoboHornet's GitHub page entitled Eliminate and outlaw micro-benchmarks where he states:

If you guys want us (in my case, Mozilla) to take robohornet seriously, I strongly recommend you write some macrobenchmarks and eliminate the microbenchmarks from your test suite.

Wednesday: Microsoft-affiliated steward, John David Dalton, drops all mention of Microsoft from his RoboHornet stewardship. Meanwhile, the Mozilla steward, Daniel Buchner, leaves the committee entirely.

Today: We have Mozilla's official response to the RoboHornet debacle:

There are a lot of benchmarks out there and different benchmarks for the same particular task can behave very differently. Many benchmarks are self-serving, in that the creators will typically pick out a set of programs that they think are worthwhile to get faster on and then only after turning these workloads share them with the wider community. However, what developers and browser implementers really need to have here is good benchmarks that allow us to better see holistic performance. We'd like to see more benchmarks created that focus on the entire consumer experience--for example, benchmarks that focus on interaction with the browser, popular web applications or sites, and common tasks like panning and zooming on mobile. Micro-benchmarks, like RoboHornet, do not accurately reflect the user experience on the Web. RoboHornet aims to measure real performance, but it falls short. RoboHornet is currently 17 micro-benchmarks, each of which measures one thing that a website can do. But real websites do hundreds and thousands of things, so almost all of them are entirely unaccounted for by RoboHornet. RoboHornet lists some things that are currently slow in browsers and points them out for attention from vendors. That's a useful service, but it's not the same as accurately representing real performance.

It is still unclear what exactly Mozilla thinks about Microsoft's RoboHornet Pro, but now that results are published, we'll ask for a statement and update the story accordingly.

Of the remaining top five browser vendors, Opera refuses to comment, and Apple cannot be reached.

RoboHornet Redux

RoboHornet Pro is already off to a better start than RoboHornet because it works with so many more browsers than the original test. We are now able to include results from the Windows 8 Metro versions of Chrome and IE10, as well as Maxthon and Sleipnir on both Windows and OS X. iOS browsers can now run the test as well. We've included Yahoo! Axis, Google Chrome, Dolphin, Mozilla Firefox, Maxthon, Apple Safari, and Sleipnir. While Android loses its stock browser, it retains Chrome and gains Opera Mobile.

Let's quickly recap the test setup, and then see how all these browsers fare on RoboHornet Pro.

  • techcurious
    A little off topic, but something just occurred to me.. that we had Safari on Windows, but not IE on OSX..
    Now that OSX market share is increasing, I wonder if Microsoft will ever release Internet Explorer for OSX, and I wonder if Apple will allow it..
    Reply
  • techcurious
    ... nevermind.. I would rate myself down if I could.. cause after further thought, I realized that anyone that chose to use OSX over Windows will never choose to use IE over Safari or Firefox.. Something I am sure Microsoft also realizes :)
    Reply
  • dalethepcman
    My one complaint about this article, was the contrast between the massive amount of iOS browsers, and the utter lack of Android browsers. You stated yourself.
    Each of these third-party iOS browsers are essentially just a new GUI and additional functionality added to Safari

    Why not also test firefox, dolphin, skyfire and stock browsers on android?
    Reply
  • dalethepcman
    techcurious... nevermind.. I would rate myself down if I could.. cause after further thought, I realized that anyone that chose to use OSX over Windows will never choose to use IE over Safari or Firefox.. Something I am sure Microsoft also realizes
    Actually I have many Mac users that would use IE in OSX just for the convenience of pass through authentication, but since its not available they all have a separately purchased copy of windows to run in a VM or access IE through Citrix for domain resources.
    Reply
  • adamovera
    dalethepcmanMy one complaint about this article, was the contrast between the massive amount of iOS browsers, and the utter lack of Android browsers. You stated yourself.Why not also test firefox, dolphin, skyfire and stock browsers on android?I attempted to, they either wouldn't run the test at all, or they hang indefinitely and are unable to complete it. Stay tuned for the Android Web Browser Grand Prix for the full benchmark results of browsers on that platform.
    Reply
  • tipoo
    I agree with Mozilla, perceived speed > benchmarks for browsers. And ironically that's exactly where they fail hardest. Chrome, Opera, heck sometimes even IE10 now always seem more responsive and stay more responsive than Firefox in my experience. I like its font rendering, I like its smooth scrolling (well, IE10 has those too, I think it has to do with DirectWrite more than the browser) but the small instances of UI lag bug me after using Chrome for so long.
    Reply
  • tipoo
    Once more I'm puzzled why browsers in OSX are consistently and significantly slower than Windows and Ubuntu, even the same browser cross platform.
    Reply
  • adamovera
    tipooOnce more I'm puzzled why browsers in OSX are consistently and significantly slower than Windows and Ubuntu, even the same browser cross platform.From page 4: "Our current cross-platform test system provides unusually low Web results under OS X Mountain Lion compared to other operating systems. While the OS X browser scores appear to be accurate in relation to each other, none of the OS X scores should be used to draw conclusions about OS X versus the other desktop environments in this test. Until we can pin down the culprit, please view the OS X results as if they were obtained on an entirely different test system."
    I have been unable to track down the cause of this problem - I tried every single network driver I could find, multiple re-installs, and different DSDT files. This is the only Hackintosh system we've ever used that has this issue - our older Lynnfield-based rig didn't. Hopefully, when I build a totally new Ivy Bridge-based rig this problem will just go away, if not, I guess I need an actual Mac - but that could leave Linux twisting in the wind since Bootcamp is just for Windows - on paper, anyway :(
    Reply
  • tipoo
    Got ya, I missed that part. Maybe a commenter with a mac can give it a quick run to see how it compares to the hackintosh.
    Reply
  • adamovera
    tipooGot ya, I missed that part. Maybe a commenter with a mac can give it a quick run to see how it compares to the hackintosh.They have, and they're all reporting Web-related scores are higher on Mac's with lower-end hardware than our test system. Also, our old Lynnfield-based system shows OS X browsers doing way better in relation to Windows browsers on that Hackintosh versus a genuine MacBook Air - so it's definitely our current Hackintosh configuration and not OS X to blame for the lower scores versus other OSes.
    Reply