Google Now Indexing AJAX Content
Google has just announced that it is now able to crawl content loaded by asynchronous requests in many situations.
Love it or hate it, JavaScript is rapidly being adopted by developers to improve the usability of the websites that they build. Many high profile websites are using it to perform acynchronous requests after the page has loaded. Google realised that simply reading static content is no longer adequate for providing the most relevant search results for users.
Google has added a post on the Webmaster Central Blog explaining the new functionality and how to take advantage of it. In order to generate an accurate instant previews in search results and to correctly index all content on the page, Googlebot will execute any JavaScript code blocks which send an XMLHttpRequest. While previous workarounds (such as the hashbang hack used by Twitter and Facebook) worked with carefully-crafted GET requests to provide a non-AJAX snapshot of a page to search engines, Googlebot is now able to perform both GET and POST AJAX requests itself. According to the blog post, only automatic requests are currently being executed and not those which are performed as a result of user action.
Just before the announcement, Digital Inspiration noticed that Facebook comments on TechCrunch were being indexed. Prior to Googlebot's new functionality, searching Google for the content of these comments wouldn't have returned the article that they were attached to. These changes should also allow other third party commenting platforms such as Disqus to be properly indexed as well.
Note that restricting search engines' access to any of the files needed to perform the request, such as an external JavaScript file or the file that is actually being opened by the request, will break Googlebot's ability to index the content. Developers and website owners will need to ensure that their robots.txt is configured correctly and they haven't left a stray robots meta tag in any required files that is blocking search engines.
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
-
LOL, you announce it like Javascript is some brand new thing. Those who hate it will continue to use noscript.Reply
However, it's downright shocking that Google isn't already doing this, they index text found in flash, and they index the color content of images FFS. -
alidan funny how i turn off java script and flash from every website and only allow whats nessassary for functionality.Reply
the sad thing is, with this, they could force me to use javascript on more websights to get basic functions, as google now makes it searchable. -
CKKwan My 5 year old daughter also know that never execute any code / script that you do not know.Reply
I construct a web page with a fake GET / POST request, trick Googlebot to execute my script, and the script is actually tring to Hack or Attack another web site.
When I get caught, whose fault is that? I have some malicious code, but I have never executed it. -
geossj5 CKKwanMy 5 year old daughter also know that never execute any code / script that you do not know.I construct a web page with a fake GET / POST request, trick Googlebot to execute my script, and the script is actually tring to Hack or Attack another web site.When I get caught, whose fault is that? I have some malicious code, but I have never executed it.Reply
uhh.. I dont think you understand how front-end scripting (aka javascript) works -
claydavis Really great to hear this news because as a developer I had two thoughts whether to use or not Ajax in my website. But it is a relief to me now and I am going to try how it works.Reply -
gm0n3y geossj5uhh.. I dont think you understand how front-end scripting (aka javascript) worksWhile what he's saying isn't really valid in a legal sense, it would be possible to get Google to run malicious javascript code since the whole point of AJAX is that it allows js to execute server-side scripts. AJAX blurs the lines between client(front) and server(back) scripting. This is a huge part of web development now. I just really wish that Javascript didn't suck so badly.Reply