Sign in with
Sign up | Sign in

Google Now Indexing AJAX Content

By - Source: Google Webmaster Blog | B 10 comments
Tags :

Google has just announced that it is now able to crawl content loaded by asynchronous requests in many situations.

Love it or hate it, JavaScript is rapidly being adopted by developers to improve the usability of the websites that they build. Many high profile websites are using it to perform acynchronous requests after the page has loaded. Google realised that simply reading static content is no longer adequate for providing the most relevant search results for users.

Google has added a post on the Webmaster Central Blog explaining the new functionality and how to take advantage of it. In order to generate an accurate instant previews in search results and to correctly index all content on the page, Googlebot will execute any JavaScript code blocks which send an XMLHttpRequest. While previous workarounds (such as the hashbang hack used by Twitter and Facebook) worked with carefully-crafted GET requests to provide a non-AJAX snapshot of a page to search engines, Googlebot is now able to perform both GET and POST AJAX requests itself. According to the blog post, only automatic requests are currently being executed and not those which are performed as a result of user action.

Just before the announcement, Digital Inspiration noticed that Facebook comments on TechCrunch were being indexed. Prior to Googlebot's new functionality, searching Google for the content of these comments wouldn't have returned the article that they were attached to. These changes should also allow other third party commenting platforms such as Disqus to be properly indexed as well.

Note that restricting search engines' access to any of the files needed to perform the request, such as an external JavaScript file or the file that is actually being opened by the request, will break Googlebot's ability to index the content. Developers and website owners will need to ensure that their robots.txt is configured correctly and they haven't left a stray robots meta tag in any required files that is blocking search engines.

Discuss
Ask a Category Expert

Create a new thread in the News comments forum about this subject

Example: Notebook, Android, SSD hard drive

This thread is closed for comments
  • -7 Hide
    Anonymous , November 2, 2011 11:07 PM
    LOL, you announce it like Javascript is some brand new thing. Those who hate it will continue to use noscript.

    However, it's downright shocking that Google isn't already doing this, they index text found in flash, and they index the color content of images FFS.
  • 7 Hide
    festerovic , November 2, 2011 11:15 PM
    One time I woke up and found Googlebot in bed with me.
  • 4 Hide
    alidan , November 2, 2011 11:32 PM
    funny how i turn off java script and flash from every website and only allow whats nessassary for functionality.

    the sad thing is, with this, they could force me to use javascript on more websights to get basic functions, as google now makes it searchable.
  • Display all 10 comments.
  • -5 Hide
    CKKwan , November 2, 2011 11:50 PM
    My 5 year old daughter also know that never execute any code / script that you do not know.

    I construct a web page with a fake GET / POST request, trick Googlebot to execute my script, and the script is actually tring to Hack or Attack another web site.

    When I get caught, whose fault is that? I have some malicious code, but I have never executed it.
  • 5 Hide
    geossj5 , November 3, 2011 1:26 AM
    CKKwanMy 5 year old daughter also know that never execute any code / script that you do not know.I construct a web page with a fake GET / POST request, trick Googlebot to execute my script, and the script is actually tring to Hack or Attack another web site.When I get caught, whose fault is that? I have some malicious code, but I have never executed it.


    uhh.. I dont think you understand how front-end scripting (aka javascript) works
  • 1 Hide
    TunaSoda , November 3, 2011 2:23 AM
    This is good news
  • 0 Hide
    soundping , November 3, 2011 6:27 AM
    Something else for the man. lol
  • 1 Hide
    nikorr , November 3, 2011 8:23 AM
    Google is cool.
  • 0 Hide
    claydavis , November 3, 2011 12:11 PM
    Really great to hear this news because as a developer I had two thoughts whether to use or not Ajax in my website. But it is a relief to me now and I am going to try how it works.
  • 3 Hide
    gm0n3y , November 3, 2011 5:37 PM
    geossj5uhh.. I dont think you understand how front-end scripting (aka javascript) works

    While what he's saying isn't really valid in a legal sense, it would be possible to get Google to run malicious javascript code since the whole point of AJAX is that it allows js to execute server-side scripts. AJAX blurs the lines between client(front) and server(back) scripting. This is a huge part of web development now. I just really wish that Javascript didn't suck so badly.