Google Now Indexing AJAX Content

Love it or hate it, JavaScript is rapidly being adopted by developers to improve the usability of the websites that they build. Many high profile websites are using it to perform acynchronous requests after the page has loaded. Google realised that simply reading static content is no longer adequate for providing the most relevant search results for users.

Google has added a post on the Webmaster Central Blog explaining the new functionality and how to take advantage of it. In order to generate an accurate instant previews in search results and to correctly index all content on the page, Googlebot will execute any JavaScript code blocks which send an XMLHttpRequest. While previous workarounds (such as the hashbang hack used by Twitter and Facebook) worked with carefully-crafted GET requests to provide a non-AJAX snapshot of a page to search engines, Googlebot is now able to perform both GET and POST AJAX requests itself. According to the blog post, only automatic requests are currently being executed and not those which are performed as a result of user action.

Just before the announcement, Digital Inspiration noticed that Facebook comments on TechCrunch were being indexed. Prior to Googlebot's new functionality, searching Google for the content of these comments wouldn't have returned the article that they were attached to. These changes should also allow other third party commenting platforms such as Disqus to be properly indexed as well.

Note that restricting search engines' access to any of the files needed to perform the request, such as an external JavaScript file or the file that is actually being opened by the request, will break Googlebot's ability to index the content. Developers and website owners will need to ensure that their robots.txt is configured correctly and they haven't left a stray robots meta tag in any required files that is blocking search engines.

Create a new thread in the US News comments forum about this subject
This thread is closed for comments
10 comments
    Your comment
  • Anonymous
    LOL, you announce it like Javascript is some brand new thing. Those who hate it will continue to use noscript.

    However, it's downright shocking that Google isn't already doing this, they index text found in flash, and they index the color content of images FFS.
    -7
  • festerovic
    One time I woke up and found Googlebot in bed with me.
    7
  • alidan
    funny how i turn off java script and flash from every website and only allow whats nessassary for functionality.

    the sad thing is, with this, they could force me to use javascript on more websights to get basic functions, as google now makes it searchable.
    4