Until recently, Google used inbound links and sitemaps to discover new content on your site and index them. The problem was, if your site has no sitemap or inbound links, your content will not be discovered by Google. Of course, most large-scale websites/blogs do have sitemaps and plenty of inbound links, but pages within small websites might go unnoticed at times. According to a new post at Google Webmaster’s Tool, Google is now using RSS and Atom feeds to find your content and index them in “almost real-time” speeds. Here’s an excerpt from their post:
“RSS/Atom feeds have been very popular in recent years as a mechanism for content publication. They allow readers to check for new content from publishers. Using feeds for discovery allows us to get these new pages into our index more quickly than traditional crawling methods. We may use many potential sources to access updates from feeds including Reader, notification services, or direct crawls of feeds. Going forward, we might also explore mechanisms such as PubSubHubbub to identify updated items.”
With services like Google Reader and FeedBurner under their control, it wouldn’t be that hard for Google to index content in real-time as and when they are published. We don’t know whether Google uses the feed just to discover pages or if they use the content in the feed as well.
If Google is using the content in feeds for indexing, then I guess they must come out and talk about it openly because some of us publish partial feeds, whereas other publish entire content. Those publishing partial feeds would be losing the juice!
Googlebot needs to access your feeds inorder to get your fresh content indexed, so if you are blocking your feeds using Robots.txt, it may be wise to modify and grant access to Googlebot. To know more about Robots.txt, please refer Google Webmaster Central.
Discussion
Comments for “Google Uses RSS Feeds to Index Your Site”