Add Sitemaps Autodiscovery in Robots.txt File

ADVERTISEMENTS

After creating a robots.txt file for this site I have experienced a good increase in my web site traffic. I am not really sure if the traffic increase was completely due to the robots.txt file, but it might have helped me like it helped Neil Patel.

I just came to know through Quick Online Tips that Ask.com, Google, Microsoft Live Search and Yahoo! have announced support of “autodiscovery” of Sitemaps. The new open-format autodiscovery allows webmasters to specify the location of their Sitemaps within their robots.txt file, eliminating the need to submit sitemaps to each search engine separately.

This step is very useful as webmasters can easily submit their content to the search engines and benefit from reduced unnecessary traffic by the crawlers. The search engines get information with regards to pages to index as well as metadata with clues about which pages are newly updated and which pages are identified as the most important and search users get more fresh content.

Just by adding the below line of code to your robots.txt file, you can help search engines find your sitemaps file and index your site better.

Sitemap: http://www.example.com/sitemap.xml

banner

Search JohnTP.com or view a random post

To receive this blogs articles for FREE on your email inbox, just enter your email address below and click 'Go':

Enter your email address: or .

Find out what I am doing currently by .

20 responses so far, Leave a comment

  1. 1

    egon

    April 13, 2007 at 12:53 am

    After reading your post on using a robots.txt file, I changed it a lot, it used to be just allowing all traffic through. While I didn’t see an increase in traffic after this, hopefully it helps some. I also just added the autodiscovery sitemap to my robots.txt, it’s a good idea and hopefully this will help standardize indexing a little bit.

  2. 2

    Gili

    April 13, 2007 at 3:30 am

    Thanks for this useful tip.

  3. 3

    Vijay

    April 13, 2007 at 1:51 pm

    I never found a way to submit sitemaps to livesearch.. Though I have submitted it to yahoo and google

  4. 4

    Mr.Byte

    April 14, 2007 at 12:46 am

    Does this mean, robots will read only the sitemap and not your site?

  5. 5

    egon

    April 14, 2007 at 12:55 am

    : No, a sitemap just tells the SEs what pages are on your site. This helps the search engines make sure all of your pages are crawled.

    For example…say I have an informational page that isn’t linked to from my main site, but I still want it indexed. Normally the search engines wouldn’t see that page because it doesn’t have a link to it, so it doesn’t know to look for it. A sitemap will tell the bot to go to that page and index it. Without a sitemap the bot wouldn’t know the page is there.

    Even with a sitemap, the search engine will still crawl you page and content and follow links out. the sitemap does not have this critical information.

    I hope that all made sense, I’ve never been good at explaining things.

  6. 6

    Dan and Jennifer

    April 14, 2007 at 1:00 am

    Hey John,

    This is great stuff. We get a lot of traffic from Google, but very little from MSN, Yahoo, or Ask.com.

    Just implemented it.

    I had already overhauled our robots.txt file last night after the threads on your blog and Everton’s about it. We didn’t have any special Wordpress considerations in there, so it should be much better now.

    Have an awesome day!
    Dan

  7. 7

    JohnTP

    April 14, 2007 at 1:13 am

    Has anyone’s traffic increased after creating a Robots.txt file?

  8. 8

    egon

    April 14, 2007 at 2:01 am

    : Yes, but I don’t know if it was related to the file or not. Ny traffic was already growing when I added mine, so it’s hard to tell.

  9. 9

    Mr.Byte

    April 14, 2007 at 11:51 am

    , that’s what I thought myself. So, if crawlers still crawl other pages, how can we say this will reduce unnecessary traffic by crawlers? As far as I know, sitemap is just to tell the crawlers what we think is important and asking them to crawl.

  10. 10

    Ashish Mohta

    April 14, 2007 at 12:12 pm

    are still required to be submitted to webmasters so you can know your crawl errors. Inclusion of this tag is just to tell crawlers more clearly. As they have agreed on common points. It will be lot better if all can take sitemap submissions

  11. 11

    egon

    April 14, 2007 at 12:57 pm

    : To reduce unnecessary crawler traffic, you use the robots.txt file to tell them which pages not to crawl. Even if those pages are in the sitemap file, whatever crawlers you defined to not crawl those pages, still don’t.

    When a bot comes across a page without a sitemap, it has to rely on links from your main page to direct it to all the pages you want indexed. When you have a sitemap, it makes it easier on the bot by saying “Hey Googlebot, here are all the pages on my site, now crawl them unless they’re disallowed in my robots.txt file.” That way Googlebot already knows all of your pages to crawl before it starts following links.

  12. 12

    Ashish Mohta

    April 14, 2007 at 1:37 pm

    Byte Check out JTP links

    “Create A Robots.txt File And Increase Your Search Engine Rankings” Its in the related articles.

  13. 13

    Ronald

    April 14, 2007 at 3:23 pm

    Mohta ; Just put the url http://www.example.com/sitemap.xml or Sitemap: http://www.example.com/sitemap.xml ?

    Thanks

  14. 14

    Mr.Byte

    April 14, 2007 at 3:29 pm

    & Ashish, I understood now what he meant. Thanks for the clarification.

  15. 15

    sourcer

    April 14, 2007 at 5:38 pm

    That is a nice feature, after I have added the robots.txt file on my blog the traffic from google increased, I hope this will also be good for my blog.

    Off-topic: John I noticed that you are using Kontera, I will be happy to post some stats or reviews cuz I am trying them right now and I not so satisfied.

  16. 16

    Mr.Byte

    April 14, 2007 at 9:33 pm

    , The later one… Check out https://johntp.com/robots.txt for example.

  17. 17

    Ashish Mohta

    April 14, 2007 at 10:50 pm

    : Byte is right. You have to put

    Sitemap: http://www.example.com/sitemap.xml ?

    “Sitemap” is the keyword which is understood by crawlers.

  18. 18

    apartamentos em portugal

    April 20, 2007 at 1:56 am

    thanks

  19. 19

    Arthur

    April 21, 2007 at 8:41 pm

    If you have any problems to create XML sitemap files for your sites or to notify search engines about updated sitemap files, try Sitemap Writer Pro. It is a powerfull tool that helps you to create and keep up-to-date sitemaps for your websites.
    Sitemap Writer Pro has new tools - FTP manager for uploading sitemaps, search engine notification tool (now supports Ask.com and MSN.com), site crawler for adding an URLs into the sitemap and Yahoo Index viewer.

  20. 20

    Adil

    January 27, 2008 at 2:14 am

    has anyone of you seen any difference after linking ur sitemaps in robots.txt file?

    Have something to say?




    Copyright ©2005-2008 JohnTP, All rights reserved.