Add Sitemaps Autodiscovery in Robots.txt File
ADVERTISEMENTSAfter creating a robots.txt file for this site I have experienced a good increase in my web site traffic. I am not really sure if the traffic increase was completely due to the robots.txt file, but it might have helped me like it helped Neil Patel.
I just came to know through Quick Online Tips that Ask.com, Google, Microsoft Live Search and Yahoo! have announced support of “autodiscovery” of Sitemaps. The new open-format autodiscovery allows webmasters to specify the location of their Sitemaps within their robots.txt file, eliminating the need to submit sitemaps to each search engine separately.
This step is very useful as webmasters can easily submit their content to the search engines and benefit from reduced unnecessary traffic by the crawlers. The search engines get information with regards to pages to index as well as metadata with clues about which pages are newly updated and which pages are identified as the most important and search users get more fresh content.
Just by adding the below line of code to your robots.txt file, you can help search engines find your sitemaps file and index your site better.
Sitemap: http://www.example.com/sitemap.xml
Search JohnTP.com or view a random post
Related Articles
Find out what I am doing currently by .
Posted on April 12th, 2007 | Category: Internet |
egon
April 13, 2007 at 12:53 am
After reading your post on using a robots.txt file, I changed it a lot, it used to be just allowing all traffic through. While I didn’t see an increase in traffic after this, hopefully it helps some. I also just added the autodiscovery sitemap to my robots.txt, it’s a good idea and hopefully this will help standardize indexing a little bit.
Gili
April 13, 2007 at 3:30 am
Thanks for this useful tip.
Vijay
April 13, 2007 at 1:51 pm
I never found a way to submit sitemaps to livesearch.. Though I have submitted it to yahoo and google
Mr.Byte
April 14, 2007 at 12:46 am
Does this mean, robots will read only the sitemap and not your site?
egon
April 14, 2007 at 12:55 am
: No, a sitemap just tells the SEs what pages are on your site. This helps the search engines make sure all of your pages are crawled.
For example…say I have an informational page that isn’t linked to from my main site, but I still want it indexed. Normally the search engines wouldn’t see that page because it doesn’t have a link to it, so it doesn’t know to look for it. A sitemap will tell the bot to go to that page and index it. Without a sitemap the bot wouldn’t know the page is there.
Even with a sitemap, the search engine will still crawl you page and content and follow links out. the sitemap does not have this critical information.
I hope that all made sense, I’ve never been good at explaining things.
Dan and Jennifer
April 14, 2007 at 1:00 am
Hey John,
This is great stuff. We get a lot of traffic from Google, but very little from MSN, Yahoo, or Ask.com.
Just implemented it.
I had already overhauled our robots.txt file last night after the threads on your blog and Everton’s about it. We didn’t have any special Wordpress considerations in there, so it should be much better now.
Have an awesome day!
Dan
JohnTP
April 14, 2007 at 1:13 am
Has anyone’s traffic increased after creating a Robots.txt file?
egon
April 14, 2007 at 2:01 am
: Yes, but I don’t know if it was related to the file or not. Ny traffic was already growing when I added mine, so it’s hard to tell.
Mr.Byte
April 14, 2007 at 11:51 am
, that’s what I thought myself. So, if crawlers still crawl other pages, how can we say this will reduce unnecessary traffic by crawlers? As far as I know, sitemap is just to tell the crawlers what we think is important and asking them to crawl.
Ashish Mohta
April 14, 2007 at 12:12 pm
are still required to be submitted to webmasters so you can know your crawl errors. Inclusion of this tag is just to tell crawlers more clearly. As they have agreed on common points. It will be lot better if all can take sitemap submissions
egon
April 14, 2007 at 12:57 pm
: To reduce unnecessary crawler traffic, you use the robots.txt file to tell them which pages not to crawl. Even if those pages are in the sitemap file, whatever crawlers you defined to not crawl those pages, still don’t.
When a bot comes across a page without a sitemap, it has to rely on links from your main page to direct it to all the pages you want indexed. When you have a sitemap, it makes it easier on the bot by saying “Hey Googlebot, here are all the pages on my site, now crawl them unless they’re disallowed in my robots.txt file.” That way Googlebot already knows all of your pages to crawl before it starts following links.
Ashish Mohta
April 14, 2007 at 1:37 pm
Byte Check out JTP links
“Create A Robots.txt File And Increase Your Search Engine Rankings” Its in the related articles.
Ronald
April 14, 2007 at 3:23 pm
Mohta ; Just put the url http://www.example.com/sitemap.xml or Sitemap: http://www.example.com/sitemap.xml ?
Thanks
Mr.Byte
April 14, 2007 at 3:29 pm
& Ashish, I understood now what he meant. Thanks for the clarification.
sourcer
April 14, 2007 at 5:38 pm
That is a nice feature, after I have added the robots.txt file on my blog the traffic from google increased, I hope this will also be good for my blog.
Off-topic: John I noticed that you are using Kontera, I will be happy to post some stats or reviews cuz I am trying them right now and I not so satisfied.
Mr.Byte
April 14, 2007 at 9:33 pm
, The later one… Check out https://johntp.com/robots.txt for example.
Ashish Mohta
April 14, 2007 at 10:50 pm
: Byte is right. You have to put
Sitemap: http://www.example.com/sitemap.xml ?
“Sitemap” is the keyword which is understood by crawlers.
apartamentos em portugal
April 20, 2007 at 1:56 am
thanks
Arthur
April 21, 2007 at 8:41 pm
If you have any problems to create XML sitemap files for your sites or to notify search engines about updated sitemap files, try Sitemap Writer Pro. It is a powerfull tool that helps you to create and keep up-to-date sitemaps for your websites.
Sitemap Writer Pro has new tools - FTP manager for uploading sitemaps, search engine notification tool (now supports Ask.com and MSN.com), site crawler for adding an URLs into the sitemap and Yahoo Index viewer.
Adil
January 27, 2008 at 2:14 am
has anyone of you seen any difference after linking ur sitemaps in robots.txt file?