Helping people with computers... one answer at a time.
While robots.txt won't remove links from Google, it is a powerful tool for webmasters to use to control what Google sees.
Google has informed me that a link can be removed from Google if the webmaster of the site uses a robots.txt file. However, Google can not confirm if other engines can do the same-MSN, Yahoo etc. Should a webmaster be able to remove a link from other search engines by using the robots.txt file described by Google?
First, let's be clear - it won't remove anything. I assume that someday the link will disappear off of Google as it loses relevance, or maybe Google does something more related to cleaning dead links.
But robots.txt is still important for several reasons.
Robots.txt allows you to tell Google not to look at the pages in the first place. Google doesn't look, and the contents don't get added to its index.
All search engines are supposed to pay attention to robots.txt. And most well behaved major engines do. MSN, Yahoo, and so on are all major engines and do the right thing.
However do know that there are other search engines and crawlers that don't pay attention and will spider your site anyway.
Search Engine World has a good tutorial on robots.txt.
Comments on this entry are closed.
If you have a question, start by using the search box up at the top of the page - there's a very good chance that your question has already been answered on Ask Leo!.
If you don't find your answer, head out to http://askleo.com/ask to ask your question.