Pages tagged robots_txt:

10 common mistakes using robots.txt on your website | Houston Web Designer & SEO - Tips on Website Development & Website Marketing tips and Strategies
http://www.thomsonchemmanoor.com/10-common-mistakes-using-robotstxt-on-your-website.html
European Public Policy Blog: Working with News Publishers
http://googlepolicyeurope.blogspot.com/2009/07/working-with-news-publishers.html

Webmasters who do not wish their sites to be indexed can and do use the following two lines to deny permission. If a webmaster wants to stop us from indexing a specific page, he or she can do so by adding '<meta name="googlebot" content="noindex">' to the page. In short, if you don't want to show up in Google search results, it doesn't require more than one or two lines of code.
Webmasters who do not wish their sites to be indexed can and do use the following two lines to deny permission: User-agent: * Disallow: /
Robots.txt vs Rel=Nofollow vs Meta Robots Nofollow : SEO Book.com
http://www.seobook.com/robots-txt-vs-rel-nofollow-vs-meta-robots-nofollow
Aug 6, 2008 - Aaron Wall's chart comparing elements
SEO hard to understand
I was just fixing up our Robots.txt tutorial today, and figured that I should blog this as well. From Eric Enge's interview of Matt Cutts I created the following chart. Please note that Matt did not say they are more likely to ban you for using rel=nofollow, but they have on multiple occasions stated that they treat issues differently if they think it was an accident done by an ignorant person or a malicious attempt to spam their search engine by a known SEO (in language that is more rosy than what I just wrote).