robots.txt file

[borntoloaf]

Tweet

What are your thoughts on robots.txt files? I’ve read arguments that they are necessary for search engines, especially for pages you wish to block and for better spidering. I’ve also heard the argument that they are not necessary. Would love to hear opinions about it.

[Supercod]

I always put one in, even if I am not blocking anything.

If it is not doing anything for Search Engines, at least it's not making my error logs grow bigger

Here is the 2 I use the most

User-agent: *
Disallow:

and

User-agent: *
Disallow: /cgi-bin/

[Mogga]

Yes my main reason for putting one up was errors of things looking for it.

[jhood]

I'm considering doing the same.

[dmorison]

If you're running big sites, it's probably worth blocking ia_archiver:

User-agent: ia_archiver
Disallow: /

http://www.archive.org/about/exclude.php

[Sammy aff]

http://www.webmasterworld.com/robots.txt

take at look at others, find 'top sites' see what they block, does it work for the positive?