Is there any harm in not having a robots.txt file at all?

[Azam.net]

Tweet

Just wondering if the search engine bots count against a site with no robots.txt at all?

Thank you.

[Fraser]

I think I read something one that said you were as well to have a standard blank robots.txt then none at all.

But in saying that most of my sites don't have one and I think this was only to prevent 404 errors in your logs when robots.txt was requested!

[Nosmo_King]

Personally, I don't think it makes any difference at all.

The only time I use robots.txt is when I need to stop a spider.

[Azam.net]

Thanks Fraser and Nosmo King. I think I'll create a blank one as per Fraser's advice. I've read that elsewhere as well. Having no robots.txt creates a lot of errors in logs.

Interesting - just saw a CJ bot visiting. Didn't know they had one: "CJNetworkQuality; http://www.cj.com/networkquality "

[Fraser]

As for CJ's bot there have been a few interesting old threads about it before

http://www.a4uforum.co.uk/showthread.php?t=19003
http://www.a4uforum.co.uk/showthread.php?t=14756
http://www.a4uforum.co.uk/showthread.php?t=9942

[Tom]

I don't think it makes any difference from the bots point of view if you have one or not as far as rankings etc are concerned.

[Brendon]

There's no SE effect from a robots.txt (unless you are deliberately blocking part of your site of course - typically this would be an image directory, or /cgi-bin, things like that)

Putting a blank file up is a good idea though - reduces errors, makes the 404 report more meaningful. There's even a miniscule saving on bandwidth (404 pages take more bytes than 200 responses, especially custom 404s' in most cases)

[SimonDance]

Should i delete log files that are on my server??