Robots.txt and robots meta tag use?

[blackstar]

Tweet

Hi all

Not far away from the launch of my first site now, and have read a little about robots.txt and meta tags.

I've got a pretty small site about 30 pages at most for launch, but this is going to grow over time. Do all search engines look at the robot meta tag, or is the robots.txt file the only way to stop bots indexing unneccessary content?

Also read all sites should have a robots.txt file even if it's blank?? Is that the case?

Most grateful of any advise.
N.

[tbp]

All search engines and crawlers should read the robots.txt tag, and follow the directives within, which tells them the files or directories they aren't allowed to index.

You can also use the meta tag:

< remove_this_meta name="robots" content="noindex,nofollow">

on a page to stop it being indexed, but the robots.txt file is the best solution, although you have to be very carefully with pattern matching so you don't accidentally tell them not to index your whole site!

Its best to create a robots.txt site for every site, even if its empty, as the spiders frequently request it, and if it doesn't exist your error log files can fill up with 404 requests for the file.

Have a look at http://www.robotstxt.org for lots of info on this topic.

[blackstar]

Thanks for the advice, looks like the Robots.txt file is the way to go.

You couldn't answer another minor question I have talking about 404's. Want to create a custom 404 page on site, rather than sending them to a standard one, in order to keep visitors on my site. How would I go about adding this to my site?

[tbp]

If you're using Apache, add the following to an .htaccess file in the root of your site:

ErrorDocument 404 /path_to_custom_404_page.htm

You`ll need to have the relevant permissions for the htaccess file, as some directives can be set to be ignored, so you`ll need to talk to your host if this doesnt work.