Affiliate Marketing
Forum Search

Reply
 
LinkBack Thread Tools Display Modes

  #1 (permalink)  
Old 17-12-02
bertm
Guest
 
Posts: n/a
  robots.txt


I'm not bothered about restricting any spider's from accessing my sites but everytime they request robots.txt (which doesn't exist just now) I get an error in the logs too.

Can I just put an empty file up as robots.txt or is there some way I can write "welcome, come on in everybody" in robot language :-)
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Sponsored Links
  #2 (permalink)  
Old 17-12-02
999GI
Guest
 
Posts: n/a
  Re: robots.txt


empty file in the root works fine, although personally I dont think you need it. Spiders are a lot more persistent these days
Digg this Post!Add Post to del.icio.usBookmark Post in TechnoratiFurl this Post!
Reply With Quote
Reply

Bookmarks


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On

Similar Threads
Thread Thread Starter Forum Replies Last Post
robots.txt and engine rewrite - will both be followed? Elaine Widgets, Coding, AJAX, PHP - Technology & Affiliate Marketing 2 18-11-04 10:51 AM
ignoring robots.txt whysleep.co.uk Organic Google Search Optimisation 3 06-11-04 12:27 PM
Robots.txt and ask jeeves scifind Widgets, Coding, AJAX, PHP - Technology & Affiliate Marketing 2 13-08-04 03:20 PM
Robots.txt tomj Search Engine Strategies 6 27-02-04 09:57 PM
CJ to do SEO Kandevil Commission Junction - CJ UK 21 26-11-03 10:08 PM


Affiliate Marketing RSS Feeds - Contact Us - Affiliate Marketing - Archive - Privacy Statement - Top

Content Relevant URLs by vBSEO 3.2.0 RC7