that should do itCode:User-agent: * Disallow: /redirect.php
Hi Everyone,
I have just put all my affiliate links in a redirect.php file on one of my sites but I am not sure what the robots.txt file should look like so that it isn't crawled.
Can anyone tell me what needs to be in it or give an example of one?
Thanks.
that should do itCode:User-agent: * Disallow: /redirect.php
Oh right.
Thanks.
If you have a Google Webmaster Tools account, you can put the contents of your robots.txt file in with some of your URL's and it'll tell you wheter they'll be crawled.
Make sure you nofollow all the links, otherwise you will create lots of hanging/dangling pages
There are currently 1 users browsing this thread. (0 members and 1 guests)
Bookmarks