redirect.php and the robots.txt

[warren_p]

Tweet

Hi Everyone,

I have just put all my affiliate links in a redirect.php file on one of my sites but I am not sure what the robots.txt file should look like so that it isn't crawled.

Can anyone tell me what needs to be in it or give an example of one?

Thanks.

[Andrew Starr]

User-agent: *
Disallow: /redirect.php

that should do it

[warren_p]

Oh right.

Thanks.

[monkeyman]

If you have a Google Webmaster Tools account, you can put the contents of your robots.txt file in with some of your URL's and it'll tell you wheter they'll be crawled.

[AndyBeard]

Make sure you nofollow all the links, otherwise you will create lots of hanging/dangling pages