Robots.txt & Domains

[eyefall]

Tweet

Hi there,

I wonder if anyone can help me.

We have a site that we are launching an affiliate program for. They have the .com and .co.uk domain names and do their own PPC using the .co.uk URL.

I would like them to replicate the .com site and allow affiliates to use this for their own PPC ads.

However - they do not want two different version of the files flying about, but can replicate it based on using the same fileset in apache.

Here is my problem - if they do this they will have duplicate content - is there a way of setting up the robots.txt file to be DOMAIN specific. If they are using the same files then they would use the same robots.txt - so ideally we only want to block on the .com

Hope that makes sense - if anyone has any suggestions please let me know.

Cheers

[Supercod]

Mirror the site in a different directory, point the domain to that directory, and just put a robots.txt that blocks everything in that directory, also might be worth adding norobot,noindex,nofollow tags to all the pages on the mirror site but that be a bit more complex to do.

Oh and I am not 100% sure on this but even doing all that I am not sure if google will think your breaking PPC rules or not.

Any possibility of coming up with a fair plan to allow Affiliates and Merchant to live side by side in the PPC space, the client might even remove themselves from everything accept the brand if the Affiliates do a good job.

[Brendon]

>> is there a way of setting up the robots.txt file to be DOMAIN specific.

Fortunately, there's no other way it can be. Robots.txt files need to uploaded to the root of a domain, so the path is domain.com/robots.txt - that's the only place it will be valid, so it's the only place a spider will look. All commands are therefore assumed to relate to the domain it has been placed in.

Also, note that different subdomains are treated differently, too. You could simply have a subdomain affiliates., and robots.txt it out, then use a global 301 to the www version. Done that before elsewhere, worked a treat

[tbp]

I haven't had a need to try this, but I think this would work? You can test it anyway through google sitemaps using the robots.txt test facility.

Have two robots.txt files, one called robots1.txt and one called robots2.txt

Then set up mod rewrite to redirect

http://www.domain.co.uk/robots.txt to robots1.txt

and

http://www.domain.com/robots.txt to robots2.txt

To the search engine it would appear that its just a standard robots.txt at the root of the website.

[Brendon]

It'd work, I'm just not sure why you'd do it... 2 URLs, 2 sets of content... why not just put different robots.txt files in the root of each domain? What purpose does the redirect serve?

[tbp]

No, I didn't mean have 2 sets of content. You have one site with both the .co.uk and .com pointing at it.

You then redirect just robots.txt, to the appropriate one for the domain, allowing all for .co.uk and blocking all for .com. To the search engine it just appears as a standard robots.txt in the root of the website, but depending on which domain they visited the site on depends on which robots.txt file is served to them ie robots1.txt or robots2.txt.

[Supercod]

It's like if both domains have there own VirtualHost setup in the Apache config, simply use Alias /robots.txt and point it to the file outside of the normal directory structure so it won't be picked up when using the other domain name.

Result same site, same directory's but the VirtualHost is setup to give each domain it's own robots.txt one would be a every search engine welcome and the other would be set up to tell them all to stay away. Like tbp, never actually done it but it's very possible to do.

[eyefall]

Thanks tbp & Supercod - I think both of these options will work so will send to the web developer to take a look at.

Cheers
J