robots.txt and engine rewrite - will both be followed?

[Elaine]

I'm trying to stop Google indexing sites that it might construe as a duplicate - I have 'killed' off all the domains that pointed to my main site but there is one other site that is listed quite heavily in all the other search engines - and to save time on continually updating both I use the 'engine rewrite code in the .htaccess file on the second site to send everything to my first site.

so I have placed the robots.txt file

User-agent: *googlebot
Disallow: /

- created it in notepad and then used Aceftp to transfer it into the second site - and uploaded it.

will Google now read the robots.txt file first and ignore the site - or

will it read the htaccess file first and simply move onto my main site.

All I want is Google to only see the main site - will this work.

[ChrisM]

Does it matter if you had multiple domain names pointing at a site? I assumed the big G is clever enough to look at the IP address the domain name resolves to and if more than one are the same then treat it as the same site rather than duplicate content.

The reason for my assumption is this must be the case where the domain name extension is the only difference, i.e. mydomain.co.uk and mydomain.com. I suppose this might be a specific case that Google will ignore, but if they're going to the lengths of checking for duplicate content I'm sure they'll compare ip addresses.

[Elaine]

I've always thought it was OK - but I'm seeing a heck of a lot more results showing the 'ghost' sites instead of my main one - I'm just being cautious.

The second site is a different kettle of fish as it was a 'stand alone site' and is featured heavily in MSN and Yahoo - so I don't want to upset that relationship, either.


oh its a difficult business sometimes - like having continually squabbling spoilt kids