Google Duplicate Content Filter

darrenw · 12-01-05

Myself and a fellow affiliate acquaintance have noticed some significant changes to the way Google's duplicate content filter works. This may have been raised before, so apologies if so.

This change is going to be very significant to affiliate sites that heavily rely on SEO with product feeds.

Searching Google for "3 changes of costumes, and a crowd of wild fans" returns 170 results - but only one is displayed, with the following text shown at the bottom:

In order to show you the most relevant results, we have omitted some entries very similar to the 1 already displayed.
If you like, you can repeat the search with the omitted results included.

The one result that is shown is from Amazon UK (probably the "original" content).

The "hidden" results look like affiliate sites that are displaying content from Amazon.

The duplicate content filter seems to be working very well (from Google's point of view!) It catches very short phrases. For example, "You see choreographed" The duplicate content pages are all hidden - only unique content is shown.

At first we thought that Google might have been manually fudged to treat Amazon as the "master" but this is not the case.

For example, searching for "Take the strain out of KS2 National Tests revision with" The top seven are unique enough, but results 8 to 15 have been hidden as duplicates - and some of the hidden results are from Amazon.

Just a few thoughts at this stage, but this looks like a significant change that could severely affect some affiliate sites.

Cheers,
Darren.

Rich · 12-01-05

Taking your last example, the book is published by the BBC, so bbcshop is probably the right site to show instead of Amazon - though I'm not sure if Google has figured that out or if its just luck!

For books,dvds,etc this could also make quite a change for merchants as the majority use the same databases to build their product descriptions.

Having said that, there are still a lot of duplicates around - try "Dan Brown masterfully concocts an intelligent".

I can't really see any pattern in how it chooses which sites to hide.

darrenw · 12-01-05

Yes - in the last example, it is indeed the BBC site that can "claim" to have the original.

I wonder if we have stumbled across just the begining of a duplicate content filter, as it only seems to be working at the moment in certain specific areas...

Supercod · 12-01-05

The only issue I see with this at present is I have Espotting site and all the content is the exact same as other Espotting sites, yet I am getting listed and making a few bucks.. as are other Espotting sites so can't be a complete duplicate content filter yet.. plus you have the age old question as to who was first and who has the rights to show the content over someone else. Would you remove every English Dept in the world because they all have the same text from a work of Shakespeare? What about news sites that all take a feed from Reuters (is that how you spell it), are they all not duplicate content also.. this is the problem Google face and I don't think their be any easy ways to get it right, but you can certainly try and keep it tidy.

giveasyouget · 12-01-05

I use google news quite a lot and on the reuters feed I know google news does strip duplicate news stories into an "expand your results", tho i've never checked if it's emulated in the main index...

perhaps we should do an analysis of what percentage of the content of sites that are filtered is duplicate and what is unique?

or perhaps we'll find that searching with "quotes" activates a far more stringent filter than simply searching alone?

Just done a quick scan of the stats for an amazon feed site I run - of the 1000 keyphrases displayed there isn't a single one with quotes in it, so i'd guess this is probably it. I still recieve plenty of traffic to the site.

Mogga · 13-01-05

OK just tried google with a strip of text I've written that is on one of my sites and the google filter doesn't only show mine - it shows 8 in total.. with the message at the bottom

In order to show you the most relevant results, we have omitted some entries very similar to the 8 already displayed.
If you like, you can repeat the search with the omitted results included.

BUT annoying as hell the text being mine doesn't mean my site is first.
AND I have a higher PR than the site which is first.

hmmm

giveasyouget · 13-01-05

got to be a pain, its true.

what i'd be interested to know is if other affiliates using feeds could check their logs and see if they have any key phrases surrounded by quotes for the month of january?

purple · 13-01-05

i found 241 sites with the content of one of my pages!!! the desrption on google is my text but if you click on it it just goes to search results.

How can I stop this?

Rich · 13-01-05

If you are searching for a phrase from your site, the Google snippets will typically just show that phrase and you will find a lot of search results pages as those are the most likely to have included the phase from your site.

Also, if any of the words from the phrase have been used in the search that generated that search results page, it is likely to appear as a better match for that phrase than your own site.

As for what you can do, you could find out what search engine feed they use and then ban its bot, but thats well shooting yourself in the foot by removing a possible traffic source when there is no real problem with it - how many people will type a phrase from your site into Google wanting to get to your site?

purple · 13-01-05

I really am confused because since the last update I am getting much less google traffic, and for my key keywood these "directories" are getting much higher listings.

I would not mind if they listed the whole of my content to boost their listings as long as I was still above them!! It seems google has penalised my site in some way as being duplicate content.

I definitly will be concentrating on yahoo and msn who now provide 90% of my traffic.

Also now use yahoo and msn for my own searches as google serves up poor results.

Mogga · 08-03-05

I just tried a horrid 302 redirect on copyscape and it shows def as being the site its redirecting to.
(ie: the url is for all intents and purposes read as my url rathter than mine trapped in a redirect. Does that make it clearer?)

I have contacted the company doing it AGAIN ...