Wordpress robots.txt settings

[paultp]

Tweet

I'm sure G has just slapped my site for duplicate content, I'm beginning to hate wordpress!

Is there a good, up to date guide on basic robots.txt settings for wordpress anywhere?

Also, should I be fooling with meta settings too?

Ta
Paul

[kickflip]

I use this in my robots.txt's on wordpress, can't remember where I picked it up from originally:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

Sitemap: http://www.yourdomain.co.uk/sitemap.xml

[gadget]

Why do you think this has anything to do with WP?

[paultp]

Why do you think this has anything to do with WP?

Because initially I had a single page with some products and some text on it and since publishing the wordpress site it dropped out of the rankings and then came back lower.

I have a side menu that presents the products from a database in different ways e.g. by manufacturer, by type of product, by manufacturer and type of product etc. I'm still working on the site.

I don't have anything other than products on these pages. They link through to either; a post with detailed description and price comparison; or direct to the item on the merchant's site. So I think they are being seen as duplicate content. I also think that the same content can be seen by going through different files in the WP directories.

I had a quick look at duplicate content and WP and found a load of articles that are mostly out of date on setting up robots.txt to prevent duplicate content as it is a common problem evidently.

Just trying to close off everthing obvious to see what effect it has.

[paultp]

I use this in my robots.txt's on wordpress, can't remember where I picked it up from originally:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# digg mirror
User-agent: duggmirror
Disallow: /

Sitemap: http://www.yourdomain.co.uk/sitemap.xml

I think these are the settings from the wordpress codex

I've started with that, having changed it to reflect my directory structure (and domain!) but there is normally something a bit better. I did notice someone that had changed their header.php to insert meta settings depending what type of page/post the bot was on.

[Keith]

This robots meta pluging is great for avoiding dupe content Meta Robots WordPress plugin - Yoast - Tweaking Websites

[gadget]

Because initially I had a single page with some products and some text on it and since publishing the wordpress site it dropped out of the rankings and then came back lower.

I have a side menu that presents the products from a database in different ways e.g. by manufacturer, by type of product, by manufacturer and type of product etc. I'm still working on the site.

I don't have anything other than products on these pages. They link through to either; a post with detailed description and price comparison; or direct to the item on the merchant's site. So I think they are being seen as duplicate content. I also think that the same content can be seen by going through different files in the WP directories.

I had a quick look at duplicate content and WP and found a load of articles that are mostly out of date on setting up robots.txt to prevent duplicate content as it is a common problem evidently.

Just trying to close off everthing obvious to see what effect it has.

Can you post an example?

[pete_coles]

This robots meta pluging is great for avoiding dupe content Meta Robots WordPress plugin - Yoast - Tweaking Websites

On the money mate, simple a brilliant plugin. Joost is also excellent at keeping his plugins up to date

[paultp]

On the money mate, simple a brilliant plugin. Joost is also excellent at keeping his plugins up to date

I think I had this site recommended to me before. Now that I understand a bit more about wordpress I'll have another look.

Cheers
Paul

[gadget]

This is quite a good article on the subject .. Official Google Webmaster Central Blog: Demystifying the "duplicate content penalty"