Sitemap questions

[axod]

Tweet

Hi, been trying to get google to index more of my pages etc.
So I created a sitemap, and uploaded to google.
I logged in again, but there seem to be some problems:

1. It says 51 pages were restricted by robots.txt. So I check my robots.txt and it says

User-agent: *
Disallow:

That does mean "crawl anything you like". right? Anyway, I try out the google robots.txt tool, taking one of the urls it said was restricted, and in its own tool, it says it's NOT restricted! Anyone seen this before? Any idea what's going on?

2. I check what pages google has indexed, and the results are just bizarre.
http://www.google.com/search?q=site:...es.co.uk&hl=en

Notice the 'privacy policy' result (2nd down). It seems to have taken the text on the page, and repeated it loads of times to make a summary! Why would it do that?

I realise the answers to these may just be unknown, but if anyone has any ideas I'd be really interested.

Thanks

[seekituk]

Hi,

Funnily enough I was speaking to someone who had the same problem on Friday.

If you want Google to crawl the whole site, do you need to have a robots.txt file? I used to have one for my sites but removed it.

I have also seen my sites pages with duplicated descriptions in Google but it only seems to last a few hours and then reverts to normal. I've just checked yours via the link provided and the descriptions all look fine.

Regards

Jason