URL restricted by robots.txt...

[polkadot]

Tweet

Hi guys,

Without disclosing my niche/url I have a blog that is indexed and ranked by Google but in a Google search only the name of the site shows up (no black or green text underneath).

Google webmaster tools reports 'URL restricted by robots.txt' against 'Sitemaps' and 'Restricted by robots.txt'.

I have checked my /robots.txt page and it says:

---------------------------------

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search

Sitemap: http://www.myurl.co.uk/feeds/posts/d...rderby=updated

---------------------------------

Can anyone spot any problems here?

Cheers

[moredial]

Does that sitemap line refer to your actual file name?

All examples I have ever seen use sitemap.xml.gz for the sitemap file. Does anyone know whether the bit after the ? could be messing up?

On the other hand, googlebot does have a few gremlins so try adding a section specifically for googlebot as well as the mediapartners bot.

What about meta tags - any syntax errors on the noindex,nocache or perhaps a nofollow?

[polkadot]

Thanks for the speedy reply

Yeh, I just replaced my url with 'myurl'. Do you know if it's possible to add a section specifically for googlebot in Blogger?

There are no other errors reported....

[moredial]

If you want a googlebot only area the easiest way is to disallow googlebot from all other areas via the robots.txt. Plus add the meta tag for googlebot to noindex all the pages you want google to ignore.

This really reads like a nocache or noindex problem. Anything like permissions in htaccess that you may have forgotten?

Can you PM me the domain name so that I can send a bot to visit and find out what sort of error that gives me.

[polkadot]

Thanks, have done.

Hmm, I'm thinking as it's a newish site the URL may have been picked up from a backlink and the content not crawled yet, although that doesn't explain the error on webmaster tools. Could this be the case?

[babloodmax]

Hi Buddy!!!


URL is restricted by robot text... is true for google while crawled your site...

Website Design Chennai | PHP Development Chennai