Restricted by robots.txt

[crazihos]

Tweet

Hi,

Can anyone tell me how I can tell if my loan site Compare cheap loans, personal loans, unsecured and secured loans online. is restricting google by robots.txt and if it is how do I stop it?

Google Webmaster tools says it was restricted by robots.txt on the 31st of July but I think i've changed that setting now (the sites built in WP so i've changed the settings and allowed google, yahoo etc to see it now).

Webmaster tools also says it last tried to crawl the homepage on the 15th of July - does that simply mean its not trying to index my homepage at the moment and if thats the case how come its picking up my blog posts in seconds?

Any advice would be great guys.

Cheers

Russ

[moredial]

Your robots.txt file looks OK now.

The problem is, if you do block bots they have a habit of not coming back for some time. It looks like googlebot has now indexed 38 of your pages. Live has 96 pages and Yahoo 53 pages.

[crazihos]

Cheers for checking that for me mate,

Looks like its just a case of being patient now than and waiting for that pesky spider to come back around. I'm hoping that it will index it from one of my blog posts but its just not happening at the moment.:td

Russ

[tbp]

In Webmaster tools you have the facility to test your robots.txt file against certain url's, so you can see whether its blocking things or not.

[lauriebred]

Webmaster tools has (have?) been telling me for months that's it's blocked from accessing a whole stack of pages that's it's been merrily spidering throughout. I reckon it's broken.

[kolli-tosg]

I have the same problem with G Webmasters Tools as well, tells me they are blocked when they haven’t been for about a year.