robots.txt important?

[DavidSWP]

Tweet

Hi,

I created 4 websites using a blog package called Wordpress over the last 3 or 4 months but I forgot to put a robots.txt file in the root directory, I've now added it in basically setting no disallowed pages.

All of my sites got indexed by Google within the first week without me even submitting them, I guess this is because I used RSS to ping places like pingomatic and google found my sites from there. However, Google has not spidered my sites, its only indexed the front page. I'm getting a bit worried this is because I ommited the robots.txt but someone told me that if the robots.txt isn't there Google just assumes it can spider the whole site anyhow.

Is there any way to make google return to my pages if this is the case, or has Google just earmarked the sites and not got around to fully spidering them yet?

[sem4u]

Build up your links to your site and Google will spider your site more frequently.

You could also try submitting a Google sitemap to them.

[Itchy]

Hi,

I created 4 websites using a blog package called Wordpress over the last 3 or 4 months but I forgot to put a robots.txt file in the root directory, I've now added it in basically setting no disallowed pages.

All of my sites got indexed by Google within the first week without me even submitting them, I guess this is because I used RSS to ping places like pingomatic and google found my sites from there. However, Google has not spidered my sites, its only indexed the front page. I'm getting a bit worried this is because I ommited the robots.txt but someone told me that if the robots.txt isn't there Google just assumes it can spider the whole site anyhow.

Is there any way to make google return to my pages if this is the case, or has Google just earmarked the sites and not got around to fully spidering them yet?

Missing the robots.txt wouldnt make google not index all of your sites
things that would make them not bother at the moment is time
youve got a couple of new sites ..... so have 1000's of other people everyday
so your palying the waiting game just like 1000's of others
do like old matey said and add links + keep adding new content and then one day itll happen untill then theres nothing you can really do to make google do anything

[bentong]

add your site on your sigs.

you should block some robots because there are bots that are only a waste of bandwidth..

try installing the sitemap plugin for wordpress