Online XML sitemap generators that dont need to be installed on your own server

[alitech]

Tweet

Hey guys

I am in a bit of a pickle right now with my XML sitemap generation.

Background:

My website is over 400,000 pages and growing constantly every day. This is proving to be hard to keep up with when actually tryign to find out exactly how many pages there are. the site is built on code ignitor PHP framework and references to 5 DBs.

I have tried to create an xml sitemap via a script on my server that generates URLs from the DB. This is not working really as it breaks every now and then. Its too server intensive and kills the site. I have had my developers try to optimise this by creating delays in the DB queries etc and what not.. there are just too many pages to index.

I did a few google searches and found xml-sitemaps.com which is also a script that is installed on the server to do pretty much the same thing, only it doesnt access the DB tables but actually crawls all the pages. This is also not really working as it breaks all the time and never gets to complete the full sitemap generation. By introducing delays in the indexing, it will take more than 5 days to index everything. But I am not able to get it to work continously. It breaks every few hours and I have to restart.

Anyways, I am looking into alternative solutions where I can get an external sitemap generation tool to do all of this for me. This way, I don't have to worry about an extra load on the server resources. I would like to have some control over the crawling like introducing delays after indexing every 10 pages so that the site speed is not affected.

Any ideas where I can find such a tool / service?

[Barry]

If you get an external site to scan your site for its link architecture I would have thought it would place more pressure on your server than to do it internally using your own script?

Dont forget you can have a sitemap of your sitemaps with google. You could split your website into its natural categories and process the sitemaps on the fly to google.

Ta

Baz