Spidering and indexing your own site

[jsp_1983]

Tweet

Does anybody know the best way to do this? Can you recommend any particular tools?

I want to spider and index my entire site and then be able to search it for a list of keyphrases - so far, I've not been able to find any tools that can do both.

[Mogga]

You need a sitewide keyword density checker
keyword density across website - Google Search

[scriptmonkey]

Am I right in thinking that you want the equivalent of a search engine for your own site?

If so, then atomz does quite a good job free with a few ads, and offers an ad-free solution for a subscription $80 a year up to 5,000 pages

Atomz Site Search

[jsp_1983]

Maybe I should clarify.

I need a tool (for my own benefit - not my visitors) that can spider all of my site, index it and then run a check against a list of imported keyphrases, reporting back which phrases are used on which pages. I don't need to know density- just where the phrases occur and, if possible, which phrases don't occur.

I know that there are tools available that will spider the site (I think Visio used to be able to do it), but I haven't found any that will spider the site and index it for me to be able to search through.

Something like Trellian Sitespider (Trellian Software - Site Spider Software Features), but can handle dynamic sites (like WP etc) and can index what it maps and allow you to search through the index.

I'm guessing it's not a particularly common tool....