Useful bot script (PHP)

[monkeyman]

Tweet

Want to know when the bots last crawled your site?

Add this to your PHP page(s).

<?php 
// send an e-mail if google crawls this page 
if(eregi("googlebot",$HTTP_USER_AGENT)){ 
// to test this script, change "googlebot" to "mozilla" 
if ($QUERY_STRING!= ""){ 
$url = "http://".$SERVER_NAME.$PHP_SELF.'?'.$QUERY_STRING; 
}else{ 
$url = "http://".$SERVER_NAME.$PHP_SELF; 
} 
$today = date("F j, Y, g:i a"); 
mail("email@domain.com", "Googlebot detected on http://$SERVER_NAME", "$today - Google crawled $url"); 
} 
?>

Just change to 'mail' variable to your own and then just sit and wait!
You can test it by using 'mozilla' instead of 'googlebot'. Remember to change it back tho, or you'll get a shedload of emails

[Frostie]

Only just found this - great little script monkeyman, have added it and already received an email about googlebot arriving

Any other little snippets would be welcome. thanks a million

[monkeyman]

No probs - glad to help.

I actually put it onto one of my template pages which pulls over 1000 products out and Google hit onto every one of em, resulting in over a 1000 emails!

[Frostie]

The script above, I guess, will work with other bots but obviously change the "googlebot" bit to something else.

if(eregi("ANY.OLD.OTHER.BOT",$HTTP_USER_AGENT)){

So what other bots could I type in, where it says "googlebot"?

I have found a list of loads of bots here, but which are the main ones to watch out for? Any help would be appreciated.

Mushy Grassy Ass

[monkeyman]

Shedloads here mate - http://www.jafsoft.com/searchengines...ots_and_others

I've mainly seen these ones in my logs :-

ia_archiver
Scooter
FAST-WebCrawler
GalaxyBot
Slurp
HenryTheMiragoRobot
teoma_agent1

Hope that helps!

[Frostie]

So if I added the following code, I would get an email if Google, Yahoo, Alexa or Mirago came calling?

<?php
// SET DETAILS
$url = "http://".$SERVER_NAME.$PHP_SELF;
$today = date("F j, Y, g:i a");

// GOOGLE DETECTOR
if(eregi("Googlebot",$HTTP_USER_AGENT))
{mail("email@domain.com", "GOOGLE detected", "Google crawled $url at $today");}

// YAHOO DETECTOR
elseif(eregi("YahooSeeker",$HTTP_USER_AGENT))
{mail("email@domain.com", "YAHOO detected", "Yahoo crawled $url at $today");}

// AELXA DETECTOR
elseif(eregi("ia_archiver",$HTTP_USER_AGENT))
{mail("email@domain.com", "ALEXA detected", "ALEXA crawled $url at $today");}

// MIRAGO DETECTOR
elseif(eregi("HenryTheMiragoRobot",$HTTP_USER_AGEN T))
{mail("email@domain.com", "MIRAGO detected", "Mirago crawled $url at $today");}
?>

[monkeyman]

Umm, looks ok to me.

Suppose the only way to test it, is to add it to your pages and hope they come along.

You could add 'Xenu's Link Sleuth 1.1b' to the list and run the Xenu link sleuth prog to see if accepts more than 1 bot.

Let me know, hoe you get on.