Why does google spider my robots.txt file several times a day?
One one of my websites Google crawls the robots.txt file several times a day which doesn't actually have anything in it but only crawls my index page between every 1-3 days although I update it every day. Why is it doing this and how can I switch it around!
|