Time Wasting Search Bots

[scifind]

Tweet

Are there any search bots out there that are a sap ot bandwidth with no real benefit.
Looking for ones that are big timewasters.

I see more and more in my server logs - I am tempted to just restrict bots to yahoo, google, ask and a couple of others.

Any comments or suggestions

[deadgoodundies]

How are you going to block them??
Most of the less well known bots tend to ignore robots txts anyway

[Brendon]

Webmasterworld has a pretty comprehensive robots.txt (located at http://www.webmasterworld.com/robots3). Brett has spent considerable effort over the years ID'ing and banning resource hog 'bots - as enormous as the place it, it ran off a single medium-spec box for years, and when he started naming G updates, the box reached 120% rated load before it finally fried...

Most of these are somewhat obedient, at least - for the really naughty ones, you need to be banning by IP in any case, since they often run with spoofed UAs, like the standard IE on, or googlebot