Amazon bot scraping my site?

[ep90]

Tweet

I've been looking at our search logs and it appears a bot of some type has been entering hundreds of search terms via our searchbox. I've seen google do this before, but this IP has a reverse lookup of AmazonAWS (ip 174.129.139.28)

Has anyone seen this before? The queries they are doing seem slightly odd and the phrases they are entering have no spaces "nintendods" rather than "nintendo ds" for example.

[pollensa]

I've had loads of these eating up the bandwidth of my forum. I don't know what they're doing but when they've been they're they uses up masses of bandwidth - more than googlebot etc. When I notice them I ban their IP addresses (via cPanel). It's a real pain & can cost.

Apparently you can complain to amazon but you need to produce extracts of raw logs so that they can identify who was using that Ip address at that time & I can't be bothered.

[maverick]

Hi,

You should be able to block any legitimate search bot by adding the appropriate entry into your robots.txt file.

cheers,

mav

[pollensa]

Hi,

You should be able to block any legitimate search bot by adding the appropriate entry into your robots.txt file.

cheers,

mav

I don't think these are search bots. What I get is 'traffic' from a variety of Ip addresses which include the terms "compute-1.amazonaws.com" (Amazon Web Services) On some stats packages it shows as an 'unknown bot'. If you search the net you'll see that other people have been complaining about it.