Merchant Scraping

[victor_m]

Tweet

This is purely for above board reasons and to save me inputting each and every product individually as the merchant doesn't have a datafeed. Any help would be great. Does anyone know of any good screen scraping programs or links to good tutorials on how to build your own?

[scifind]

Before you do this - have you had a chat with the Merchant?
They may be accommodating.
It would be better for the merchant to have a datafeed and a more stable system for yourself, as minor site changes can kill data scrapers.

[victor_m]

Just as a quick update I have found this tutorial for their tool (not affiliated) here which seems quite decent. Will have a play around with it but it seems to be along the lines of what I need, plus their demo is based on scraping the same type of store that I will be looking to scrape.

[tbp]

There was a thread about this topic a couple of weeks ago.

Some merchants are really against scraping their sites, as they believe it would put too much load on their servers.

Its also quite difficult, as getting the computer to work out where on the page the price is for example can be quite difficult. If the price is between two specific tags its easy, but if the price is just tacked onto the end of the product description for example, its much harder.

As a result, you end up having to write code for each and every site that you want to scrape, telling the computer where the data is that you want to capture.

[pistol101]

As TBP has said some merchants are really against it. In certain sectors the merchants are actually charged for searches. Many merchants will simply block your spiders.

[QualityNonsense]

Anthracite (Metafy LLC >> MacOS X Software for Web Mining Automation) is excellent, costs $99 or so and there's a free trial. It's Mac only, though.

You create a custom scraper for each site with a drag 'n' drop interface.

[John Jupp]

Hmmm. $99 eh! We've just developed a software tool that does all that and more. Guess I'll have to look into retailing it. It creates lovely datafeeds.

[wackimonki]

Yes, I would always ask permissions before scraping data. I've produced a website before using data scraped from a government website. After getting some decent traffic, we also got a cease and desist phone call.

I would say datafeeds all the way. I'm using PPC, and they have all the information I need.