screen scraping

[nts123]

Hi All

I've just started to play around with datafeeds, and love the ability to display the products exactly how I want them. My only issue with this is that you have to download the feed once a day, and hope then that the merchant has actually updated it.

The only other option appears to be to scrape the required info from the merchant site, but if like me you want to run a price comparison site, this means scraping each merchant site, and creating a new scrape instruction for each product and each merchant.

If I'm correct in the paragraph above, can someone please shed some light on a efficient and managable method to perform the scraping and manipulation of the data?

Also I intend to use a CMS like Joomla and the virtuemart plugin to handle the products, does anyone know of something that may already offer what I want?

cheers
Andy

[Fuller]

MetaSeeker is a FREE HTML wrapper factory. A new HTML wrapper for a target site is created in minutes without coding a single line.

XPath, XSLT and XML are made use of to express Web data extraction rules and to store extraction results.

It is very suitable for extracting web data in bulk. With the help of proprietary technology named as FreeFormat, the tool is very resistant to changes of target web pages. As a consequence, it is suitable for monitoring and extracting a site for a long duration. Not only can the data in form of tables be extracted, tree-like data can also be extracted.

For more information please visit GooSeeker, a community on web data extraction | GooSeeker

[Mogga]

Make sure you have merchant permission to scrape

[nts123]

Thanks, will check out the scripts above, I've not heard of those ones and I've searched all over the place!

Generally, are merchants quite happy to allow you to scrape their sites? I don't want to go to the hassle of sending out twenty odd emails, and waiting for responses, only to find 5 are happy.

If anyone else has experience of screen scraping that they're willing to share, it would be much apreciated.

cheers