Frustration with feeds

[Tobman]

Tweet

I run a few comparison sites and am having a hard time getting up-to-date merchant feeds. The prices are often different to what's on the merchant's website.

It would seem the only way around this would be to scrape the price from the merchant's site, but surely there must be a better way?

How does everyone else handle this? Even if it's a few pounds out, I think it will damage customer trust in my sites.

Thanks!

[accelerator]

Have you had any visitors to your site complain to you about this yet?

Rgds

[mrbishi]

Nope until merchants invest in proper real time feeds or APIs then the best and only way to get upto date accurate price and stock information is to scrape the information on their site

[Tobman]

Hi,

Unless the price is way off, I don't get any complaints. I run cron to keep the feeds up to date with what the merchants release, which stopped the complaints I used to get now and then about the prices being out of date. Visitors DO seem to buy if the price is say a few pounds more or less than what's on my site (and in the merchant's feed).

Have you had any visitors to your site complain to you about this yet?

Rgds

[Tobman]

Nope until merchants invest in proper real time feeds or APIs then the best and only way to get upto date accurate price and stock information is to scrape the information on their site

Thanks for the post. One thing I'd like to ask - do you have problems doing a scrape with cron jobs. I'm still in two minds whether it's actually scraping from cron. Seems to do it if I run the script by hand though.

Thanks!

[accelerator]

I guess one low-tech option would be to put a disclaimer on your site saying something like "Prices are those most recently communicated to us, and may differ when you visit the store".

Do merchants mind you scraping their site??

Rgds

[Tobman]

I guess one low-tech option would be to put a disclaimer on your site saying something like "Prices are those most recently communicated to us, and may differ when you visit the store".

Do merchants mind you scraping their site??

Rgds

Good idea on the disclaimer about the prices though. Thanks for that.

To be honest with you, I don't think merchants mind. I use use the non affiliate link from the feed, so it doesn't affect their EPC. And I use a browser user agent so it looks like a surfer just in case. I also make sure the scrape script is called visit.php or click.php - just in case they check the referrer logs and see a click from scrape.php and think something dodgy is going on.

I only do a scrape once per day - which basically amounts to one click checking the prices. Scraping sounds dodgy because it's associated with spammers scraping content but I use it to visit the odd merchant's site and check the prices. The only downside is that it seems to be slightly server intensive on my side. Hopefully in time feeds will be as up-to-date as the merchant's website prices.

Cheers!

Tom

[Darren M]

The issue I have with these out of date prices is that the feed is downloaded at 6am say and then the retailer has a few special offers that they put on at 9am. Your site shows the higher price all day and you miss out on sales.