Cross-network data feeds

[Gringo]

Hi all,

I have been trying to find some common value within data feeds across different affiliate networks/merchants and seem to be hitting a wall.

I am writing a comparison engine and all's well except that the various feeds I'm pulling in need heavy editing prior to db insertion and use. Some merchants provide brand names, some don't; some provide model numbers, some don't, some provide...

Am I missing something (a common thread across aff nets/merchants perhaps) or am I going about feed utilisation in the right way?

If I am on the right track, how often would you recommend that I update the feeds?

Any advice, hints or references would be appreciated!

Thanks, Paul

[tbp]

Unfortunately this is a common problem!

I started off with a big feed driven website and ended up giving up and converting to smaller niche websites instead. Spent so much time going through and manually correcting data it just wasn't worth it in the end!

I wrote a load of tools which were supposed to automate everything and make the site easy to manage, but unfortunately they just couldn't work with a number of feeds from different merchants as the data was in the wrong format, or most frequently incorrect. One of those situations where although the tools worked fine, you put rubbish in, you get rubbish out.

There are of course a number of merchants whose feeds do work fine, and you don't have problems with, and its great if you can tailor your site just to these merchants. However, writing something broad taking feeds from a number of merchants and networks, where you can't so much pick and choose specific merchants as you need a broad range is always going to be a problem and involves a lot of hard work.

[Gringo]

Originally Posted by tbp Unfortunately this is a common problem! ... you put rubbish in, you get rubbish out.

Ain't that the truth

During coding I pulled three feeds for test purposes (about 8,000 rows) and got great results as the product descriptions were closely matched. Those random feeds? Dixons, Currys and PC World! I guess euphoria is best enjoyed in small doses

Thanks tbp, I'll plug away but may well follow your example and go niche.

[tbp]

Thanks tbp, I'll plug away but may well follow your example and go niche.

Well stick at it for a bit longer, you might have better success than me!

Its always been a pet hate of mine though, as its so easy to create an accurate working feed.

Some merchants are good about it as well, if you tell them there are problems they do sort them, but their are others who just don't want to know.

Had 1 merchant change the whole URL structure of their site 3 times in the space of a month without warning! Each time it ruined the images and links on my site. Really annoying when your trying to run a quality site and all of a sudden you find a load of your images don't appear.

Its just a frustrating situation, as there is no excuse for every merchant not to have a working feed - they have all the data themselves for their own site, so putting it in a feed and keeping it up to date shouldn't be a problem.

Thankfully as I said before, there are good merchants, its just time consuming and a case of trial and error finding them!

[KGP]

for me the issue regarding datafeeds is that the networks dont provide SKU/UPC/EAN/GTIN infornation which could make the difference and us work in a more efficient way.

of course you -or the networks- will respond that its easy to blame the networks but at least they have the "power to convince" their participating merchants

[Gringo]

The Image issue is a pain, so much so I've taken to reading the image feed where possible and using a placeholder when it's missing :eek:

Seems most feeds (on the aff nets I'm signed up to) from book stores have the isbn and hence are easy to write a comparison script. Titles seem to be fairly standard for DVD, CD's etc so they are OK too. Only a little fiddling prior to stocking the db.

Time for a little merchant PR and see if it bears fruit but I will carry on with the dream

[zinc]

Yeah good luck with this one