The Good Feed Guide (For Merchants)

[clint45]

It’s surprising just how many of the data feeds currently available on the various affiliate networks are either partially or completely unusable. I suspect that the primary reason for this state of affairs stems from a simple misunderstanding about exactly what is required of a good data feed. In case my suspicions are correct, here is my attempt to explain the differences between a “good”, useable feed and a “bad”, unusable feed. Obviously the following are just my opinions, so all of the usual disclaimers apply.

Good Feeds Need Definitive, Consistent Categorisation

It might be helpful to think of a data feed as an extension of a merchants own website. A good feed, when processed correctly by an affiliate, can result in additional, highly-targeted web pages that advertise a merchant’s content to consumers that they would not have reached otherwise.

To allow this to happen, though, a merchant’s data feed – like their own website - must be well organised and categorised. No merchant would dream of organising their website by the same broad, meaningless categories that are all too often contained within their data feeds. An online clothes shop, for example, would never simply list page after page of “Clothes and Accessories”. Similarly, an online Camera shop would never simply list page after page of mixed up cameras, tripods, cases, and batteries. So why do so many of their data feeds do this? Broadly speaking, a merchant’s data feed should categorise its products in broadly the same manner as they are categorised on their own website.

Typically, affiliates take a data feed and automatically process it into a form suitable for advertising to consumers. The automatically part means that data feeds are generally read by machines (computer programs) rather than people.

And there’s the rub. While a human being can read a description or look at a picture and easily tell what a particular product is, any automated (machine) process can’t. A “dumb” machine needs to be told exactly and unequivocally what each product is; it cannot easily discern it from a photograph or some prose in a description.

A product’s category fields, therefore, need to tell a “dumb” machine exactly what the product is. We don’t care whether this is encapsulated in a single category or by several hierarchical categories, just so long as by category alone we, and our dumb machines, can tell exactly what it is.

Sadly, in my experience, the vast majority of available data feeds fail, either completely or partially, to meet this basic requirement. I invite all merchants to take a look at their data feeds. If you can’t tell what your products are by simply looking at the category field/s, then your data feed is unusable. If you can tell what some of your products are, and not others, then your feed is partially unusable. It’s almost that simple.

E.g.

Category: Plasma Television (Very Good)
Category: Television (OK-ish)
Category: Televisions, Video, and Photography (Very Bad)

Good Feeds Need “Make” and “Model” Fields

With decent categorisation we can generally establish, for example, that a given product is a Plasma TV - so far so good. But most consumers want to know the brand of the Plasma TV they are about to buy, and a sizeable proportion of those want to know the model as well! I don’t know, maybe they read a review in a magazine or something. In any case, they are pretty stubborn about it. They don’t just want a Plasma telly; they want a Sony ZXHoojimaflip Plasma telly.

A good feed, therefore, needs both make and model fields. Again, it isn’t enough to bury this information within the prose of a description field. It also isn’t enough to combine them within a single field:

Make: Sony
Model: KXXXX7P (Very Good)

Name: Sony KXXXX7P (Bad)

Description: The amazing KXXXX7P from Sony is... (Very Bad)

Other Common Causes of Bad Data Feeds

• Mixing up Accessories for Products with the Products themselves. A wall-bracket for a Plasma TV is not a “Plasma TV”, it’s a “Plasma TV: Accessory”
• Other, more bizarre, “pollution” within categories – e.g. a Toaster within a “Plasma TV” Category
• ImageURLs that pull in images of hugely varying size – from tiny to absolutely enormous (these can wreak havoc on the layout of a Web page and are difficult to constrain)
• Missing out the additional contextual fields that are often esssential for the type of products being offered. Around 90% of Mobile Phone suppliers, for example, do not include the Monthly Tariff for their Pay-Monthly phones.

Turning a Good Feed into a Great Feed

In addition to the obligatory fields - such as category, make, model, price, description, image, and url - really good feeds also contain:

• Short Summary fields – e.g. “Plasma television with wall mount and speakers”
• “Availability” – is it in stock? How long to delivery? Etc.
• Delivery Cost
• In-store Price (if different from Web Price)
• Special Offer Text

And finally, the best data feeds are up to date data feeds. It should go without saying that a data feed needs to be as up to date as the Web data to which its URLs refer. Any stale, broken links or price discrepancies inevitably lead to lost sales for merchants and to lost commissions and credibility for affiliates. Moreover, prices tend to go down rather than up. If your data feed lags those of your competitors you are going to appear overpriced on the growing number of sites that offer comparisons.

Data Feed Problems Specific to Specific Affiliate Networks

Buy.at
Good Points: Probably the best feeds of the bunch. BAs feeds have an excellent field structure that actively encourages good categorisation as well as offering lots of scope for the inclusion of additional information.

Bad Points: There aren’t a lot of these feeds available. Also, despite their good field structure, BA has one or two really good examples of really badly categorised feeds. Successfully extracting more than handful of meaningful product details from their John Lewis feed, for example, remains one of the Grand Challenges to modern science.

Affiliate Window

Good Points: Huge number and variety of well maintained (up to date) feeds. Plus numerous other nice touches, such as the provision of a time stamp for all of their feeds, so that stale ones can be easily identified and avoided.

Bad Points: Over-simplistic feed structure that actively discourages adequate categorisation and the inclusion of essential information. AW’s field structure doesn’t even have a natural placeholder for “model” information, for example. The end result is that AW contains by far the largest proportion of “unusable” data feeds.

TradeDoubler

Good Points: Good scope for categorisation and a good provision of fields for the level of detail required (e.g. make, model, etc.)

Bad Points: Poor reliability. Many of TD’s feeds update sporadically while many others don’t update at all. When they do update, many contain broken links, and many more are just empty files. TD has no mechanism for monitoring or detecting broken feeds – they seem to rely entirely on affiliates reporting problems. Once reported, problems tend to either be solved very quickly, or else not at all.



Right. I’ve run out of steam. I hope that this overly looong posting is received in the manner in which it was intended: as a small step in helping to raise the data feed quality bar. From my experience with US data feeds; it seems to me that the UK is far ahead of the game. It would be a shame to squander this lead.

[Ste]

Nice post and a topic which is bought up every now and then, but it seems that you want everything done for you. You are missing one vital point. Not everyone wants a feed in the way you describe. I use feeds on most of my sites and if they do not suit my site, I make them suit or find another feed.

Building a website is not just about grabbing a feed and slapping it in a database and making sales, although this does work, but for a good site, you need to tweak them to your requirements.

Personally I like the feeds which have multiple Category names such as,

Category: Televisions, Video, and Photography

They fit my site exactly. Affiliate marketing is about making money at the end of the day and if you use data feeds then good, but don’t think for one minute that everyone else uses feeds the same way as yourself. As it stands I know a couple of people who use feeds, but change the content within to suit each website. A long winded way to do it, but it is worth it for the results and in some cases only using a number of the products.

The only sites which will benefit from having a fully categorised and product model feed would be the price comparison websites, and if the feeds were changed, we would all have one price comparison site as they would be very easy to create.

On a lighter note, I agree on the images problem though.

[clint45]

Ste,

I'm afraid it is misconceptions/misunderstandings like those you express that have allowed the overall quality of data feeds to remain so low for so long.

Your assumption that a good feed somehow does all the work for you, is a long way from the truth. I don't/didn't suggest, for example, that all feeds conform to any kind of "standard" categorisation - that would be impossible in my view. I just ask that they be self-consistsent and meaningful. This being the case, how do you imagine that simply slapping a bunch of these disparate feeds into a database would somehow automatically result in a comparison site?

More importantly, you do not seem to have grasped the most fundamental aspect of data (information). While, you can easily ignore information that is there, you cannot create information that isn't there. So, if "Televisions, Video, and Photography" is to your liking, great: You can simply combine all of these categories to get exactly what you want/need. Without more specific categories, however, those of us that require a finer granularity of information are stuck.

In summary, if all merchant feeds conformed broadly along the lines that I suggest in my post, then wouldn't we all be happy? We would have what we need and you would have what you need. I don't understand what would motivate you to try and put a stop to this call for more and better information? Why on earth would you argue that less information is better than more?

PS: "automatically" doesn't mean "easily" or "without significant work creating the automatic process".

[drivetowin]

Clint

I dont think anyone would argue that less information is better than more but I think some of your comments re datafeeds are very subjective - i.e. a feed may be 'unusable' to you but there are several of us on here who have been using feeds for the last few years (and believe me if you think some feeds are basic now you should have seen what we had to cope with 2-3 years ago - feeds now are brilliant compared to then and thanks should go to all of the networks and merchants for making them so) - who make great use of them each and every day - its not easy, its a pain in the butt sometimes and yes sometimes the only option is to manually go through feeds and categorise them and edit descriptions etc yourself - but they are implicitly not 'unusable' - the only feeds I would say are genuinely 'unusable' are those where the urls in the feed no longer exist on the merchant site.

The danger for affiliates I think is that if we make the task of producing feeds too onerous for the merchant, then the merchant simply will not bother - after all most of the feeds are produced automatically from the merchants stock systems - if the categories were there I'm pretty sure they would include them but if they're not a merchant is not going to pay someone to sit down and do it manually - its not worth it to them for them to do so. Personally given a choice between feeds with no category, brand or product fields or no feed at all, I'd take the former every day - yes it means work for me but whoever said life was meant to be easy.

I for one actually quite like the fact that feeds are not a walk in the park to work with, it cuts down on the competition - the more people who look at it and think 'no thats too much like hard work' - the better as far as I'm concerned.

So yes, come on merchants, give us all the feeds you can, and give us the best information you can, but whatever you can give us is better than nothing.

[Ste]

Clint, I am not at all trying to put a stop to the generalisation of datafeeds at all, but making them so useable will only have one effect, every man and his dog will be slapping up datafeed websites if they are fully categorised.

Keith has hit the nail on the head with this statement

I for one actually quite like the fact that feeds are not a walk in the park to work with, it cuts down on the competition - the more people who look at it and think 'no thats too much like hard work' - the better as far as I'm concerned.

Re the category situation. What is stopping you adding your own extra categories to the feeds or re-categorising the feed? I have been doing it for nearly 2 years!

Look at the larger picture, datafeeds are not simply there to be used as is, they are a marketing tool provided by Merchants to assist in getting you and them sales. Datafeeds differ across every network and merchant just gotta get on with life and the less people use them the better for people who do.

The US are going the right way at the moment with feeds, well a couple of merchants I use are, they will only provide you with a feed is you change/modify it at their discretion. Life as an affiliate is all too easy and anyone can create a site from a datafeed but from what I grasp of your posts, you want all merchants to provide all the same category system and fields so you can just automate the whole process into your website. Google would have a field day with dupe content.

Anyone who is thinking about getting into datafeed driven sites is best doing it the hard way by changing the feed to suit themselves.

Having said that, I am all for a generalised feed layout and if you search this board you will find its been a hot topic for years. I just don’t want it to come anytime soon!

[Gavinio]

From a merchant's perspective, there is nothing more annoying than developing a swanky feed, only for no-one to use it!

Would anyone like to show off any particularly good uses of feeds on a website of their own or someone elses? It'd be good to show merchants what they can do.

Keith - it's a good point about having any sort of feed rather than none!

[Supercod]

While Paid On Results data feeds are (not yet) perfect , we have been listening and improving things a great deal. For example we have a choice of different image sizes for every product over all Merchants. You will never get broken images on Paid On Results feeds, as the images come from the Network, and should a product not have an image you get a "no image" image in its place. As for the categories, we use the Merchants own categories as we feel they suit a product better than shoehorning them into categories we have created, however, we are looking at ways to create a network-wide category list as an option for affiliates to use instead of the merchants categories.

Remember we do have the same issue as everyone else that we have to work inside certain limitations from the Merchants own feeds, but if you ever tried the Merchants feed and then tried our version of it, in 9 out of 10 cases ours is way better as a stating point for Affiliates to use.

[clint45]

We currently integrate more than 80 disparate data feeds on an ongoing, daily basis. Any kind of manual intervention - beyond an initial configuration - is therefore out of the question.

I've said it twice before (see above), but I'll say it again: I am not suggesting any kind of "standardised" categorisation. I don't mind if one merchant categorises a television as a "Donut" while another calls it a "Fridge-Magnet"; all I ask is that they (a) categorise and (b) be consistent within each feed.

Some of you seem to be sugegsting that the current state of affairs is good news. You suggest that the fact that many feeds are as bad as they are serves as a useful barrier to entry for other affiliates, thus reducing competition. Any merchants who stumble upon this thread will hopefully notice that this is of course bad news from their perspective. Which, I guess, was my point in the first place.

Incidentally, an abundance of good, but disparate, data feeds does not make it easy to provide a comparison service. Rather, it makes it possible for those with the know-how.

[haggul]

In defence of the merchants (spending some of my time that side of the fence) they are often forced to crunch their expansive and descriptive categorisation down to suit the aff scheme's predefined topography - although I have to admit the PPC engines are the worst for this - making you put everything from screwdrivers to lathes in the "Tools" section!!

We have spent many a man hour developing the tools (and then using them) to recategorise the feeds - still a massive work in progress on half a million products.

[drivetowin]

We currently integrate more than 80 disparate data feeds on an ongoing, daily basis. Any kind of manual intervention - beyond an initial configuration - is therefore out of the question.

I integrate more than 450 disparate data feeds on an ongoing daily basis - and manual intervention is most definitely not out of the question - you just have to think about how you (or others you pay to do it for you) go about it - I'm not going to give any secrets of how its achieved here but you do have to think 'outside of the box'

I dont quite get your argument on categories - if they're not standardised how are they going to help - if one person puts plasma tvs under ''crumpets" and another puts them under "donuts" and another puts them under "sausages" and yet another puts them under "televisions" - then to me thats just as much work to sort out as if they weren't categorised in the first place.

Yes I do want barriers to entry but I dont see that being against merchants interests - from the merchants perspective you've got to do an awful lot of extra sales to justify spending £15K a year on someone to sort the data out for you. (and for a merchant with lets say 100,000+ products it could easily be £30K a year for two people)

[clint45]

Let me get this straight...

You're advising me to "think outside the box" in order to come up with a way of manually fixing 450 datafeeds with just a small army of employees?

And then you suggest:

I dont quite get your argument on categories - if they're not standardised how are they going to help - if one person puts plasma tvs under ''crumpets" and another puts them under "donuts" and another puts them under "sausages" and yet another puts them under "televisions" - then to me thats just as much work to sort out as if they weren't categorised in the first place.

You really do need to try thinking outside the box.

Anyway. I've had enough of this malarky. It's an embarassment really (all of this blinkered, narrow-minded, and misdirected bickering in a thread that was designed for merchants to read). I appologise for ever opening this can of worms in the first place.

In conclusion, I guess we get the data feeds we deserve.

[drivetowin]

Let me get this straight...
You're advising me to "think outside the box" in order to come up with a way of manually fixing 450 datafeeds with just a small army of employees?

Yep, that's what I said - and it can be done - most of the time there's only me but it can still be done (and its not just me I know of at least one other affiliate on this forum who does the same - and he's only got his Dad to help him) - you just need to work smarter not harder

You really do need to try thinking outside the box.

I do - thats why I'm good at what I do - but dont just take my word for it - ask anyone who's attended one of my seminars or had a one to one discussion with me

Anyway. I've had enough of this malarky. It's an embarassment really (all of this blinkered, narrow-minded, and misdirected bickering in a thread that was designed for merchants to read).

Sorry if we've embarassed you - I've never been accused of being blinkered, narrow minded or misdirected before - so thank you for making today unique.

[pricethat]

We currently import EVERY data feed from every network with the exception of Affiliate future and some POR feeds.

This puts us up in the same kind of levels as keith and or maybe i might say a little more (seperate datas from private listings).

I have contributed to many discussions about feeds in the past, whilst i myself welcome anything that has to be said that might help it has been so many times the same old dog had been brought up, why not just take an old post and make it a sticky.

In some of my last posts i really took the time to explain to people why categorisation and other things just wouldnt happen, but am kind of suffering from cant be bothered syndrome at the moment. Categorisation or classification of products simply wont happen on a standardised basis. I know this as a network manager and also as a merchant and affiliate. I can see why and i can see there is jack diddly that is going to change that.

However as keith points out EAN numbers or common identifiers of products are a must, but even then that doesnt really help loads as i can probably total at least 500,000 product in my database that probably dont have an EAN number, (cheap import gifts, novelties, etc).

One single common thing everyone can do though is to strip HTML from feeds, but even then when a lot of data comes straight from the website listings then i do think its the job of the networks to strip the HTML anyway.

Images again are a must and networks can simply employ the GD library or better to standardise sizes, childs work really in the scale of things.

Pricing information in the description as well as title.

STUPID TITLES OF PRODUCTS need to be reviewed.


To cut this short, i have had to develop updaters so sophisticated to handle all the errors, repair data, process it and convert it that it puts even amazon webservices to shame. With that in mind networks should take some of the burden, if it took me 2 years to develop what i would call one of the most advanced updaters just to get to what i call minimum standard data then how long has it been that i have been away from the common task of making you money.

As many have said, i dont care, stops a bunch of muppets from playing with something they have no real clue about but for those with a genuine heart and a will to try i think its a shame that networks put affiliates through this **** every day. Its not the merchants faults to be honest, its the networks for not educating the merchants as to whats needed. Its the networks customer, not ours, we in turn are the networks customers so wake up networks.

I offered many times to share the information from my log files to each network detailing every little fault, what lines, general overview stats as well as every conceivable little detail you can think of. Pretty much snubbed because an affiliate is basically showing the network the errors of their ways, the only network that wanted to look was Webgains, fair play to them.

I even offered to syndicate the merchants list of datafeeds to other sources like this forum to stop the perpetual merchants with datafeeds posts, it would of provided affiliates with 24/7 information about feeds, feed quality, how many products in the feed were fit for use, it even gave merchants the ability to have automatic reports sent to their email addresses or through RSS.

The end of the day there is too few people who really give a stuff, many easy words, but nothing that really comes to light at the end of the day. The same people that read the same posts that have been made on the subject then start off new threads about the same subject after not having done anything themselves about it.

Step 1:
Syndicate active data files and reports to a common resource, maybe linked to from the forum so all can use it

Step 2:
Send affiliate networks reports and log files forcing them to contact and resolve THEIR merchants problems

Step 3:
Email merchants with report and log files forcing them to sort it to get us out of their hair.

Other options? I think a league table of affiliat data quality in percentages would be a great start, i suspect a few networks would not like the outcome and that would be enough to embarass them into sorting the problem out.

Can someone in this forum start a more practicle, efficient way of starting things off, perhaps someone wants to help with the league tables.

Sorry if the post sounds offish, im just rambling and havent the matches to keep my eyelids awake, i hope you get my point, whatever it is i was trying to make.

[Itchy]

I've never been accused of being blinkered, narrow minded or misdirected before - so thank you for making today unique.

If you send me a list of what you have been called then i can make everyday unique for you


As for the feed thing yes i agree

[averasolutions]

This is a great thread!

You know, it is so frustrating to the developers of price comparison websites when merchants/retailers send us feeds that are like your example above!

Name: "The excellent SONY XYZ123 TV".

The problem is that some of these merchants, PC World included, don't have a seperate field for product code.

They are getting better as affiliate networks are giving them a good kick in the backside.

Maybe some time in the future, we might just get a realistic feed.

I'll tel you what would be really good, a feed that contains the EAN barcode!

Comet are the only company who seem to do so!