Data Feeds Are Soooo Last Year

[csn]

Tweet

The majority of my sites lately have been built off the back of product feeds. All good and well but I'm sure many of you are familiar with how out-of-date feeds can be, rarely including the latest products, the right prices or worst case scenario - the retailer doesn't actually have one.

There's been a lot of talk lately of duplicate content and if we are all using product feeds, that's a lot of duplicated content.

So what is the next step? I like product feeds in the sense that I can have one page and create virtual pages. One of my sites has about 40 'index' pages but about a million virtual pages, all powered by product feeds.

What I don't like about them is the fact that it seems like an uphill struggle to keep them updated.

Is the way forward going backward? Should we now be looking at creating fresh unique content again and ditching the product feeds? Or will the future be even more exciting and based around feeds?

Any ideas?

[Travel Pixel]

APIs are the way forward IMO. No need to update, no redundancy in the data and great to build useful applications for your audience.

[uklejon]

It all depends how you use the feeds. They can still be used very effectivly if you don't re-produce them verbatum.

You'll always need fresh original content - without it you'll get nowehere unless you PPC.

[Brendon]

>> There's been a lot of talk lately of duplicate content and if we are all using product feeds, that's a lot of duplicated content.

Keeps me in business though

>> So what is the next step?

The next logical step is realtime XML feeds - data feeds as they should have been all along. Pulling live data directly from the database used by the merchant themselves. That solves almost all the problems with data accuracy and freshness.

Using an XML structure also allows the possibility to allow for data deduping at multiple levels, customising feeds on a per affiliate basis with minimal effort

[getvisible]

They can still be used very effectivly if you don't re-produce them verbatum.

completely agree!

Taking a feed, using a standard template, uploading and forgetting won't get you anywhere anymore.

Taking a feed, adding it to other feeds, manipulating the content, creating your own content, adding web 2.0 features, monitoring, tweaking and using it in co-operation with unique editorial will get you everywhere.

[csn]

APIs are the way forward IMO.

Are there any merchants that offer this yet?

As far as I'm aware, CJ offer some sort of API as do AW in their new shop thing.

[morleymouse]

flat file feeds are good in the sense that you know what you have, you can build pages around them and start to get into google on niche terms and aquire traffic that way.

flat file feeds + unique content takes it a step further, product reviews, extended descriptions add to the value of the content you have and also protect you against duplicate content. Anyone using just a flat feed runs a very high risk of getting pulled on duplicate content.
Using your own content you can then as Lee said fancy it up a bit, build in your own little features, compare products etc

xml API's - real time availability, these are fantastic. Obviously it varies from sector to sector and I imagine say a gadget retailer if they had a flat feed online updated daily wouldnt really benefit from a real time xml feed, as the flat file being updated regularly would contain correct prices and product info.

However sectors that have information that changes based on variables - such as travel - live xml feeds basically give you the ability to do what you want with the data in so many different ways. (we have affiliates that quite possibly use our xml to do better things than we do IMHO)

So for now flat file feed sites seem to be doing okay, getting traffic but there are a lot of sites that are taking it a step further and doing some very clever things and I dont think we're too far off a google shuffle that recognises these a lot more, not that some aren't already recognised of course!

[Travel Pixel]

Are there any merchants that offer this yet?

As far as I'm aware, CJ offer some sort of API as do AW in their new shop thing.

Yes there are quite a few actually, especially in the travel sector where, as Dan quite rightly points out, there is data that fluctuates a lot.

Best to contact each merchant you want to work with individually to see if they have any type of live data feed setup.

[Brendon]

APIs are too open to abuse, IMO. They work fine until the wrong people find them. Then your API server melts, and eveyone using it gets shafted simultaneously. Non-ideal, I think

[morleymouse]

APIs are too open to abuse, IMO. They work fine until the wrong people find them. Then your API server melts, and eveyone using it gets shafted simultaneously. Non-ideal, I think

limited access prevents this surely? we dont just give the keys to our API to anyone

[Brendon]

>> limited access prevents this surely?

Yes, but it defeats half the purpose of an API, to allow open access to your data so people can do cool stuff with it. Openness and security are diametrically opposed, and as you increase one, you decrease the other.

That's why I prefer XML feeds, conceptually. Rather than being a "pull" mechanism (where the affiliate site has to go and take what it needs), it's a "push" one, allowing the affs to buffer data in the event of a failure. As long as your data churn rate isn't VERY high (and I accept that things like room availability are), it works much better, in theory

[morleymouse]

we have a system that is open access, for people with access.

so those with access can basically do anything with the data we have, people without access codes cant do a thing with the feed as any requests made are just rejected.

And each xml account is seperate (not just a master 'affiliate' one) so if one affiliate gets a little jumpy and makes too many requests we can slow them down so they dont slow the whole thing down.

I agree that it is a 'pull' system where you pull the results but some sites even cache the results. So if a live request fails they just revert to saved data if available. Its not 100% fool proof but its a good a fix as any, any once you've handled enoug hsearches you basically find yourself with a database of cached room types, dates, price and can start doing your own deals!

[Brendon]

>> we have a system that is open access, for people with access.

LOL, that's kind of my point... great for those in "the circle", pointless for those outside. I'm not saying it's a bad thing, it's just that "open" should mean "open", not "open, if you have a key", because that's equivalent to "closed"

>> And each xml account is seperate (not just a master 'affiliate' one) so if one affiliate gets a little jumpy and makes too many requests we can slow them down so they dont slow the whole thing down.

Good idea. Solves a lot of problems, as it also provides a method of IDing any less than helpful affs. However, again it illustrates the conflict between openness and security. If an aff who have traditionally not done that much suddenly explode, they'll find that the throttling will restrict their ability to take advantage of the traffic

>> So if a live request fails they just revert to saved data if available

Which is fine, but if your API were down for a week, say, that stale data could lead to bookings being made that couldn't be fulfilled. There really isn't a genuinely perfect solution to any of these things. Essentially the whole underlying structure of the WWW is unsuitable for any kind of commercial activity, and it's a wonder it works as well as it does

[Travel Pixel]

APIs definitely need some controls on access thats for sure.

Thats pretty easy to do though with a good affiliate program/manager.

D

[morleymouse]

all valid points

getting a key isnt that hard, just got to say / show how your going to use it and demonstrate a bit of know how. Its only a restrictive as a stanrds feed, for exampel to get our flat file datafeed from TD you need to be a TD affiliate, you need to have applied and have been accepted ont othe programme.

if someone suddenly explodes and is making a request a minute we would investigate, if its an automated script just gathering prices for different dates across all our desitnations we'd slow them down a bit and let them know. If its live traffic, people making the searches we'd let it continue and send someone to the shop to upgrade the xml server

our xml service has been going since oooo about 2004/05 and it doesnt even get xmas off never mind a week!

>> WWW is unsuitable for any kind of commercial activity, and it's a wonder it works as well as it does

thats why that guy made web 2.0 isnt it