Product Data Feed System

[paidonresults]

Tweet

More and more people have been asking when will the Paid On Results product data feed system go live. We are looking at 1st week in December for it going live, ideally we would like to have had it in place before Christmas but most important to us was that we brought out something that was good and that worked, this after all is a long term project.

We listened to all the feedback we have been given both directly and on this forum, as it’s getting closer to launch we thought we would share some of the details here. The system it’s self is working great, we finally picked a name for it today, and all what’s really left is the demo site of how it works and putting together some support information.

With the new system we are able to have ready made content units that constantly keep themselves up to date without any merchant input needed. A new one stop location for people who just like their feeds the old fashioned way but most exciting you will be able to do lots of new neat things that you may not have had the technical know how to do before.

Our product feed will require no technical experience or knowledge, any affiliate can customise a feed to fit into any web site, or enable you to create a whole web site around the product feeds. You will be able to use our feeds though JavaScript or an IFrame, or call the feed from any scripting language.

If you are using Perl, PHP or ASP, no non-standard modules or add-ons are required. With PHP you can integrate your feed with just 1 line of code. Our feeds work though a simple URL, just fetch the URL from a script and output the contents without any other work if you’re using a template.

Our template system allows total control over the design, your not limited to any layout or options, just use your own HTML to create the design and layout you want for your site.
Or if you just want a CSV or XML feed then you can. Any number of templates can be added, and more than one template can be used per feed.

If using templates, your feed is given to you in a instantly usable format, just output the contents!

You can sort, search and query the feed just by adding options to the URL - this works whatever method you use to fetch the feed. Each product image is available in 6 different sizes, and you can select which image size you want.

Our "SQL Like Queries" allows you to have more control over the products, if you want to only show products which are between £10 and £50 in price, contain the word "Football" in the product name and appear in the category "toys" then you can, just enter the query in the URL along with any other options, like sort options.

You can search your feed; just add the keyword to the URL. If you’re using a template then the results are formatted as per your template and ready to use.

Let’s say you wanted to create a web site which used products from a merchant, you wanted to show 10 products per page, using a short description, and the visitor could then click a link to go to a page with a longer description. This may sound complex, but with our product feeds it would just require you to design 2 templates, one for the listing page and one for the "more information" page. You would need to create a small script in your chosen language to fetch the right pages or use our IFrame or JavaScript method.

- Sort method, you can sort on any field within the feed
- You can show only products added after a time/date you give
- Use products from selected category or multiple categories, across different merchants
- Search products and output the results as per your template or CSV/XML
- Complete template control, no limiting features or restrictions

Coming soon.

[Barry]

It good to see a network thats committed to giving us a descent quality datafeed system.

keep up the good work!

looking forward to it

[Mogga]

Sounds great!

[Bone-Clothing]

Sounds very good indeed

I look forward to seeing it in action. Does it let affiliates know about a merchant refreshing the data? Or will merchants still have to let affiliates know by the POR message system / email?

Can't wait to try it out!

Cheers

AJ

[graeme]

Originally posted by Bone-Clothing
Sounds very good indeed

I look forward to seeing it in action. Does it let affiliates know about a merchant refreshing the data? Or will merchants still have to let affiliates know by the POR message system / email?

Once a merchant updates their feed with us, all affiliates who are using the feed on their site will have the latest product information without doing anything. Affiliates who have downloaded an CSV or XML copy of the feed will need to download it again or download an update.

Within the Affiliate Control Panel will be a "status" page which will list the time/date each merchant last updated their feed.

[Bone-Clothing]

Thanks for that graeme. Look forward to using it.

Cheers

AJ

[gary]

Sounds great! Look forward to it!

Gary

[CompareStore]

How many servers have you got to supply the feeds?
I'm sure you've thought of this but if you are allowing things like SQL "like" statements you could find the database and servers under quite heavy load so make sure you have over estimated and have enough servers with load balancing set up to handle all the requests.

I can't wait to get hold of all that lovely data!

[graeme]

Originally posted by CompareStore
How many servers have you got to supply the feeds?
I'm sure you've thought of this but if you are allowing things like SQL "like" statements you could find the database and servers under quite heavy load so make sure you have over estimated and have enough servers with load balancing set up to handle all the requests.

I can't wait to get hold of all that lovely data!

SQL Like queries is basically a way of querying data in a way you would do using SQL but they aren't real SQL queries. For example lets say you wanted all products which were between £10 and £50, plus the name contained the word football. The query you would use is

Price >= 10 and Price <= 50 and ProductName =~ football

So its like SQL, but it isn't, although over time we will extend what you can do with these queries. As for server load, we will keep a close eye on load and before its needed add another server to cope with the load and so on.

[999gi]

Sounds great, but like CompareStore it scares me how slow this might be unless you are really building a huge server farm to deal with this.

[Rich]

Sounds good, I seem to have missed this thread with the get together and stuff.

With the right database structures and query & output caches, a massive amount of requests could be handled without too much hardware and it should scale quite well, particually since the queries shouldn't be that complicated.

I'm looking forward to giving it a try.

[CompareStore]

I assume =~ is the equivalent of the mySQL "Like" statement such as "where Product like '%toy%'" ?
If so this can be quite intensive on databases.

I have had to write a caching system as the minute my main site got really busy the LIKE statements were killing it!
This might be a deficiency in mySQL but if you have a 100 like request executing at the same time you could find the database grinding to a halt.
All I'm saying is load test (if you haven't done so already) before you release it otherwise you will be panic buying servers!

Another suggestion I would have is for you to use different servers for the standard CSV & XML downloads and the "Shop Window" type functionality. The later will be much more demanding on the machines as the requests will come in everytime a page on an affiliates web site is displayed.

[Rich]

I took "SQL Like Queries" to mean queries like SQL rather than using queries with LIKE conditions - as LIKE "%something%" is an absolute database killer.

Of topic, but you could create your own index tables to try to avoid avoid having to do a LIKE "%something%" condition. At a minimum, if your product id 1 has name 'big blue widgets' you could create another table that has: -
1 | big blue widgets
1 | blue widgets
1 | widgets
Then you could create an index on this new table and do a LIKE "something%" query - LIKE checks that don't start with a % can use an index. I haven't actually needed to do this, but I expect that, even with the extra index keys to check, this would be an improvement over a %..% search. Fulltext indexes can also sometimes work out quite well.

[Supercod]

We are not going to release it until we are happy we can fully cope with the demand, I personally expect this system to generate a lot of extra income for everyone, and would be daft not to reinvest in more servers as we need them.

The systems we have in place means the product feed functionality can work over multiple servers and even different hosting locations, and the plan would be to add more capacity as we need it, and yes panic buying is something we will want to avoid.

This is a big project and one we hope will give us an edge.

[drivetowin]

Agreed that like '%x%' will put a large load on the server

I find using mysql that fulltext searches are considerably quicker than like '%x%' but the downside is that adding records in to the database is much slower - which needs some careful thinking around so you dont lock the whole table out to others doing select statements while you are updating....