Direct XML vs Upload to Database

[crippsy]

Tweet

Hi, Very new to this all help is most welcome.
I will be creating a price comparisom site taking various affilate feeds. Just to make sure that I am starting on a firm platform I would like to poll view/advice on what is the best way to read data feeds.
Is it best to read a XML or would it be better to schedule a batch to read into a database?
Any pointers, pro, cons, examples or advice on what your prefered solution is would be most welcome.
I look forward to your responces

[D-Mac]

I like to use cron jobs (a script that runs automatically), to import data feeds (xml or csv) into my database. This way all the data is stored on your server, and is available for fast access at the point a visitor requests the information. The downside of this method is that the data can potentially be slightly out of date. E.g. your cron job runs at 3am, and the merchant updates their data feed at 9am when the techie gets to work.

An alternative is to use an API (if the merchants you are working with have one available). The drawback to these are that the delivery of data to your site can sometimes be slowed by the other server, or by general network traffic.

Overall the easiest method is to import the data feeds nightly. Unfortunately the quality of data from some merchants is not that great, so you may need to supplement it in other ways.

[jonsp]

Hi, Very new to this all help is most welcome.
I will be creating a price comparisom site taking various affilate feeds. Just to make sure that I am starting on a firm platform I would like to poll view/advice on what is the best way to read data feeds.
Is it best to read a XML or would it be better to schedule a batch to read into a database?
Any pointers, pro, cons, examples or advice on what your prefered solution is would be most welcome.
I look forward to your responces

Personally I like to stick a csv feed direct into a database then query it as needed. This way you don't need to rely on the xml being available. The only exception would be Amazon who have a rock solid xml system

[crippsy]

Thanks, suspected that uploading data would be the prefered choice. What is your prefered method of data capture, jonsp suggests that CSV is more stable accross the merchants.

[D-Mac]

When a feed is available as CSV and XML, as is the case on most UK affiliate networks, it really doesn't matter which format you use as both with almost certainly contain exactly the same data.
Importing either into mysql is just as easy/hard as the other, so it really comes down your personal preference. CSV can be read in Excel, but is limited to about 65000 rows. XML can be read in your browser, but is perhaps slightly more difficult for a human to read.
Posted via my Mobile Device

[Pete Cheyne]

Hi,

If I had the choice, I would go for XML every time. I don’t have anything against CSV, but I’ve always been put off by its limitations and annoying idiosyncrasies.

You can be pretty sure that 99% of the time, the XML you are downloading from a network has passed various levels of validation which ensures the structure and integrity of the data. CSV doesn’t have an inherent validation process, unless there has been one specifically written in the engine which is generating the file. Therefore more often that not, when parsing and writing data from a large CSV into your database, the parser will break or you will find random bits of data which don’t relate to the column which it’s been placed in. This is usually because of poor encapsulation of the data, or a comma being present in the product description which messes with the delimitation of the data.

Also, I find XML more readable as more often than not, the data is contained within meaningful node names and due to its hierarchal nature the format of the data also lends itself to a better understanding of what the data means and what it belongs to.

Finally, I would also say that writing a parser to traverse an XML feed and write to a DB takes considerably less effort than doing the same for CSV – its about 3 lines of code in PHP thanks to its inbuilt functions (simplexml, dom, etc). If you are interested to see how easy it is, have a look at this guide which is a step by step walkthrough of how to download and parse an XML feed to a MYSQL database.

I could go on, but half of you will already be asleep by now and the CSV fan boys will be rapidly reducing my forum reputation score

If you want any help with PHP on parsing XML/CSV just let me know.

Cheers,