Getting XML files automatically

accelerator · 13-09-06

Hi Guys

I am looking at trying to automatically get product feed XML files onto my web server, then parse them into a MySql database. I have a Windows server with asp.net and php available.

My question is, what are the techniques available for getting the xml files onto the webserver. I know a lot of the LAMP boys would use a cron job, but how could I do it with my setup?

Please let me know any thoughts / suggestions.

Thanks

Accelerator

CompareStore · 13-09-06

If you want to do it all on the server you can use XML HTTP within ASP scripts to retrieve the feed and the parse it directly into the database. The trouble with this approach is you need to have some way to automatically restart from where you left off if a script fails or times out.

Alternatively, and it's the way I do it, you can us a separate PC to take the load off the server and use a program (such as one written in Visual Basic) to download the feeds and manipulate them locally then use myODBC to update the server.

esnagel · 13-09-06

I use XML feeds on 2 of my sites - one stores them in the database, the other simply displays to the user.

When storing in the database, I check the feeds using a cron (a scheduled task should do the trick, or see http://www.smarterscripts.com/cron/). When displaying, I load up the page then, using AJAX, grab the XML feeds, parse the results and display them where I want the data to be shown.

So far, when saving to a database, the only problem I've found is someone posts an article, I grab it, then they edit the title and I grab another copy of it, so I have to manually delete it. It's a beta site that I think only I use, so no big deal.

With comments, my cron script is only 45 lines of code, and that fetches the RSS list from my database, gets each XML file, parses it, and stores the new entries in the database.

See http://scratchprojects.com/2006/07/r..._part1_p01.php to get the code that fetches & nicely parses the XML feeds (rss_fetch.inc).

Eric

accelerator · 14-09-06

Thanks for your replies guys. Yes, doing it all on the server would be nice, but probably a bit unrealistic to expect it's going to work reliably. If I was to download the feeds locally, can anyone recommend ways to parse the feeds? Is it best to write a vb program, or is there a good third party XML editor/parser out there that could be used?

Many thanks

Accelerator

pricethat · 14-09-06

for what you want by the sounds of it dmorissons pricetapestry will probably do the job written in php, just ignore the front end part of the script if you dont want to use it.

I would not fancy my chances downloading large files though...

dmorison · 14-09-06

Hi,

Yes - a number of people use Price Tapestry just for the backend (the database and admin area). Once all the merchants/products are in the products table you can simply query it any way you like to generate, for example, a single merchant datafeed site that isn't part of a price comparison website.

I've written some comprehensive notes regarding product feed automation on the following page - even if you're not using Price Tapestry 80% of the information will still be useful in terms of how to create a shell script to conveniently download, unzip and save feeds where you want them...!

http://www.pricetapestry.com/node/198

Hope this helps!

kevquinlan · 14-09-06

i can wholeheartedly agree that PriceTapestry - see DMorrison's site above is the best way i have found to automate scripts (at least for us smaller / medium size players)

Support of his scripts are second to none (especially considering the tiny amount you pay for the MagicParser Program)

check out the forum on his website if you want to check out the help and support in advance.

(and no he hasnt paid me to say this you cynical lot!)