Product Feed Strategy for a VB.net Developer

[accelerator]

Hi All

I would like to spark some discussion about Product Feed strategy for VB.net developers, in particular, whether you think I am going in the right direction. So far, I have:

- Written a VB.net script which can parse an Awin XML file sitting on my server into my MySQL database, note I only use the following Awin fields:

Advertiser
productID
productname
brand
description
deepLink
imageURL
Price

- Written an XSLT file that can take a Tradedoubler XML file and convert it into a XML file containing the above fields (i.e. a cut down version of the Awin XML file).

- My next step is to write a script that can take the Tradedoubler XML file and insert it into my MySQL database.

The eventual aim is to automate the process using some sort of batch process, and to get it working for other networks, allowing me to automatically import the various network feeds.

I would like to hear what people think of this approach. Do you think there would be other better ways to do it, in particular:

- What batch processes could I use in VB.net?
- Should I be looking at XML Web Services instead?

All comments much appreciated.

Many thanks

Accelerator

[jonsp]

Unless you really want to use xml I'd suggest parsing the text (.csv) feed. It's always going to be smaller than xml so should be more efficient to process.

You can grab the file using a web client
using system.net

dim oWeb as new webclient
oweb.downloadfile("http://network.com/file.csv", server.mappath("/files/file.csv"))
oweb.dispose()

Once you've got the file It's an easy job to parse it vb.net . Say you have a file like this

ProductID, ProductName, Price
1, Widget, 23.99
2, Grommit , 12.99

You can parse it thus
Dim fields As String()
Dim delimiter As String = ","
Dim filename As String = Server.MapPath("~/Files/File.csv")
Using parser As New TextFieldParser(filename)
parser.SetDelimiters(delimiter)
While Not parser.EndOfData
fields = parser.ReadFields()
dim productid as string = trim(fields(0))
' grab the rest of the fields
' stick them into db
end while
end using

You'd obviously want to include code to catch bad data/missing fields etc but usually this way works out more efficient than dealing with xml files.

Originally Posted by accelerator Hi All I would like to spark some discussion about Product Feed strategy for VB.net developers, in particular, whether you think I am going in the right direction. So far, I have: - Written a VB.net script which can parse an Awin XML file sitting on my server into my MySQL database, note I only use the following Awin fields: Advertiser productID productname brand description deepLink imageURL Price - Written an XSLT file that can take a Tradedoubler XML file and convert it into a XML file containing the above fields (i.e. a cut down version of the Awin XML file). - My next step is to write a script that can take the Tradedoubler XML file and insert it into my MySQL database. The eventual aim is to automate the process using some sort of batch process, and to get it working for other networks, allowing me to automatically import the various network feeds. I would like to hear what people think of this approach. Do you think there would be other better ways to do it, in particular: - What batch processes could I use in VB.net? - Should I be looking at XML Web Services instead? All comments much appreciated. Many thanks Accelerator

[mcol]

afraid I'd disagree with jonsp.

xml is so important nowadays that the .Net functions are very heavily optimised and getting better all the time. I'd recommend you stick with xml.

Then again as an ancient programmer myself I'd do it in csv as I dislike xml on principle. Loved the code sample, textfieldparser was a new one on me.

For batch automation, what about using the windows scheduler to run a console app every hour which could check for updated files and download and process them if it found any. That's probably a very old fashioned suggestion.

Mick

[jonsp]

Originally Posted by mcol afraid I'd disagree with jonsp. xml is so important nowadays that the .Net functions are very heavily optimised and getting better all the time. I'd recommend you stick with xml.

I'd agree that xml is very important, .net has some good xml functions and one should know how to use them, with that said part of knowing how to use something is knowing when not to use it. The advantage of .csv is efficiency, a good sized xml product feed is going to weigh in around 3 times the size of the same data in .csv. To parse that file we need to pull it down from the merchant then load it into memory and read it. csv will do that quicker than xml. I don't see any advantage to xml that would outwigh the efficiency gains of csv.

As far as running this on a schedule the "correct" way would be to write a windows service and schedule it. The quick and dirty way would be use windows script host (a .vbs file) that just does an httprequest to the asp.net page and schedule it with task scheduler. One could argue this is a bit of a hack but it should get the job done.

Cheers,
Jon

[zinc]

Originally Posted by accelerator So far, I have: - Written a VB.net script which can parse an Awin XML file sitting on my server into my MySQL database, note I only use the following Awin fields:

Hi Accel

By the above you mean you have written an ASP.net application using VB Script? Or you have created a windows app using VB.net?

Zinc