How long to Parse a data feed?

[rayneham]

Tweet

Hi
Ive had a new site developed by a programmer and he has set up a programme to grab the data feed and then parse this data into the correct headings and fields etc. The process seems to be taking forever, one retailer with over 300,000 products as been running for about 5 days now! im not a programmer myself so im not in a position to query this with the person who has done the site, he says there is nothing to speed this process up and I just have to be patient. Has anybody else experienced this length of time to parse a datafeed?
Thanks

[Mogga]

5 days?

I'd go as far to say (and I'm not technical) its broken.

[Tom]

Hehe 5 days No I would say that is a little bit long really. It may be that the script has been coded to go slow and be 'nice' to your server. Which is a very good idea as it can reduce the stress you're placing by inserting so many rows. But 5 days is a bit long I think! How is the script made do you know? PHP? Is it using a CSV file or XML?

I'd get back to the guy and ask for details.

[davidcragg]

I'd go as far to say that if it's taking more than a couple of minutes its broken :-)

[rayneham]

From what I can tell the site is php and the datafeed is a csv file

[Barry]

Maybe it sends an email to someone and they type the data in manually

Na 5 days sounds ages infact thats far too long considering you should be updating your datafeeds on a daily basis.

Ta

[rayneham]

Is it just that it can be done quicker but the server wont allow it unless I pay more money?

[Maxafi]

To be brutally honest, it sounds like you need to be finding yourself a new developer. No developer worth their salt would accept a relatively small task lasting 5 days and simply tell the client 'to be patient'.

Parsing 300,000 items from a feed shouldn't take more than 5 minutes, even for a badly optimised script running on a shared server.

[davidcragg]

Put it this way, it you had say a Woolworths feed, then you could use the following code to load it into a MYSQL database and it would take a couple of minutes tops.

LOAD DATA INFILE 'myfilename'
INTO TABLE mytablename
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'

regards,
David

[Tom]

To be fair to the developer here it might not be their fault. You might just have to get a host that will allow you to upload more data, quicker. It can take quite a bit of to process large files and insert into mysql especially if they are over 50mb.

Saying that I would expect 5 mins rather than 5 days

What kind of hosting have you got? Is it a shared server?

[rayneham]

Yes it is a shared server, he said that only way to make it quicker would be to move the site to its own dedicated server. "The script uses both PHP and Linux commands to download and extract a ZIP file allowing the unzipped CSV file to be processed into the MySQL database" this is apparently the way its being dealt with at the moment!

[DavidCartlidge]

"The script uses both PHP and Linux commands to download and extract a ZIP file allowing the unzipped CSV file to be processed into the MySQL database" this is apparently the way its being dealt with at the moment!

Nothing wrong with that - ask him why it takes 5 days and not 5 minutes.

[lethal0r]

doing something like that will or SHOULD only take hours not days.

its probably more your server than the coder - if they can code something like that then im sure they know what theyre doing.

[rayneham]

From what the programmer as told me the script goes through every entry checking that its not a duplicate entry, he says they used to use the code as described above and that it would work really quick but give messy results in the fields!

[tbp]

There is definately something wrong, as I mentioned in the other thread about this.

Even on a shared server, doing duplicate checking, it should not take 5 days!

Something is wrong with the code, it must be incredibly inefficient! With code there are a number of different ways of getting the same job done. All the ways "work", but some take much longer than other ways.

If you want to pm me with more details, and possibly send the file over, i`ll take a quick look and give you specific advice.