Datafeed not being completely inserted into DB

[Adrian1]

Tweet

Hi,

Hopefully someone will be able to help me here,

I've got a PHP script which downloads each merchants datafeed and then loads the feeds into a MySQL database using the LOAD DATA LOCAL INFILE command

This works perfectly for most merchants, but when it comes to John Lewis for example who have 61K products, only 12K or so are inserted into the database. The same happens for other merchant feeds which have a high number of records

Does anyone have any ideas?

[Cheapdvds]

Hi,

Hopefully someone will be able to help me here,

I've got a PHP script which downloads each merchants datafeed and then loads the feeds into a MySQL database using the LOAD DATA LOCAL INFILE command

This works perfectly for most merchants, but when it comes to John Lewis for example who have 61K products, only 12K or so are inserted into the database. The same happens for other merchant feeds which have a high number of records

Does anyone have any ideas?

sounds like it maybe timing out , u could try adding the below to a .htaccess at the top

php_value max_execution_time 0

[Adrian1]

Thanks for the reply, I've just tried the code you suggested but with no success

I've also been using:
set_time_limit(0);
ini_set("memory_limit","3072M");

Got any other ideas?

[Cheapdvds]

i would assume it is timing out

the host will sometimes limit this

have you got the script to load the datafeed straight into the database all at once? you could split it up by adding a 1 or 2 second wait between every 1000 records or so as that could help

[Adrian1]

Thanks for the replies cheapdvds

I've found the problem!

It turns out that I wasn't allocating enough bytes to gzread for it to read all the data in the gzipped CSV file

Doh

[scriptmonkey]

lol, that is always the problem when unzipping on the server, you have to go large!

[babrees]

Thanks for the replies cheapdvds

I've found the problem!

It turns out that I wasn't allocating enough bytes to gzread for it to read all the data in the gzipped CSV file

Doh

hmm, I wonder if this is my problem!! But where/how do you fix this please

Cheers

[scriptmonkey]

where you call gzread, you have an object and an amount of bytes, eg.

gzread($object, 500000);

change the number by adding a 0 to the end until you are successful

[babrees]

where you call gzread, you have an object and an amount of bytes, eg.

gzread($object, 500000);

change the number by adding a 0 to the end until you are successful

hmm, thanks but I have no idea where this may be. I use Price Tapestry, but thought you may have been talking about something on the server rather than script?

[Cheapdvds]

hmm, thanks but I have no idea where this may be. I use Price Tapestry, but thought you may have been talking about something on the server rather than script?

in your .htaccess file at the top have you added anything like the below?

php_value upload_max_filesize = 0
php_value max_execution_time = 0
php_value memory_limit = 256
php_value post_max_size = 256

if u havent you could add this and keep incresing the numbers to see if it works