How do I install e2save datafeed to Magento cart

[WilkoWilko]

Tweet

Hi all,

As title really, I have signed up to e2saves affiliate program and received their datafeeds as below

•CSV datafeed - The full product feed CSV file is located at http://media.e2save.com/datafeeds/wl_feed_www.csv.zip
•XML Datafeed - The full product feed XML file is located at http://media.e2save.com/datafeeds/wl_feed_www.xml.zip

I would like to install these onto my magneto website, yet I have no idea how to do this, can anybody help?

I need step by step instructions?

Also another problem, is that I really want to edit there data feed, becuase I only want to upload spcific products. My website is directsmartphones.com and so really I only want to upload there smartphone products. Is there anyway of editing the data feed to do this?

[michaeldim]

I have no experience with Magento, but looking at their website I think you would need to use their Data Flow Engine : Magento - Knowledge Base - Magento Development: Introduction to Magento Dataflow - eCommerce Software for Growth

[WilkoWilko]

Thankyou for your reply,

Yes I think you are right I would need to understand magneto dataflow system. However looking at the page you listed I have no idea the how coding side works. I think its best if I pay someone to do this for me. Any ideas of where I can get someone to do this for me?

Many Thanks

Joe

[michaeldim]

I see that the Magento forums aren't much help and the documentation is sparse. I will try and have a look at the platform and see if I can work out what you are asking, it won't be till the middle of the week until I can really look at it.

If you are desperate you could outsource your job on a freelance website.

[WilkoWilko]

Yeah I have posted this question in the Magento fourms already but as of yet not had a reply. Also, as I have no experiance in coding the help documents are quite hard to understand. I came to the conclution I would not be able to do this myself. That would be great if you could look at this for me, the middle of next week is no problem.

Yeah I am thinking of posting it on elance, maybe a way forward

[michaeldim]

Ok I have come up with a solution to your second problem (editing feed for just smartphones) first as then we can work on the magento problem.

The below script opens the product feed scans through it and removes any product that does not have 'Smartphone' in the tname node. It then saves the feed to a new file called smartphones.xml and deletes the original. I have tried to keep the memory usage of the script to a minimum because the original feed is 500mb big, nonetheless on average it will take about 3-5 minutes to process the feed. Depending on how often this feed is refreshed by e2save you may want to run it as a Cronjob once or month or so.


Please can you run this on your server and make sure it is removing the correct products and is running to your satisfaction...I will then move to parsing this with Magento.

<?php
set_time_limit( 0 );

function memReport( )
  {
    print "\n-------\n";
    print "Memory Usage PHP:       " . round( memory_get_usage() / 1024, 2 ) . " kB";
    print "\n";
    $mem = `ps aux | grep php5 | head -1`;
    $mem = split( " +", $mem );
    
    print "Memory Usage Total VSZ: $mem[4] kB\n";
    print "Memory Usage Total RSS: $mem[5] kB\n";
  }

$mtime     = microtime();
$mtime     = explode( " ", $mtime );
$mtime     = $mtime[ 1 ] + $mtime[ 0 ];
$starttime = $mtime;

$file   = 'http://media.e2save.com/datafeeds/wl_feed_www.xml.zip';
$target = 'feeds'; // directory where we want xml file to download to
$zipped = 'tmp'; // temporary directory for downloaded zip files

/* Download Zip */
$client = curl_init( $file );
curl_setopt( $client, CURLOPT_RETURNTRANSFER, 1 );
$fileData = curl_exec( $client );
file_put_contents( $zipped, $fileData );

/* Extract Zip */
$zip = new ZipArchive();
$x   = $zip->open( $zipped );
if ( $x === true )
  {
    $zip->extractTo( $target );
    $filename = $zip->getNameIndex( 0 );
    $zip->close();
    unlink( $zipped ); // delete zipped file
  }

$feed = $target . '/' . $filename;

$reader = new XMLReader();
$reader->open( $feed, NULL, LIBXML_NOCDATA );

$doc = new DOMDocument();

$findme = 'Smartphone';

while ( $reader->read() )
  {
    if ( $reader->nodeType == XMLReader::ELEMENT && $reader->localName == 'product' )
      {
        $xml = simplexml_import_dom( $doc->importNode( $reader->expand(), true ) );
        
        if ( strpos( $xml->tname, $findme ) )
          {
            $new_xml .= $xml->asXML();
          }
      }
    
  }
  
file_put_contents( $target . '/smartphone.xml', $new_xml );
unlink($feed);

$mtime     = microtime();
$mtime     = explode( " ", $mtime );
$mtime     = $mtime[ 1 ] + $mtime[ 0 ];
$endtime   = $mtime;
$totaltime = ( $endtime - $starttime );
echo "This page was created in " . $totaltime . " seconds</br>";

memReport();
?>

Note: this is script may change once I look at the magento side of things.

[WilkoWilko]

Hi Michaeldim, I ow you big time if this works. I have been asking for
some quotes, form some developers and they have been pricing me some quotes
Way out of my budget. It would be amazing if this works. Thanks a lot.

However what I have learnt is that magneto only allows
a 15mb file upload and as you have said the file is 500mb. However after the file has
been scripted will it be less than 15mb.

So how do I use this script? And will it be best if
Allow you to login to my magneto admin section?

[michaeldim]

Well the new xml file after stripping out the non-smartphone products comes in at 39mb. Do you need to import all of the supplied product details or is there anything further I can strip out? I mean what do you intend to display for each product on your shopfront?

If the file size is still above 15mb, i'll have to look at splitting the file into chunks and processing it that way.

What I will do is install magneto on my local server tonight/tomorrow and have a play about with the data flow importing, so I don't think I will need access to your account. Do you intend just to display products from this feed on your site or have you got other products from elsewhere as well?

Thanks

[WilkoWilko]

Out of their smartphone I intend to resell products from the following manufactories apple, htc, blackberry, lg, samsung and nokia. I don't imagin their are any Smartphones left in the feed after scripting out the products from these manufactures? Although I may be wrong. In terms of the details I would like to display product images, product description and tariff information. Again not sure if that will reduce the file size?

As you say if it is possible to split the file, that may be best. Ok if you don't need access to my admin side that is fine. Although because the file needs uploading regually ( they advise every 24 hours) is there any chance you could show me how to use the script you have built and how to upload the file to magneto as I would have to do this regually myself right?

No their are no more data feeds I would like uploading as of yet. They may be some in the long term but not yet.

Again thanks very much for your help and time. I ow you one

[michaeldim]

No well as I say the script pulls out all the products that have 'Smartphone' in the product name and removes everything else.

Could you direct me to a link where it states it has a 15mb upload limit?

If I can get it all to work then yes I will show you how to set it to update regularly (i.e. every 24 hours).

[WilkoWilko]

Well yes all smartphone products would be great. It was actually a friend that had a go at uploading the feed for me and it was him who told me it had a 15mb limit. He said the program kept timing out. I aswell can not find a link that says it has a 15mb upload limit, so I'm not sure if this is true. However I have found this code that apparenty stops the timeout

Magento Dataflow Export/Import form the command line :: Prema Solutions

This this any help?

Yes that would be great if you could show me how to work the script and upload. Do you think it will be possible to upload this feed?

[michaeldim]

Yeh I think it is a 32mb limit if you use the manual import.

I have just installed Magento now and having a look around the documentation.

I think I may have found an easier way of importing the products by making a call to the Magento api instead of through the data flow engine. It will save a lot of the hassle of getting the current feed in the right format etc.

I will have a more detailed look at it tomorrow. At this stage I am pretty confident I can get this to work.

[WilkoWilko]

That's great if you have found an alternative way of uploading the file. Will the timeout code be off any help? Yes take aslong as you like. I can not thankyou enough.

[WilkoWilko]

I'm may have thought of a small problem. Iv been looking at the product descriptions on e2saves and i have noticed that all smartphone brands but HTC mention the word smartphone in thier description. Does this mean that the script you have made will script out all the smartphones I want but the HTC products? Is this is the case is it possible to make a second script that scripts out the phase "HTC" from the orginal datafeed. Then upload this as a second feed? This is because all htc products are smartphones. Is this a large problem? Sorry if it is

[michaeldim]

I can amend the script to search for both variants...

Right this morning I have conjured up a test script which is successfully creating products remotely using SOAP. Im working on getting the xml feed integrated now and uploading the product images etc. Might take me a couple of days to get it all working properly, Magento is a bit of ***** with all it customisations! You will need to create an attribute set called "smartphones" on your system and create some attributes for it as well; The API doesn't seem to allow it to be created remotely. I will let you know what you when the script is finished what you will need to setup.