5

I have about 50.000 of records to import in a Magento store. What I have already tested: The file is about 50 MB.

  • Splitted files
  • API
  • Magento Classes

Splitting the file doesn't improve the speed of the importing of the products. Api are very slow. Magento Classes are slow.

This is a snipped of code using the Magento Classes:

// Build the product
$product->setIsMassupdate(true)
        ->setExcludeUrlRewrite(true)
        ->setManufacturer($this->addManufacturers(utf8_encode($record[4])))
        ->setSku($record[3])
        ->setAttributeSetId($this->attribute_set)# 9 is for default
        ->setTypeId(Mage_Catalog_Model_Product_Type::TYPE_SIMPLE)
        ->setName(utf8_encode($record[5]))
        ->setCategoryIds($this->getCategories(array($record[0], $record[1], $record[2]))) # some cat id's,
        ->setWebsiteIDs(array(1)) # Website id, 1 is default
        ->setDescription(utf8_encode($record[6]))
        ->setShortDescription($this->shortText(utf8_encode($record[6]), 150))
        ->setPrice($price) # Set some price
        ->setSpecialPrice($special_price)
        ->setWeight($record[12])
        ->setStatus( Mage_Catalog_Model_Product_Status::STATUS_ENABLED )
        ->setVisibility(Mage_Catalog_Model_Product_Visibility::VISIBILITY_BOTH)
        ->setTaxClassId(2)     // default tax class
        ->setPixmaniaimg($record[10])
        ->setStockData(array('is_in_stock' => $inStock, 'qty' => $qty))
        ->setCreatedAt(strtotime('now'));

$product->save();     
$ID = is_numeric($productID) ? $productID : $product->getId(); 

So the above method is correct but it spends about 5 hours in order to insert only 2300 records!!

Which are the simple SQL inserts that I have to execute in the Magento DB in order to add a new product?

6
  • Hi. Have you finally worked out how to import directly to db?
    – Sol
    Commented Nov 17, 2015 at 9:29
  • Hi @Sol yes! I have created a module for my company that save all data by a direct query to the db but unfortunatelly I cannot share it. Commented Nov 17, 2015 at 10:09
  • Nice - was it hard work? I'd like to know what time and affort it takes to develop such module. Can you shortly describe what does it do?
    – Sol
    Commented Nov 17, 2015 at 12:22
  • I have "spoofed/read" the magento query when it tries to create a product and I have logged all the executed sql strings. Then I have create a batch file with simple placeholders that I use to change the data information of the product that I need to create. Commented Nov 17, 2015 at 13:09
  • Thanks. How many hours approximately did it take you to do all this stuff?
    – Sol
    Commented Nov 17, 2015 at 13:23

4 Answers 4

3

I strongly recommend that you avoid writing raw SQL at all costs, you will almost certainly spend days and days writing to map the attribute IDs and probably get it wrong. It will also bypass all the important indexing and other system updates that Magento relies on.

If speed is your issue, I suggest that you consider uRapidFlow from Unirgy. Usual disclaimers apply, I have no affiliation with Unirgy, but my observations has been that the quality of this work is excellent.

HTH, JD

5
  • 2
    I agree with you in spirit, but there is a scale (many hundreds of thousands of records) where I have not found a single in-framework solution for importing records. A small wrapper to load the IDs at the beginning of runtime can mitigate the risks you brought up and make manual import a viable solution.
    – Joe Mastey
    Commented Oct 21, 2010 at 12:52
  • 1
    I suspect those all important indexes are the biggest slowdown at these sorts of scales. Manually writing SQL in this case might be beneficial, the indexes can be rebuilt afterwards. Commented Oct 21, 2010 at 21:53
  • @Joseph, good idea to load the inserted products and "inform" Mage of their existence. @clockworkgeek - no doubt you're right, it is the indexes, but also potentially some foreign keys relationships too. Anyone who can solve this problem has the potential to make a lot of money (as I suspect Unirgy has discovered!). Commented Oct 21, 2010 at 22:55
  • 1
    Indexes are a problem, but I don't even think that they are the greatest problem. While reloading an index 50k times is clearly inefficient, loading a new object that many times (and, I suspect, loading some table metadata that many times) is completely crippling. I know that Magento has some memory leaks to compound the problem and put a cap on each page load to boot, so loading through the framework is a bit of a mess.
    – Joe Mastey
    Commented Oct 22, 2010 at 1:00
  • Hi guys, I have tested the import using the various ways and I have choosed to use a direct connection that it will be more fast than the others. Anyway I have to update it everytime when Varien change something in the core tables but the main important thing is the SPEED. The software mentioned above use a direct connection with the db! ;) Commented Oct 22, 2010 at 11:39
3

If you disable the indexer while your load runs and then re-enable and run afterwards, it should improve your load time.

$indexer = Mage::getSingleton('index/indexer');
$indexer->lockIndexer();

// ... run your processing ...

$indexer->unlockIndexer();

// Reindex everything
$processes = $indexer->getProcessesCollection();
foreach ($processes as $process)
{
    // echo 'Processing: ' . $process->getIndexerCode() . "n";
    $process->reindexEverything();
}
3
  • The reindex time remains the same. Commented Dec 6, 2011 at 21:42
  • 1
    What about the load time? Reindex time isn't going to be changed much. What this does is tell Magento to not index after every product object is saved. Commented Dec 6, 2011 at 22:18
  • Just profiled this with XHProf and the results are interesting: 15 configurable products (222 simple products) took "95,112,290 microsecs cpu time" without locking and "59,320,733 microsecs cpu time" with locking: with DB/walltime taken into account it saved over 2 minutes. Memory usage also down by around 1/3 and the number of function calls almost halved. This is pretty much a necessity. Note that both of these tests had indexing set to manual anyway (though that could be a bug of sorts).
    – tonyhb
    Commented May 30, 2012 at 22:33
2

It's very hard to create products using raw SQL queries, because Magento uses EAV pattern for storing products.

6
  • 3
    Yes it is but I'm a developer ;) Commented Oct 22, 2010 at 11:40
  • 1
    So, use debugger and investigate SQL queries for adding products :) Commented Oct 22, 2010 at 16:25
  • Ok Done! I have inserted 50000 records in 4 hours! Now the problem is the re-indexing the Magento Catalog URL Rewrites!! Commented Oct 23, 2010 at 8:17
  • @Michelangelo that's amazing, I'm looking to import 22.000 records. It would be awesome if you could share how you did it!
    – narzero
    Commented Aug 25, 2014 at 0:05
  • @Michelangelo I agree, please share. Commented Feb 16, 2015 at 10:53
1

Occasionally I've noticed bulk inserts that work by first creating a template model...

$blankProduct = Mage::getModel('catalog/product');

...then avoid the creation of the model for each record...

$newProduct = clone $blankProduct;
$newProduct->setIsMassupdate(true)
    ...
$newProduct->save();

It's slightly more efficient but probably not enough to get that massive import to a reasonable time.

1
  • 1
    It would be interesting to use the profiler on these imports. It's actually quite simple to hook into Magento's profiler, I found it helpful to track down the bottleneck on a nav renderer. My suspicion is that it is the save() that is the expensive step, rather than the getModel(), but profiling would prove it. It would be nice if you could create all your objects in a collection then commit the collection in one step, rather than each product individually... hmmm. Commented Oct 21, 2010 at 22:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.