13

Magento Enterprise. 1.10.1.1. The data set of customers and addresses are semi-large (125k+) CSRs are often on this grid (sometimes 25+ concurrent users at a time).

Here is the code snippet were the collection is being generated in the Customer Grid.php Block file. Nothing fancy or out of ordinary, simply adding attributes to the collection mainly.

$collection = Mage::getResourceModel('customer/customer_collection')
    ->addNameToSelect()
    ->addAttributeToSelect('email')
    ->addAttributeToSelect('group_id')
    ->addAttributeToSelect('prod_codes')
    ->addAttributeToSelect('last_called_date')
    ->addAttributeToSelect('time_zone')
    ->addAttributeToSelect('salesrep')
    ->addAttributeToSelect('do_not_call')
    ->addAttributeToSelect('club_member')
    ->addAttributeToSelect('call_back_date')
    ->addAttributeToSelect('marketing_code_outcome')
    ->joinAttribute('billing_postcode', 'customer_address/postcode', 'default_billing', null, 'left')
    ->joinAttribute('billing_city', 'customer_address/city', 'default_billing', null, 'left')
    ->joinAttribute('billing_telephone', 'customer_address/telephone', 'default_billing', null, 'left')
    ->joinAttribute('billing_region', 'customer_address/region', 'default_billing', null, 'left');

$this->setCollection($collection);

Generates this query, which is misbehaving causing very long load times while in the customer grid:

SELECT 
    e . *,
    _table_prefix.value AS prefix,
    _table_firstname.value AS firstname,
    _table_middlename.value AS middlename,
    _table_lastname.value AS lastname,
    _table_suffix.value AS suffix,
    CONCAT(IF(_table_prefix.value IS NOT NULL AND _table_prefix.value != '',
                CONCAT(TRIM(_table_prefix.value), ' '),
                ''),
            TRIM(_table_firstname.value),
            IF(_table_middlename.value IS NOT NULL AND _table_middlename.value != '',
                CONCAT(' ', TRIM(_table_middlename.value)),
                ''),
            ' ',
            TRIM(_table_lastname.value),
            IF(_table_suffix.value IS NOT NULL AND _table_suffix.value != '',
                CONCAT(' ', TRIM(_table_suffix.value)),
                '')) AS name,
    _table_default_billing.value AS default_billing,
    _table_billing_postcode.value AS billing_postcode,
    _table_billing_city.value AS billing_city,
    _table_billing_telephone.value AS billing_telephone,
    _table_billing_region.value AS billing_region
FROM
    customer_entity AS e
        LEFT JOIN
    customer_entity_varchar AS _table_prefix ON (_table_prefix.entity_id = e.entity_id) AND (_table_prefix.attribute_id = '4')
        LEFT JOIN
    customer_entity_varchar AS _table_firstname ON (_table_firstname.entity_id = e.entity_id) AND (_table_firstname.attribute_id = '5')
        LEFT JOIN
    customer_entity_varchar AS _table_middlename ON (_table_middlename.entity_id = e.entity_id) AND (_table_middlename.attribute_id = '6')
        LEFT JOIN
    customer_entity_varchar AS _table_lastname ON (_table_lastname.entity_id = e.entity_id) AND (_table_lastname.attribute_id = '7')
        LEFT JOIN
    customer_entity_varchar AS _table_suffix ON (_table_suffix.entity_id = e.entity_id) AND (_table_suffix.attribute_id = '8')
        LEFT JOIN
    customer_entity_int AS _table_default_billing ON (_table_default_billing.entity_id = e.entity_id) AND (_table_default_billing.attribute_id = '13')
        LEFT JOIN
    customer_address_entity_varchar AS _table_billing_postcode ON (_table_billing_postcode.entity_id = _table_default_billing.value) AND (_table_billing_postcode.attribute_id = '29')
        LEFT JOIN
    customer_address_entity_varchar AS _table_billing_city ON (_table_billing_city.entity_id = _table_default_billing.value) AND (_table_billing_city.attribute_id = '25')
        LEFT JOIN
    customer_address_entity_varchar AS _table_billing_telephone ON (_table_billing_telephone.entity_id = _table_default_billing.value) AND (_table_billing_telephone.attribute_id = '30')
        LEFT JOIN
    customer_address_entity_varchar AS _table_billing_region ON (_table_billing_region.entity_id = _table_default_billing.value) AND (_table_billing_region.attribute_id = '27')
WHERE
    (e.entity_type_id = '1')
ORDER BY CONCAT(IF(_table_prefix.value IS NOT NULL AND _table_prefix.value != '',
            CONCAT(TRIM(_table_prefix.value), ' '),
            ''),
        TRIM(_table_firstname.value),
        IF(_table_middlename.value IS NOT NULL AND _table_middlename.value != '',
            CONCAT(' ', TRIM(_table_middlename.value)),
            ''),
        ' ',
        TRIM(_table_lastname.value),
        IF(_table_suffix.value IS NOT NULL AND _table_suffix.value != '',
            CONCAT(' ', TRIM(_table_suffix.value)),
            '')) desc
LIMIT 20 OFFSET 60

an EXPLAIN on the query shows, NOTE the Extra on table e, Using temporary and Using filesort:

*************************** 1. row ***************************
           id: 1
  select_type: SIMPLE
        table: e
         type: ref
possible_keys: IDX_ENTITY_TYPE
          key: IDX_ENTITY_TYPE
      key_len: 2
          ref: const
         rows: 55556
        Extra: Using temporary; Using filesort
*************************** 2. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_prefix
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_VARCHAR_ATTRIBUTE,FK_CUSTOMER_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod.e.entity_id,const
         rows: 1
        Extra:
*************************** 3. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_firstname
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_VARCHAR_ATTRIBUTE,FK_CUSTOMER_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod.e.entity_id,const
         rows: 1
        Extra:
*************************** 4. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_middlename
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_VARCHAR_ATTRIBUTE,FK_CUSTOMER_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod.e.entity_id,const
         rows: 1
        Extra:
*************************** 5. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_lastname
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_VARCHAR_ATTRIBUTE,FK_CUSTOMER_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod.e.entity_id,const
         rows: 1
        Extra:
*************************** 6. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_suffix
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_VARCHAR_ATTRIBUTE,FK_CUSTOMER_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod.e.entity_id,const
         rows: 1
        Extra:
*************************** 7. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_default_billing
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_INT_ATTRIBUTE,FK_CUSTOMER_INT_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod.e.entity_id,const
         rows: 1
        Extra:
*************************** 8. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_billing_postcode
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_ADDRESS_VARCHAR_ATTRIBUTE,FK_CUSTOMER_ADDRESS_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod._table_default_billing.value,const
         rows: 1
        Extra:
*************************** 9. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_billing_city
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_ADDRESS_VARCHAR_ATTRIBUTE,FK_CUSTOMER_ADDRESS_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod._table_default_billing.value,const
         rows: 1
        Extra:
*************************** 10. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_billing_telephone
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_ADDRESS_VARCHAR_ATTRIBUTE,FK_CUSTOMER_ADDRESS_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod._table_default_billing.value,const
         rows: 1
        Extra:
*************************** 11. row ***************************
           id: 1
  select_type: SIMPLE
        table: _table_billing_region
         type: eq_ref
possible_keys: IDX_ATTRIBUTE_VALUE,FK_CUSTOMER_ADDRESS_VARCHAR_ATTRIBUTE,FK_CUSTOMER_ADDRESS_VARCHAR_ENTITY,IDX_VALUE
          key: IDX_ATTRIBUTE_VALUE
      key_len: 6
          ref: prod._table_default_billing.value,const
         rows: 1
        Extra:
11 rows in set (0.00 sec)

No indexes have been modified other than the default ones from Magento themselves for 1.10.1 See Structure of 1.5.1 (CE) here: http://www.magereverse.com/index/magento-sql-structure/version/1-5-1-0

Here is the alias table referred to AS e. on the scan:

CREATE TABLE `customer_entity` (
    `entity_id` INT(10) UNSIGNED NOT NULL AUTO_INCREMENT,
    `entity_type_id` SMALLINT(8) UNSIGNED NOT NULL DEFAULT '0',
    `attribute_set_id` SMALLINT(5) UNSIGNED NOT NULL DEFAULT '0',
    `website_id` SMALLINT(5) UNSIGNED NULL DEFAULT NULL,
    `email` VARCHAR(255) NOT NULL DEFAULT '',
    `group_id` SMALLINT(3) UNSIGNED NOT NULL DEFAULT '0',
    `increment_id` VARCHAR(50) NOT NULL DEFAULT '',
    `store_id` SMALLINT(5) UNSIGNED NULL DEFAULT '0',
    `created_at` DATETIME NOT NULL DEFAULT '0000-00-00 00:00:00',
    `updated_at` DATETIME NOT NULL DEFAULT '0000-00-00 00:00:00',
    `is_active` TINYINT(1) UNSIGNED NOT NULL DEFAULT '1',
    PRIMARY KEY (`entity_id`),
    INDEX `FK_CUSTOMER_ENTITY_STORE` (`store_id`),
    INDEX `IDX_ENTITY_TYPE` (`entity_type_id`),
    INDEX `IDX_AUTH` (`email`, `website_id`),
    INDEX `FK_CUSTOMER_WEBSITE` (`website_id`),
    CONSTRAINT `FK_CUSTOMER_ENTITY_STORE` FOREIGN KEY (`store_id`) REFERENCES `core_store` (`store_id`) ON UPDATE CASCADE ON DELETE SET NULL,
    CONSTRAINT `FK_CUSTOMER_WEBSITE` FOREIGN KEY (`website_id`) REFERENCES `core_website` (`website_id`) ON UPDATE CASCADE ON DELETE SET NULL
)

So the question is how can I get this query to perform better and not cause the temporary tables being created and scanned.

I'm not exactly sure what I can index to improve this queries performance, and I don't want to dig too much into modifying Magento's ORM.

5
  • 1
    If you remove the order by clause, what does the explain return? EDIT: I ran it. Removing the order by solves the issue. In this case you are sorting by the prefix. Is there an index on that? Look at the indexes as they compare to the name. Might not be easy to fix though. Nov 7, 2012 at 18:26
  • 1
    Not sure this can be easily fixed with the current EAV model. Changing the ORDER By to a simple "_table_firstname.value" causes the same table scan. Best solution is creating flat tables and changing the grid to search on that. Best... not easiest. Nov 7, 2012 at 18:34
  • Thanks for the response Tim. Yea, I noticed that as well, except there are some other similar queries that do not have the order by clause but still getting slow table scans.
    – B00MER
    Nov 7, 2012 at 20:14
  • 1
    The main thing that will slow down the queries is the add joins, if there is another method you could use, try it, it should greatly improve performance as ADD JOIN is a massive drain on the MySQL performance. (don't quote me this is from what my supervisor was saying to me yesterday)
    – Steve P
    Nov 16, 2012 at 14:49
  • @A_Wheel_Monkey, thanks for the response. Doing such would mean I would have to modify Magento's ORM.
    – B00MER
    Nov 19, 2012 at 16:24

3 Answers 3

1

I'm not familiar with Magento Enterprise, but from MySQL point of view I'd look for a way to replace this

ORDER BY CONCAT(IF(_table_prefix.value IS NOT NULL AND _table_prefix.value != '',
            CONCAT(TRIM(_table_prefix.value), ' '),
            ''),
        TRIM(_table_firstname.value),
        IF(_table_middlename.value IS NOT NULL AND _table_middlename.value != '',
            CONCAT(' ', TRIM(_table_middlename.value)),
            ''),
        ' ',
        TRIM(_table_lastname.value),
        IF(_table_suffix.value IS NOT NULL AND _table_suffix.value != '',
            CONCAT(' ', TRIM(_table_suffix.value)),
            '')) desc

with pre-calculated value of name. This will remove double calculation.

0

I am not remembering the horrible Magento table structure much anymore, but if it is possible you can speed up such things like this:

Select only the primary ID of each entry you need (doing only the required joins for row elimination and ordering). Then, do that big join and value selection with just the former IDs. In this case it looks like you can eliminate all the address joins, which should be quite a win.

Having less raw data to sort and later less in the big join you'll have quite a speed up. In other systems I already made >100sec queries run in <3 like this, good luck!

0

Not sure if this is true in your case, but...

If you are linking tables using _table_prefix.attribute_id and it is an integer, then remove the quotes, or mysql will convert all the values in the integer field to string before comparing. This would cause a filesort.

AND (_table_prefix.attribute_id = '4')

...would change to...

AND (_table_prefix.attribute_id = 4)

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.