-2

I've got a problem with a slow MySQL query (MySQL 5+). Let's think of three tables:

customers:
- id_customer : int (PRIMARY)
- name        : varchar(255)

customers_addresses:
- id_customers_addresses : int (PRIMARY)
- id_customer : int (INDEX)
- street : varchar(255)
- zipcode : varchar(255)
- city : varchar(255)

customers_contacts:
- id_customers_contacts : int (PRIMARY)
- id_customer : int (INDEX)
- type : varchar(255)
- value : varchar(255)

Now, my goal is to gather all addresses and contact information in one query and with one row per customer. My first try was by using LEFT JOINs as some customers do not have any address and/or contact information:

SELECT customers.id_customer,
       customers.name,
       X.contact AS contact,
       Y.street,
       Y.zipcode,
       Y.city
FROM customers
LEFT JOIN
(
  SELECT
    GROUP_CONCAT( CONCAT( type, ': ', value ) SEPARATOR ', ' ) AS contact,
    id_customer
  FROM customers_contacts
  GROUP BY id_customer
) AS X
ON X.id_customer = customers.id_customer

LEFT JOIN
(
  SELECT
    GROUP_CONCAT(street SEPARATOR '<br>' ) AS street,
    GROUP_CONCAT(zipcode SEPARATOR '<br>' ) AS zipcode,
    GROUP_CONCAT(city SEPARATOR '<br>' ) AS city,
    id_customer
  FROM customers_addresses
  GROUP BY id_customer
) AS Y
ON Y.id_customer = customers.id_customer
WHERE Y.street LIKE '%Avenue%'
ORDER BY customers.name DESC
LIMIT 0, 20

This query took over 130 seconds to complete (for ~7000 entries in each table), which is far from good.

Prepending EXPLAIN EXTENDED gives:

id  select_type table               type    possible_keys   key            key_len     ref     rows    filtered    Extra
1   PRIMARY     customers           ref     name            name           3           const   4334    100.00      Using where; Using temporary; Using filesort
1   PRIMARY     <derived2>          ALL     NULL            NULL           NULL        NULL    7793    100.00
1   PRIMARY     <derived3>          ALL     NULL            NULL           NULL        NULL    8580    100.00      Using where
3   DERIVED     customers_addresses index   NULL            id_customer    5           NULL    8651    100.00
2   DERIVED     customers_contacts  index   NULL            id_customer    4           NULL    9314    100.00

I read some stackoverflow posts and the MySQL Documentation. Both saying INNER JOIN is a lot faster. I tried to replicate LEFT JOIN behaviour with INNER JOINs by using UNION ALL:

SELECT customers.id_customer,
       customers.name,
       X.contact AS contact,
       Y.street,
       Y.zipcode,
       Y.city
FROM customers
INNER JOIN
(
  SELECT
    GROUP_CONCAT( CONCAT( type, ': ', value ) SEPARATOR ', ' ) AS contact,
    id_customer
  FROM customers_contacts
  GROUP BY id_customer
  UNION ALL
  SELECT
    '' AS contact,
    id_customer
  FROM customers
  WHERE id_customer NOT IN (SELECT DISTINCT id_customer FROM customers_contacts)
) AS X
ON X.id_customer = customers.id_customer

INNER JOIN
(
  SELECT
    GROUP_CONCAT(street SEPARATOR '<br>' ) AS street,
    GROUP_CONCAT(zipcode SEPARATOR '<br>' ) AS zipcode,
    GROUP_CONCAT(city SEPARATOR '<br>' ) AS city,
    id_customer
  FROM customers_addresses
  GROUP BY id_customer
  UNION ALL
  SELECT
    '' AS street,
    '' AS zipcode,
    '' AS city,
    id_customer
  FROM customers
  WHERE id_customer NOT IN (SELECT DISTINCT id_customer FROM customers_addresses)
) AS Y
ON Y.id_customer = customers.id_customer
WHERE Y.street LIKE '%Avenue%'
ORDER BY customers.name DESC
LIMIT 0, 20

This query improved the performance by 20 seconds. But 110 seconds is still not acceptable.

Prepending EXPLAIN EXTENDED:

id   select_type         table               type           possible_keys      key      key_len ref         rows    filtered    Extra
1    PRIMARY             <derived2>          ALL            NULL               NULL     NULL    NULL        8596    100.00      Using temporary; Using filesort
1    PRIMARY             <derived5>          ALL            NULL               NULL     NULL    NULL        8604    100.00      Using join buffer
1    PRIMARY             customers           eq_ref         PRIMARY,name,name3 PRIMARY  4       Y.id_kunde  1       100.00      Using where
5    DERIVED             customers_addresses index          NULL               id_kunde 5       NULL        8651    100.00
6    UNION               customers           index          NULL               name2    767     NULL        8677    100.00      Using where; Using index
7    DEPENDENT SUBQUERY  customers_addresses index_subquery id_kunde           id_kunde 5       func        2       100.00      Using index
NULL UNION RESULT        <union5,6>          ALL            NULL               NULL     NULL    NULL        NULL    NULL
2    DERIVED             customers_contacts  index          NULL               id_kunde 4       NULL        10411   100.00
3    UNION               customers           index          NULL               name2    767     NULL        8677    100.00      Using where; Using index
4    DEPENDENT SUBQUERY  customers_contacts  index_subquery id_kunde           id_kunde 4       func        1       100.00      Using index
NULL UNION RESULT        <union2,3>          ALL            NULL               NULL     NULL    NULL        NULL    NULL

So here's my question: How to improve one of these queries and/or database tables to get a super fast response? I'm not only interested in the solution but also in strategies how to prevent such performance kills in future.

Best regards.

4
  • "This query took over 130 seconds to complete" Er, no. That query is syntactically incorrect.
    – Strawberry
    Jan 21, 2015 at 14:51
  • sorry, I misplaced the WHERE part while tidying up the query for stack overflow. should be correct now (tested)
    – Timm
    Jan 21, 2015 at 15:23
  • Why a LEFT JOIN on customers_addresses, when it has to have a street LIKE '%Avenue%'? Jan 21, 2015 at 15:49
  • @MarcusAdams This is not always the case, the WHERE-statements are auto generated by a search-routine in my underlying framework.
    – Timm
    Jan 22, 2015 at 7:51

1 Answer 1

4

As a general rule, which applies here, you could say the following:

Whenever you use a query that joins a select result (the subqueries), MySQL has to run these subqueries first, then create a table from the results. You do that twice, which means that MySQL creates 2 tables first, only to drop them after the result is done. With proper memory management for MySQL, this is done in memory. But these tables are created without an index, as MySQL cannot magically determine which index would be best for these derived tables, and because they usually are created in memory, queries on these are quite fast (not as fast as SELECTs using keys).

Then, when the two tables are conplete, MySQL has to join your original table with both, and create a third table on the fly that needs to be filtered and sorted, based on your criteria.

This is a performance killer. One of the problems your requirements have is that every customer should only result in one line. That's not how the database saves the information, and therefor you pay a price in runtime for the transformation of the data (your GROUP_CONCAT statements). I'm not 100% sure what the current MySQL database engine does with UNION statements, so I'd rather not comment on those.

Using easy INNER JOINs over the available keys, but resulting in multiple rows for a customer when multiple addresses are the result, you will find that the performance jumps ahead alot. You can easily iterate over the customers in your programming language layer, requesting addresses for one customer at a time, if you don't feel comfortable splitting a result of all customers and all associated addresses into customers on that layer.

TL;DR: Drop your requirement or live with the overhead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.