3

TransactionHistory Table:

    CREATE TABLE `TransactionHistory` (
      `id` varchar(200) NOT NULL,
      `transactionType` varchar(200) DEFAULT NULL,
      `startDate` bigint(20) DEFAULT NULL,
      `completionDate` bigint(20) DEFAULT NULL,
      `userId` varchar(200) DEFAULT NULL,
      `status` varchar(200) DEFAULT NULL,
      `error_code` varchar(200) DEFAULT NULL,
      `transactioNumber` varchar(200) DEFAULT NULL,
      PRIMARY KEY (`id`),
      KEY `transactioNumber_index` (`transactioNumber`)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

User Table:

    CREATE TABLE `User` (
      `userId` varchar(200) NOT NULL,
      `name` varchar(200) DEFAULT NULL,
      PRIMARY KEY (`userId`),
      KEY `userId_index` (`userId`)
    ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Scenario:

  • Group TransactionHistory by transactioNumber
    • If groupSize == 1,
      • display value in transactionType, startDate, completionDate, status, error_code
    • If groupSize > 1
      • display '' for transactionType
      • display MIN startdate, and MAX startdate
      • for STATUS and ERROR_CODE
        • display status = SUCCESS, error_code = '0' if all status in group = SUCCESS,
        • display status = FAILED, error_code = '99' if all status in group = FAILED,
        • display status = WARNING, error_code = '-1' if mixed
    • Display name of userName (if transaction has userId)

I came up with this query:

    SELECT tx.id, 
        CASE WHEN COUNT(*) = 1 THEN transactionType ELSE '' END as transactionType,
        CASE WHEN COUNT(*) = 1 THEN status ELSE ( 
            CASE WHEN COUNT(CASE WHEN STATUS = 'SUCCESS' THEN 1 END) = 0 THEN 'FAILED' 
            WHEN COUNT(CASE WHEN STATUS = 'FAILED' THEN 1 END) = 0 THEN 'SUCCESS' 
            ELSE 'WARNING' END) END as status,
        CASE WHEN COUNT(*) = 1 THEN error_code ELSE ( 
            CASE WHEN COUNT(CASE WHEN STATUS = 'SUCCESS' THEN 1 END) = 0 THEN '99' 
            WHEN COUNT(CASE WHEN STATUS = 'FAILED' THEN 1 END) = 0 THEN '0' 
            ELSE '-1' END) END as status
        MAX(completionDate) as completionDate, 
        MIN(startDate) as startDate,
        a.userId, a.name,
        transactioNumber
    FROM TransactionHistory tx LEFT JOIN User a ON tx.userId = a.userId 
    GROUP BY transactioNumber
    LIMIT 0, 20 //pagination

However if I need to add filtering, the query takes too long to complete. I read it will be faster to put WHERE filter before GROUP BY instead of HAVING, but I cannot filter status and error_code correctly as WARNING and -1 values are only present after GROUP BY

    HAVING STATUS = 'WARNING'

Also if I need to count the total number of grouped entries, it takes too long.

My EXPLAIN shows the following

    select_type: SIMPLE
    table: tx
    type: ALL
    possible_keys: NULL
    key_len: NULL
    ref: NULL
    rows: 1140654
    Extra: Using temporary; Using filesort

    select_type: SIMPLE
    table: e
    type: eq_ref
    possible_keys: PRIMARY,id_index
    key_len: 202
    ref: db.tx.userId
    rows: 1
    Extra: Using where   

1 Answer 1

1
COUNT(CASE WHEN STATUS = 'SUCCESS' THEN 1 END)

can be shortened to

SUM(STATUS = 'SUCCESS')

These must be written in this order, and they will be executed in this order: WHERE, GROUP BY, HAVING. You correctly observed that your HAVING can't be turned into a WHERE.

Also if I need to count the total number of grouped entries, it takes too long.

I don't know what you mean -- you are using COUNT(*) several times.

Is transactioNumber in a 1:1 relationship with id? If not, then the GROUP BY is invalid.

You don't have an ORDER BY, so (technically), the LIMIT is ill-defined.

Run EXPLAIN SELECT ... to see how the optimizer is performing the query.

Here's a technique that might help -- by deferring the JOIN. First, strip all mention of User out of your query. Then, make that SELECT the subquery in:

SELECT z.id,
       z.transactionType,
       ...
       a.userId, a.name,
       z.transactioNumber
FROM ( SELECT id, 
              IF(COUNT(*) = 1, transactionType, '') as transactionType,
              ...
           FROM TransactionHistory
           GROUP BY transactioNumber
           ORDER BY transactioNumber
           LIMIT 0, 20
     ) z
LEFT JOIN User a ON z.userId = a.userId 

That way, the JOIN will occur only 20 times, not once per row in TransactionHistory.

Edit

Without a WHERE clause, the optimizer will look for an index that helps with GROUP BY. If the ORDER BY is identical to the GROUP BY, then it can do both the GROUP BY and ORDER BY simultaneously. If they differ, then the ORDER BY becomes a separate sort step.

An ORDER BY with mixed directions (such as startdate DESC, transactionType ASC) can never use an index. It is destined to need a tmp table and sort. Using startdate DESC, transactionType DESC (both DESC) is likely to work much better, without changing the semantics too much.

If the optimizer cannot use an index for both the GROUP BY and the ORDER BY, then it must gather all the rows and sort them before applying the LIMIT.

With 1140654 rows, you want to try hard to have the query and the INDEX let the optimizer make it all the way through the ORDER BY -- so that it only needs to look at 20 rows, not 1140654. My pagination blog goes into some of that.

EXPLAIN may say "using temporary, using filesort". That could be for a GROUP BY and/or an ORDER BY. However, this hides the case where it needs two sorts, one for GROUP BY, one for ORDER BY. EXPLAIN FORMAT=JSON does make it clear when multiple sorts are needed.

Still, "filesort" in not the evil. The real performance killer is needing to work with 1140654 rows instead of just 20.

1
  • Hi, thanks for the tips. What I meant when counting the rows generated by group by - Select Count(*) from (group by query without pagination). If I don't need to display TransactionHistory.id, will it still make the group by invalid? Also is it possible to speed up sorting for rows other than transactioNumber? ex. ORDER by startdate DESC, transactionType ASC . Might need to sort before applying pagination to get correct output Jul 27, 2015 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.