0

My team maintains an app/database that processes millions of records each week. The process is fairly simple:

  • Send Notifications to contacts for various campaigns
  • Write the contact_id, campaign_id, message_id, created_at, updated_at to a log when a notification is sent
  • Read the record count for each notificationID/notification_messageID and display that to the user in a report.

The writing and reading process to the log takes an exceptionally long time and we're looking for a way to optimize it.

The write statement occurs when a notification is sent. It batches the insert for 20 records in one query. Here is an example:

INSERT INTO `contact_notification_logs` (`id`, `contact_id`, `campaign_id`, 
`message_id`, `created_at`, `updated_at`, `is_reset`) 
VALUES 
(NULL, '1', '1', '1', '2019-01-23 20:16:21', '2019-01-23 20:16:24', 
'0'),

There are two read statements that occurs:

  1. This one is pretty simple, it runs on a page where all campaigns are listed and displays the current count of notifications sent for TODAY:
SELECT COUNT(id) FROM contact_notification_logs 
WHERE DATE(created_at) = '[current date]'

That one, while simple, still takes a long time to execute.

  1. The second read statement is a bit more complex because it is built into a reporting tool on the app where users can specify params, but the root 'select count' is the same.

Here is an example:

SELECT COUNT(id) FROM contact_email_logs 
WHERE DATE(created_at) > '2018-12-23'
AND DATE(created_at) < '2019-01-23'
AND campaign_id = 27
AND message_id = 133

A couple of extra points:

  1. The data needs to be able to be pulled in real time. Meaning if I want to check the count for all notification campaigns at this exact point in time, I can. So the query runs to count all at that time.

  2. The contact_notification_logs has 28,740,585 records in it.

Am I missing something obvious here that will allow us to optimize the run times for these queries?

2 Answers 2

0

for the first read query: Do you have an index on created_at field ?

for the second read query: Do you have an index based on three fields: created_at, campaign_id and message_id ?

If not, take a look at https://dev.mysql.com/doc/refman/5.5/en/create-index.html

1
  • Hey! Thanks for the response. We have indexes on all fields, but the date fields. Not sure why - I'm checking that.
    – BLancast
    Jan 23, 2019 at 16:09
0

Inefficient date range leads to checking too many rows

WHERE DATE(created_at) > '2018-12-23'
  AND DATE(created_at) < '2019-01-23'
  AND campaign_id = 27
  AND message_id = 133

Don't write date comparisons that way. It cannot use an index involving created_at because it is hidden in a function call (DATE()). Instead:

WHERE created_at >= '2018-12-23'
  AND created_at  < '2018-12-23' + INTERVAL 1 MONTH

If that DATE() stuff is generated by a 3rd party package, you need to abandon it.

Lack of suitable index

Then... you need a composite index:

INDEX(campaign_id, message_id,   -- in either order
      created_at)                -- after those

For simply "today"

SELECT COUNT(*) FROM contact_notification_logs 
    WHERE created_at >= '[current date]'
      AND created_at  < '[current date]' + INTERVAL 1 DAY

INDEX(created_at)  -- the previous index will not help for _this_ query

Need Summary Table

With 28M rows, you may find that my suggestions above are not sufficient. To get another 10x improvement, build and maintain a Summary Table. Suggest using days, not weeks or months as the resolution.

Others

Don't use COUNT(id) unless you need to check whether id is NULL. Instead, use the common pattern: COUNT(*).

If created_at is type DATE, the original query is one month, minus one day. If it is DATETIME, then it is missing midnight of the starting date. With my code, it works correctly regardless of the datatype.

For further discussion, please provide SHOW CREATE TABLE.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.