My team maintains an app/database that processes millions of records each week. The process is fairly simple:
- Send Notifications to contacts for various campaigns
- Write the contact_id, campaign_id, message_id, created_at, updated_at to a log when a notification is sent
- Read the record count for each notificationID/notification_messageID and display that to the user in a report.
The writing and reading process to the log takes an exceptionally long time and we're looking for a way to optimize it.
The write statement occurs when a notification is sent. It batches the insert for 20 records in one query. Here is an example:
INSERT INTO `contact_notification_logs` (`id`, `contact_id`, `campaign_id`,
`message_id`, `created_at`, `updated_at`, `is_reset`)
VALUES
(NULL, '1', '1', '1', '2019-01-23 20:16:21', '2019-01-23 20:16:24',
'0'),
There are two read statements that occurs:
- This one is pretty simple, it runs on a page where all campaigns are listed and displays the current count of notifications sent for TODAY:
SELECT COUNT(id) FROM contact_notification_logs
WHERE DATE(created_at) = '[current date]'
That one, while simple, still takes a long time to execute.
- The second read statement is a bit more complex because it is built into a reporting tool on the app where users can specify params, but the root 'select count' is the same.
Here is an example:
SELECT COUNT(id) FROM contact_email_logs
WHERE DATE(created_at) > '2018-12-23'
AND DATE(created_at) < '2019-01-23'
AND campaign_id = 27
AND message_id = 133
A couple of extra points:
The data needs to be able to be pulled in real time. Meaning if I want to check the count for all notification campaigns at this exact point in time, I can. So the query runs to count all at that time.
The contact_notification_logs has 28,740,585 records in it.
Am I missing something obvious here that will allow us to optimize the run times for these queries?