3

I have a MySQL table, named transaction, which has 5 columns, id(int), from(int), to(int), value(float), time(datetime).

And I need to calculate the accumulative user (the number of unique "from") for some specific receiver ("to") everyday.

For example:

+-----+------+-----+-------+----------------------------+
| id  | from | to  | value | time                       |
+-----+------+-----+-------+----------------------------+
| 1   |  1   | 223 |     1 | 2019-01-01 01:11:30.000000 |
| 2   |  1   | 224 |     2 | 2019-01-01 21:37:30.000000 |
| 3   |  2   |  25 |   0.1 | 2019-01-02 03:05:30.000000 |
| 4   |  2   | 223 |   0.2 | 2019-01-02 13:26:30.000000 |
| 5   |  3   |  26 |     3 | 2019-01-02 19:29:30.000000 |
| 6   |  3   | 227 |     4 | 2019-01-03 21:37:30.000000 |
| 7   |  1   | 224 |     5 | 2019-01-05 22:03:30.000000 |
| 8   |  4   | 224 |     1 | 2019-01-05 23:48:30.000000 |
| 9   |  5   | 223 |     2 | 2019-01-06 05:41:30.000000 |
| 10  |  6   |  28 |     2 | 2019-01-06 20:19:30.000000 |
+-----+------+-----+-------+----------------------------+

And the specific to is [223, 224, 227]

Then the expected result is:

2019-01-01: 1 # [1]
2019-01-02: 3 # [1, 2, 3]
2019-01-03: 3 # [1, 2, 3]
2019-01-04: 3 # [1, 2, 3]
2019-01-05: 4 # [1, 2, 3, 4]
2019-01-05: 5 # [1, 2, 3, 4, 5]

The direct way is using SQL

SELECT COUNT(DISTINCT(`From`))
FROM `transaction`
FORCE INDEX (to_time_from)
WHERE `time` < '2019-01-0X'
AND `to` IN (223, 224, 227)

But the problem is, transaction table is big (1 million per day, about 2 years), and to list is about 1000. The above SQL is very slow, even though I have created an index on [to, time, from] and force use it.

Besides, although daily transactions amount reaches about 1 million, the daily active user is only about 10,000. So I'm considering to store DAU list in No-SQL, like

2019-01-01: [1]
2019-01-02: [2, 3]
2019-01-03: [3]
2019-01-04: []
2019-01-05: [1, 4]
2019-01-05: [5]

And when given a date d, I just retrieve all the DAU list no later than d and make a union to get the accumulative user. Something like: len(set([dau_list1]+[dau_list2]+[dau_list3]...))

But I have no idea which No-SQL to use.

  1. Redis will load everything into memory, but I only need these data when I query.
  2. MongoDB
    1. it seems I need to create a collection for every date because I need to create a unique index on from. Am I right?
    2. I know I can use an array field and $addToSet operation. But it is O(n), very slow.

So, what is the proper way to make it?

2
  • 1
    What version of MySQL are you using?
    – Salman A
    Feb 4, 2019 at 6:11
  • it's mysql 5.7.20
    – Yriuns
    Feb 4, 2019 at 15:46

1 Answer 1

2
+25

In MySQL, use something like (no redis, no MongoDB):

SELECT  DATE(`time`),
        COUNT(*),
        GROUP_CONCAT(`from`)
    FROM  tbl
    WHERE  `to` IN (...)
    GROUP BY  1;    -- shorthand for "DATE(time)"

INDEX(`to`, `from`, `time`)  -- if applying to entire table
INDEX(`to`, `time`, `from`)  -- if you have `AND time ...`

Plus some formatting. (Such could be done with a messy CONCAT, or left for the application code.)

Since this seems to be a 'scaling' question, too, perhaps you need a "Summary Table" that is updated daily with the previous day's entries, thereby making the queries much faster.

CREATE TABLE Daily (
    `day` DATE NOT NULL,
    `from` ... NOT NULL,
    `to` ... NOT NULL,
    `ct` SMALLINT UNSIGNED NOT NULL,
    PRIMARY KEY(`to`, `day`, `from`)
) ENGINE=InnoDB;

and the query becomes

SELECT  `day`,
        SUM(ct),
        GROUP_CONCAT(DISTINCT `from`)
    FROM Daily
    WHERE  `to` IN (...)`
    GROUP BY `day`;

(It could help you provided CREATE TABLE and INSERTs to build a test case from.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.