3

I have to handle 25M rows of data that I have collected and transformed from about 50 different sources. Every source leads to about 500.000 to 600.000 rows. Each record has the same structure, regardless the source (let say: id, title, author, release_date)

For flexibility, I would prefer to create a dedicated table for each source, (then I can clear/drop data from a source and reload/upload data very quickly (using LOAD INFILE)). This way, it seems very easy to truncate the table with no risk to delete rows from other sources.

But then I don't know how to select records having the same author across the different tables, and cherry on the cake, with pagination (LIMIT keyword).

Is the only solution to store everything into a single huge table and deal with the pain of indexing/backuping a 25M+ database or is there a kind of abstract layer to virtually merge 50 tables into a virtual one.

It is probably a usual question for a dba, but I could not find any answer yet... Any help/idea much appreicated. Thx

2
  • It sounds like you're describing partitioning
    – Strawberry
    Dec 17, 2020 at 23:05
  • What you're describing is partitioning. I don't know enough about MySQL to know if that's an answer for you though. Dec 17, 2020 at 23:05

1 Answer 1

5

This might be a good spot for MySQL partitoning.

This lets you handle a big volume of data, while giving you the opportunity to run DML operations on a specific partition when needed (such as truncate, or event drop) very efficiently, and without impacting the rest of your data. Partitioning selection is also supported in LOAD DATA statements.

You can run queries across partitions as you would with a normal table, or target a specific partition when you need to (which can be done very efficiently).

In your specific use case, list partitioning seems like a relevant choice: you have a pre-defined list of sources, so you would typically have one partition per source.

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.