3

I am in the process of creating a website where I need to have the activity for a user (similar to your inbox in stackoverflow) stored in sql. Currently, my teammates and I are arguing over the most effective way to do this; so far, we have come up with two alternate ways to do this:

  1. Create a new table for each user and have the table name be theirusername_activity. Then when I need to get their activity (posting, being commented on, etc.) I simply get that table and see the rows in it...
    • In the end I will have a TON of tables
    • Possibly Faster
  2. Have one huge table called activity, with an extra field for their username; when I want to get their activity I simply get the rows from that table "...WHERE username=".$loggedInUser
    • Less tables, cleaner
    • (assuming I index the tables correctly, will this still be slower?)

Any alternate methods would also be appreciated

7
  • 10
    You'd be insane to try and implement (1). Jul 17, 2011 at 18:23
  • Which is what I originally thought, by my teammates make a compelling point that 2 will be unbelievably slower, and as long as our userbase doesn't enter the thousands it shouldn't be too much of a problem Jul 17, 2011 at 18:24
  • 2
    @Damien: That's putting it politely. Databases are built to have a ton of rows, why do you want to shoot yourself in the foot? How are you going to do joins on that monstrosity? @Tomas: "Unbelievably slower"? Where'd you guys get the statistic? "Enter the thousands"? Thousands is NOTHING in the world of databases. You realize it'll take you maybe 2 ms to query a table with tens or hundreds of thousands of rows if it's properly indexed?
    – mpen
    Jul 17, 2011 at 18:26
  • 1
    @Tomas - They are wrong. #2 will not be significantly slower. They don't understand how DB's work. This is a perfect situation for a tall activity table. You can also setup periodic jobs to purge/archive old data if you need to keep the DB at a certain size. Jul 17, 2011 at 18:28
  • @hamlin11, can you link to a tutorial on how they DO work? I am having some trouble conceptualizing this Jul 17, 2011 at 18:31

7 Answers 7

6

"Create a new table for each user ... In the end I will have a TON of tables"

That is never a good way to use relational databases.

SQL databases can cope perfectly well with millions of rows (and more), even on commodity hardware. As you have already mentioned, you will obviously need usable indexes to cover all the possible queries that will be performed on this table.

3

Number 1 is just plain crazy. Can you imagine going to manage it, and seeing all those tables.

Can you imagine the backup! Or the dump! That many create tables... that would be crazy.

Get you a good index, and you will have no problem sorting through records.

3

here we talk about MySQL. So why would it be faster to make separate tables?

  • query cache efficiency, each insert from one user would'nt empty the query cache for others
  • Memory & pagination, used tables would fit in buffers, unsued data would easily not be loaded there

But as everybody here said is semms quite crazy, in term of management. But in term of performances having a lot of tables will add another problem in mySQL, you'll maybe run our of file descriptors or simply wipe out your table cache.

It may be more important here to choose the right engine, like MyIsam instead of Innodb as this is an insert-only table. And as @RC said a good partitionning policy would fix the memory & pagination problem by avoiding the load of rarely used data in active memory buffers. This should be done with an intelligent application design as well, where you avoid the load of all the activity history by default, if you reduce it to recent activity and restrict the complete history table parsing to batch processes and advanced screens you'll get a nice effect with the partitionning. You can even try a user-based partitioning policy.

For the query cache efficiency, you'll have a bigger gain by using an application level cache (like memcache) with history-per-user elements saved there and by emptying it at each new insert .

1

You want the second option, and you add the userId (and possibly a seperate table for userid, username etc etc).

If you do a lookup on that id on an properly indexed field you'd only need something like log(n) steps to find your rows. This is hardly anything at all. It will be way faster, way clearer and way better then option 1. option 1 is just silly.

1

In some cases, the first option is, in spite of not being strictly "the relational way", slightly better, because it makes it simpler to shard your database across multiple servers as you grow. (Doing this is precisely what allows wordpress.com to scale to millions of blogs.)

The key is to only do this with tables that are entirely independent from a user to the next -- i.e. never queried together.

In your case, option 2 makes the most case: you'll almost certainly want to query the activity across all or some users at some point.

4
  • Isn't wordpress a Database per Blog, not a table per user? Jul 17, 2011 at 18:26
  • It's one database per server, and one prefix per blog. The prefix looks like wp_1_, wp_2_, etc. leading to wp_1_posts, wp_2_posts, etc. but only a single wp_users (on the master/network server). Jul 17, 2011 at 18:28
  • Right, but isn't the author asking about creating a table per user? Imagine a wordpress blog where each time a user registers, a new table is created to track their activity Jul 17, 2011 at 18:29
  • In the case of WP, it's not just one table being created. It's several (albeit per new site). :D But yeah, I see your point. I misread the question indeed. Jul 17, 2011 at 18:32
1

Use option 2, and not only index the username column, but partition (consider a hash partition) on that column as well. Partitioning on username will provide you some of the same benefits as the first option and allow you to keep your sanity. Partitioning and indexing the column this way will provide a very fast and efficient means of accessing data based on the username/user_key. When querying a partitioned table, the SQL Engine can immediately lop off partitions it doesn't need to scan as it can tell based off of the username value queried vs. the ability of that username to reside within a partition. (in this case only one partition could contain records tied to that user) If you have a need to shard the table across multiple servers in the future, partitioning doesn't hinder that ability.

You will also want to normalize the table by separating the username field (and any other elements in the table related to username) into its own table with a user_key. Ensure a primary key on the user_key field in the username table.

0

This majorly depends now on where you need to retrieve the values. If its a page for single user, then use first approach. If you are showing data of all users, you should use single table. Using multiple table approach is also clean but in sql if the number of records in a single table are very high, the data retrieval is very slow

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.