14

Apologies if this has been covered thoroughly in the past - I've seen some related posts but haven't found anything that satisfies me with regards to this specific scenario.

I've been recently looking over a relatively simple game with around 10k players. In the game you can catch and breed pets that have certain attributes (i.e. wings, horns, manes). There's currently a table in the database that looks something like this:

-------------------------------------------------------------------------------
| pet_id | wings1 | wings1_hex | wings2 | wings2_hex | horns1 | horns1_hex | ...
-------------------------------------------------------------------------------
|      1 |      1 |     ffffff |   NULL |       NULL |      2 |     000000 | ...
|      2 |   NULL |       NULL |   NULL |       NULL |   NULL |       NULL | ...
|      3 |      2 |     ff0000 |      1 |     ffffff |      3 |     00ff00 | ...
|      4 |   NULL |       NULL |   NULL |       NULL |      1 |     0000ff | ...
etc...

The table goes on like that and currently has 100+ columns, but in general a single pet will only have around 1-8 of these attributes. A new attribute is added every 1-2 months which requires table columns to be added. The table is rarely updated and read frequently.

I've been proposing that we move to a more vertical design scheme for better flexibility as we want to start adding larger volumes of attributes in the future, i.e.:

----------------------------------------------------------------
| pet_id | attribute_id | attribute_color | attribute_position |
----------------------------------------------------------------
|      1 |            1 |          ffffff |                  1 |  
|      1 |            3 |          000000 |                  2 |  
|      3 |            2 |          ffffff |                  1 |  
|      3 |            1 |          ff0000 |                  2 |  
|      3 |            3 |          00ff00 |                  3 |  
|      4 |            3 |          0000ff |                  1 | 
etc...

The old developer has raised concerns that this will create performance issues as users very frequently search for pets with specific attributes (i.e. must have these attributes, must have at least one in this colour or position, must have > 30 attributes). Currently the search is quite fast as there are no JOINS required, but introducing a vertical table would presumably mean an additional join for every attribute searched and would also triple the number of rows or so.

The first part of my question is if anyone has any recommendations with regards to this? I'm not particularly experienced with database design or optimisation.

I've run tests for a variety of cases but they've been largely inconclusive - the times vary quite significantly for all of the queries that I ran (i.e. between half a second and 20+ seconds), so I suppose the second part of my question is whether there's a more reliable way of profiling query times than using microtime(true) in PHP.

Thanks.

2
  • Did you have appropriate indexes created on the columns that users would search on? 20+ seconds sounds VERY long for 10k players, even if they have a hundred pets each. Commented Feb 13, 2012 at 7:25
  • In the proposed new schema it is impossible to define appropriate indexes.
    – Thilo
    Commented Feb 13, 2012 at 7:40

5 Answers 5

21

This is called the Entity-Attribute-Value-Model, and relational database systems are really not suited for it at all.

To quote someone who deems it one of the five errors not to make:

So what are the benefits that are touted for EAV? Well, there are none. Since EAV tables will contain any kind of data, we have to PIVOT the data to a tabular representation, with appropriate columns, in order to make it useful. In many cases, there is middleware or client-side software that does this behind the scenes, thereby providing the illusion to the user that they are dealing with well-designed data.

EAV models have a host of problems.

Firstly, the massive amount of data is, in itself, essentially unmanageable.

Secondly, there is no possible way to define the necessary constraints -- any potential check constraints will have to include extensive hard-coding for appropriate attribute names. Since a single column holds all possible values, the datatype is usually VARCHAR(n).

Thirdly, don't even think about having any useful foreign keys.

Finally, there is the complexity and awkwardness of queries. Some folks consider it a benefit to be able to jam a variety of data into a single table when necessary -- they call it "scalable". In reality, since EAV mixes up data with metadata, it is lot more difficult to manipulate data even for simple requirements.

The solution to the EAV nightmare is simple: Analyze and research the users' needs and identify the data requirements up-front. A relational database maintains the integrity and consistency of data. It is virtually impossible to make a case for designing such a database without well-defined requirements. Period.


The table goes on like that and currently has 100+ columns, but in general a single pet will only have around 1-8 of these attributes.

That looks like a case for normalization: Break the table into multiple, for example one for horns, one for wings, all connected by foreign key to the main entity table. But do make sure that every attribute still maps to one or more columns, so that you can define constraints, data types, indexes, and so on.

7
  • +1 splitting into different tables indeed looks logical. Proper analysis should be performed to determine which tables to create.
    – Konerak
    Commented Feb 13, 2012 at 7:49
  • Hi Thilo! I see that you mention on my answer that the database is not designed to use a join, but in your answer you specifically suggest to the OP to do that. Perhaps I am misunderstanding you, or perhaps you have misunderstood me, but I would like to learn from your experience (I do not care about the SE rep, but I want to learn). What did you find inadequate about my answer? By saying "do the join" I did mean that the OP should break the table into multiple tables, and then perform a JOIN on the disparate tables when querying for his data.
    – dotancohen
    Commented Feb 13, 2012 at 7:53
  • @dotancohen: There is joins and then there is joins. Have you tried to write the equivalent query to select pet_id, wings1, wings1_hex, wings2, wings2_hex where wings1_hex != wings2_hex and wings1 =1 and wings2 = 2 order by horns1,horns2 in the schema proposed in the question (or by Muhammad Ahsan's answer)?
    – Thilo
    Commented Feb 13, 2012 at 10:31
  • No, a well-written query would only search for those attributes actually being searched for. In fact, you mention "there is joins and then there is joins", deride my suggestion for a join, then suggest a join. The "expert quote" that you quote only says that one should not use EAV, which is also what I said: instead use a join. I am trying very hard to understand you under the assumption that I might be able to learn from you, but your contradictions are difficult to follow. Where does your answer differ from mine?
    – dotancohen
    Commented Feb 13, 2012 at 10:53
  • @dotancohan: Your answer said "Do the join. The database was specifically designed to support joins for your use case". Read in the context of the question (and how else would one read it?), where the two alternatives are "big table" and "EAV with many joins", that can only mean "Go ahead, use EAV" (which is a bad suggestion) and "An RDBMS was designed to handle EAV-type joins" (which is wrong). If you were thinking about other types of joins, you need to be more specific about them, there is certainly not more explanation in your answer.
    – Thilo
    Commented Feb 13, 2012 at 11:59
0

Do the join. The database was specifically designed to support joins for your use case. If there is any doubt, then benchmark.

EDIT: A better way to profile the queries is to run the query directly in the MySQL interpretter on the CLI. It will give you the exact time that it took to run the query. The PHP microtime() function will also introduce other latencies (Apache, PHP, server resource allocation, network if connection to a remote MySQL instance, etc).

1
  • 2
    No! A relation database is not designed for this kind of "dynamic schema". Performance will be horrible, proper indexing will become impossible.
    – Thilo
    Commented Feb 13, 2012 at 7:23
0

What you are proposing is called 'normalization'. This is exactly what relational databases were made for - if you take care of your indexes, the joins will run almost as fast as if the data were in one table.

Actually, they might even go faster: instead of loading 1 table row with 100 columns, you can just load the columns you need. If a pet only has 8 attributes, you only load those 8.

5
  • 2
    Some normalization would probably be good here, but what he is proposing is something else: a completely opaque dynamic schema that will be very hard to query and impossible for DB to figure out how to access efficiently.
    – Thilo
    Commented Feb 13, 2012 at 7:32
  • Hmm, so what would you propose? A table with 100 columns of which most are NULL for each rows looks like an anti-pattern to me too...
    – Konerak
    Commented Feb 13, 2012 at 7:35
  • Yes, normalization by splitting the table into several (one for wings, one for horns) seems like it might be a good idea. But attributes must be mapped to columns, otherwise you cannot specify data types, indexes, constraints and so on.
    – Thilo
    Commented Feb 13, 2012 at 7:38
  • Several tables look like the good compromise between 'all in one big table with 100 columns per pet' and 'all in one big table with 100 rows per pet'.
    – Konerak
    Commented Feb 13, 2012 at 7:43
  • Especially since the pets without horns or claws will not have entries in those extra tables.
    – Thilo
    Commented Feb 13, 2012 at 7:44
0

This question is a very subjective. If you have the resources to update the middleware to reflect the column that has been added then, by all means, go with horizontal there is nothing safer and easier to learn than a fixed structure. One thing to remember, anytime you update a tables structure you have to update each one of its dependencies unless there is some catch-all like *, which I suggest you stay aware from unless you are just dumping data to a screen and order of columns is irrelevant.

With that said, Verticle is the way to go if you don't have all of your requirements in place or don't have the desire to update code in n number of areas. Most of the time you just need storage containers to store data. I would segregate things like numbers, dates, binary, and text in separate columns to preserve some data integrity, but there is nothing wrong with verticle storage, as long as you know how to formulate and structure queries to bring back the data in the appropriate format.

FYI, Wordpress uses verticle data storage for majority of the dynamic content it has to store for the millions of uses it has.

-3

First thing from Database point of view is that your data should be grow vertically not in horizontal way. So, adding a new column is not a good design at all. Second thing, this is very common scenario in DB design. And the way to solve this you have to create three tables. 1st is of Pets, 2nd is of Attributes and 3rd is mapping table between theres two. Here is the example:

Table 1 (Pet)
Pet_ID | Pet_Name
   1      |   Dog
   2      |   Cat

Table 2 (Attribute)
Attribute_ID | Attribute_Name
   1               |   Wings
   2               |   Eyes

Table 3 (Pet_Attribute)
Pet_ID | Attribute_ID | Attribute_Value
    1      |        1          |        0
    1      |        2          |        2

About Performance:
Pet_ID and Attribute_ID are the primary keys which are indexed (http://developer.mimer.com/documentation/html_92/Mimer_SQL_Engine_DocSet/Basic_concepts4.html), so the search is very fast. And this is the right way to sovle the problem. Hope, now it will be clear to you.

1
  • This is the Entity-Attribute-Value model exactly. Look at your Table 3.
    – user323094
    Commented Mar 15, 2012 at 21:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.