3

I got the following situation

class M(db.Model):

    a = db.ReferenceProperty(A)

    x = db.ReferenceProperty(X)
    y = db.ReferenceProperty(Y)
    z = db.ReferenceProperty(Z)

    items = db.StringListProperty()

    date = db.DateTimeProperty()

I want to make queries that filter on (a), (x, y or z) and (items), ordered by date i.e.

mm = M.all().filter('a =', a1).filter('x =', x1).filter('items =', i).order('-date')

There will never be a query with filter on x and y at the same time, for example.

So, my questions are:

1) How many (and which) indexes should I create?

2) How many 'strings' can I add on items? (I'd like to add in the order of thousands)

3) How many index records will I have on a single "M" if there are 1000 items?

I don't quite yet understand this index stuff, and is killing me. Your help will be very appreciated :)

3 Answers 3

2

This article explains indexes/exploding indexes quite well, and it actually fits your example: https://developers.google.com/appengine/docs/python/datastore/queries#Big_Entities_and_Exploding_Indexes

Your biggest issue will be the fact that you will probably run into the 5000 indexes per entity limit with thousands of items. If you take an index for a, x, items (1000 items), date: |a||x||items|*|date| == 1*1*1000*1 == 1000.

If you have 5001 entries in items, the put() will fail with the appropriate exception.

From the example you provided, whether you filter on x, y or anything else seems irrelevant, as there is only 1 of that property, and therefore you do not run the chance of an exploding index. 1*1 == 1.

Now, if you had two list properties, you would want to make sure that they are indexed separately, otherwise you'd get an exploding index. For example, if you had 2 list properties with 100 items each, that would produce 100*100 indexes, unless you split them, which would then result in only 200 (assuming all other properties were non-lists).

4
  • OP does not have multiple list properties in the same compound index, so he does not have exploding indexes problem. May 13, 2012 at 19:13
  • Right, I specifically called out "if you had". The exploding index was provided for illustration and clarification purposes.
    – Sologoub
    May 14, 2012 at 1:14
  • thanks for your answer, I think that Peter's is more helpful given the question, but this was also very informative. THanks!
    – fceruti
    May 14, 2012 at 1:23
  • 1
    you are welcome, though I disagree on the response - you don't need 3 compound indexes, just need one that includes all properties - that would save you writes. Try it.
    – Sologoub
    May 14, 2012 at 1:43
1
  1. For the criteria you have given, you only need to create three compound indexes: a,x,items,-date, a,y,items,-date, a,z,items,-date. Note that a list property creates an index entry for each property in the list.

  2. There is a limit of total 5000 index entries per entity. If you only have three compound indexes, than it's 5000/3 = 1666 (capped at 1000 for a single list property).

  3. In the case of three compound indexes only, 3*1000 = 3000.

NOTE: the above assumes you do not have built-in indexes per property (= properties are save as unindexed). Otherwise you need to account for built-in indexes at 2N, where N in number of single properties (2 is for asc, desc). In your case this would be 2*(5 + no_items), since items is a list property and every entry creates an index entry.

1
1

See also https://developers.google.com/appengine/articles/indexselection, which describes App Engine's (relatively recent) improved query planning capabilities. Basically, you can reduce the number of index entries needed to: (number of filters + 1) * (number of orders).
Though, as the article discusses, there can be reasons that you might still use compound indexes-- essentially, there is a time/space tradeoff.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.