4

My mongodb is rather simple: a dataset/entry has around 30 properties on 3 layers. One such entry has up to around 5000 characters. I have 500k of them. When I execute the following query...

db.images.find({ "featureData.cedd": { $exists: false}}).count()

...it is extremely slow. It's not indexed, but still.. from my MySQL experience it shouldn't take 20 minutes to execute one such query.

While being executed (directly on the mongo terminal) there's 3% CPU usage and still over 2 Gigs of free memory.

Thanks for giving me a hint on what I could do!

EDIT: An explain() of the query (without count) gives:

db.images.find({ "featureData.cedd": { $exists: false }}).explain()
{
    "cursor" : "BasicCursor",
    "nscanned" : 532537,
    "nscannedObjects" : 532537,
    "n" : 438,
    "millis" : 1170403,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "isMultiKey" : false,
    "indexOnly" : false,
    "indexBounds" : {

    }
}

Output of iostat:

Linux 3.2.0-58-generic (campartex)      03/25/2014      _x86_64_        (2 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          34.93    0.01    0.25    0.48    0.00   64.33

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda               2.08       103.79        11.26  172805914   18749067
fd0               0.00         0.00         0.00        148          0

Output of explain() after adding an index:

db.images.find({ "featureData.cedd": { $exists: false }}).explain()
{
    "cursor" : "BtreeCursor featureData.cedd_1",
    "nscanned" : 438,
    "nscannedObjects" : 438,
    "n" : 438,
    "millis" : 2,
    "nYields" : 0,
    "nChunkSkips" : 0,
    "isMultiKey" : true,
    "indexOnly" : false,
    "indexBounds" : {
            "featureData.cedd" : [
                    [
                            null,
                            null
                    ]
            ]
    }
}
10
  • 1
    explaining the query is always a good first step
    – shx2
    Mar 25, 2014 at 13:25
  • Btw please consider before writing stuff like "for counting mongo has to go through all tupels" that we're talking about 15 minutes here for only half a million records.. :)
    – Doidel
    Mar 25, 2014 at 13:25
  • I could only find an "explain()", no describe. And that "explain" runs for 1 minute now x/ Same low CPU and memory usage
    – Doidel
    Mar 25, 2014 at 13:29
  • explain() hasn't terminated yet. I'll write here once it has ^^
    – Doidel
    Mar 25, 2014 at 13:48
  • 1
    Disk IO limited probably. What does iostat say?
    – Mzzl
    Mar 25, 2014 at 15:00

2 Answers 2

6

TL;DR: Reverse the logic: add a sparse index on a new field has_cedd that is either null or some constant (low selectivity index, not ideal, but improved through sparse), or better yet, keep a global counter somewhere else that is updated on every write operation.

Indexing featureData.cedd sounds like a bad idea if it can contain up to 5k chars, because that is well beyond the maximum index size and apparently you're not interested in the data itself, only whether it's present.

Oh, and why is this slow? Probably to keep ad-hoc requests fast. MongoDB could dedicate all the resources to this OLAP-like query, but that would introduce lag on any 'regular OLTP-style queries'.


There are two problems here:

  1. $exists : false is evil, and I doubt indexing will help: Indexes are made for data, while $exists is a 'meta-query' on the structure. It can use an index if there is one on the field and the query is $exists : true, because if an indexed value exists, the field itself must also exist on a given document. Reversing that logic is tricky: if the field doesn't exist, it's not in the index or it has super low selectivity. 'Turning around' indexes is generally problematic, that is also true for queries using $ne by the way.

  2. MongoDB will have to de-serialize 500k objects and inspect each one to perform the $exists. You can't compare this to MySQL where you have a fixed table structure, in fact, $exists : false doesn't have a SQL-equivalent, because the field MUST exist, otherwise your table is badly broken.

7
  • Thanks for explaining! Still I'm wondering though why only 3% CPU is used for the OLAP query? Can I somehow increase that? Since nothing else is running on the DB that would be quite the waste of resources...
    – Doidel
    Mar 26, 2014 at 7:59
  • MAYDAY adding an index brought it from 20 minutes to 2 Miliseconds! wtf! I added the new explain() as edit to the question
    – Doidel
    Mar 26, 2014 at 8:14
  • Sooo I guess the index created a seperate list of those which do not have such a feature to index, and therefore the search only has to scan through the exact amount of results, 438 in this case?
    – Doidel
    Mar 26, 2014 at 8:25
  • 1
    ah, interesting. $exists:false looks for null values - that only works efficiently if the other objects don't have cedd:null very often (as a valid value), but that seems to be the case. Those objects are probably a lot smaller because the cedd field is so large, correct? Otherwise, the query should be faster 'only' by a factor of thousand. What data type do you use as a primary key?
    – mnemosyn
    Mar 26, 2014 at 10:06
  • int. But I don't query by id in this case, just to say. But yes, those without the cedd features tend to be a lot smaller. I'm surprised about the factor too, it improved by a factor of 600'000 ^^
    – Doidel
    Mar 26, 2014 at 10:13
0

In my case adding an index improved the query speed by a factor of 600'000. $exists:false looks for null values - that only works efficiently if the other objects don't have cedd:null very often (as a valid value). That is the case here. Further, the objects that have no cedd value are a lot smaller.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.