2

I have some issues getting my indexes right. I am trying to query a(big 80Mb) document that has a lot of "map" attributes like

<oplage version="0.2" xmlns="http://www.nvsp.nl/oplage-mapping">
<meta-data>
<!--Generated by DIKW for NetwerkVSP STTip-->
<dateCreated>2014-04-03 13:23:16.885124</dateCreated>
</meta-data>
<map ppc6_id="1001WE" wijk_id="">
   <bruto>0</bruto>
   <stickers>0</stickers>
   <netto>0</netto>
</map>

Question 1 is actually: do i need to split up this doc? It is 80MB in size i needed to increase in-memory list sizes. I read somewhere that having large documents in memory is not a good idea, in general. This document holds a n:m relationship between two types of objects "ppc6" objects and "wijk" objects. I need to get a good performance on my 'aggregation' function that finds all ppc6 objects that together make up a 'wijk' object. Typically there are around 500.000 ppc6 objects and 40.000 'wijk'objects.

I have made a fragement root for this document on the map element.

Element range index on map element.

Attribute range index on ppc6_id and wijk_id like

scalartype is string
parent namespace uri : "http://www.nvsp.nl/oplage-mapping"
parent local name : map
namespace uri equal to parent namespace (can this ever be *not* the same???)
localname :  wijk_id,ppc6_id (not sure how to add more then one here?)

my query is like:

xquery version "1.0-ml";
declare namespace html = "http://www.w3.org/1999/xhtml";
declare namespace op = "http://www.nvsp.nl/oplage-mapping";

let $d := '/data/map/oplage-mapping.xml'
let $ids := fn:doc($d)//op:map/@wijk_id
let $u := fn:distinct-values($ids)
let $id2 := cts:element-attribute-values(xs:QName("map"),
                           xs:QName("wijk_id"),
                           "*")

return (fn:count($ids),fn:count($u))

We first went down the xpath routebut this does not perform, we need cts power... so we need indexes...

The query gives me an error like:

XDMP-ELEMATTRRIDXNOTFOUND: cts:element-attribute-values(fn:QName("", "map"), fn:QName("", "wijk_id"), "*") -- No string element-attribute range index for fn:QName("", "map") fn:QName("", "wijk_id") http://marklogic.com/collation/
on line 8
expr: cts:element-attribute-values(fn:QName("", "map"), fn:QName("", "wijk_id"), "*")

index not found: string element-attribute range index for ... no clue where to go next?

I cannot find that much documentation/ working examples on setting up specific range indexes...

2 Answers 2

3

Hugo:

A few best practices for MarkLogic are:

  • To treat a document as the equivalent of a relational row, not as the equivalent of a relational table.
  • To denormalize where possible instead of aggregating on request.

From the description above, I'm wondering whether it might be possible to store each wijk object in a separate document and to store the ppc6 object for a wijk object in the wijk document.

If that's not feasible, another approach would be to store each wijk and ppc6 object in its own document, to put a range index on the foreign key for the wijk in the ppc6 documents, and to aggregate by

  1. querying for the wijk objects
  2. putting the wijk objects in a map
  3. extracting the primary keys of the wijk objects as a sequence of values
  4. using the primary key sequence as the value list for a range query on the foreign key in ppc6 documents.

Hoping that helps,

Erik Hennum

1
  • Hi Erik, thank you for your thoughts, this is exactly how I am trying to approach this. But not an answer to my main question. hugo Apr 8, 2014 at 12:19
0

Should you split up the 80-MB document? Yes, probably. As mentioned MarkLogic wants documents to act like rows, not tables. An alternative design would be to create a map:map item and store that as a document, but that would be an unusual approach and I'm not sure if it's really suitable. There would be significant extra latency in each query to load up the map, for example.

What's wrong with your range index? Namespaces. Your attribute is in the empty namespace, not the parent element namespace. Default element namespace declarations (xmlns="...") do not apply to attributes.

Also when you call cts:element-attribute-values you need to supply the right namespace for the parent QName. And drop the '*' parameter: that's for cts:element-attribute-value-match, which matches wildcards against a lexicon. If you want all the values it's more efficient to call cts:element-attribute-value with empty sequence.

Finally, look into http://docs.marklogic.com/cts:value-co-occurrences and its map option. That may be exactly what you need.

1
  • 1
    Quotes burned me in this case...Indeed my problem was a namespace problem. I had added quotes in the definition on the console. Also correct in your answer that attributes have no namespace so we can leave that blank. I did not need an element range index on map. The fragment root also does not like quotes. The element/parent does need a namespace so my query now is cts:element-attribute-values(xs:QName("op:map"),xs:QName("wijk_id"),"") , blazing fast. Apr 8, 2014 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.