3

Base Match Query: Billy Sue

Test Match Query #1: Billy Sue and

Test Match Query #2: Billy and Sue

We end up with identical scores between Base and #1, but Base and #2 have similar yet different scores.

Using the analyze API, the stop word and is removed on both test queries, but the start_offset and end_offset token properties differ for Sue between the Base query and Test Query #2.

Essentially, the pre-stop-word-removal distance between the remaining tokens is recorded and has a small yet finite impact on scoring.

The Question

Is there a way to delay the calculation of the start_offset and end_offset properties of tokens until after stop-words are removed, or otherwise prevent removed stop-words from influencing scoring in any fashion?

1 Answer 1

3

Perhaps disable position increments on the stop word filterand see if that helps? Especially if your mapping has some kind of filter after the stop word filter, you'll get strange artifacts from the position increments

E.g. something like this:

"analyzer": {
   "analyzer_example":{
      "tokenizer":"standard",
      "filter":["standard", "lowercase", "filter_stop"]
    }
},
"filter": { 
   "filter_stop":{
      "type":"stop",
      "enable_position_increments":"false"
    }
}
2
  • Wow, can't believe I overlooked that option. It has been staring me in the face several times this morning as I've viewed that specific page. Thanks for the perfect answer! Jan 23, 2013 at 16:32
  • 1
    This does not work in a newer version, see elasticsearch-users.115913.n3.nabble.com/…
    – Renaud
    Feb 6, 2015 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.