11

We have a database of movies and series, and as the data comes from many sources of varying reliability, we'd like to be able to do fuzzy string matching on the titles of episodes. We are using Solr for search in our application, but the default matching mechanisms operate on word levels, which is not good enough for short strings, like titles

I had used n-grams approximate matching in the past, and I was very happy to find that Lucene (and Solr) supports something this out of the box. Unfortunately, I haven't been able to configure it correctly.

I assumed that I need a special field type for this, so I added the following field-type to my schema.xml:

<fieldType 
   name="trigrams" 
   stored="true" 
   class="solr.StrField"> 
 <analyzer type="index"> 
   <tokenizer 
       class="solr.analysis.NGramTokenizerFactory" 
       minGramSize="3" 
       maxGramSize="5" 
       /> 
   <filter class="solr.LowerCaseFilterFactory"/> 
 </analyzer> 
</fieldType> 

and changed the appropriate field in the schema to:

<field name="title" type="trigrams" 
    indexed="true" stored="true" multiValued="false" /> 

However, this is not working as I expected. The query analysis looks correctly, but I don't get any results, which makes me believe that something happens at index time (ie. the title is indexed like a default string field instead of trigram field).

The query I am trying is something like

title:"guy walks into a psychiatrist office"

(with a typo or two) and it should match "Guy Walks into a Psychiatrist Office".

(I am not really sure if the query is correct.)

Moreover, I would like to be able to do something more in fact. I'd like to lowercace the string, remove all punctuation marks and spaces, remove English stopwords and THEN change the string into trigrams. However, the filters are applied only after the string has been tokenized...

Thanks in advance for your answers.

2
  • Can you post the query you are using?
    – olle
    Aug 21, 2009 at 17:05
  • I edited the question to include an example query. Aug 21, 2009 at 22:17

2 Answers 2

10

To answer to the last part of your question: solr has also an ngram filter. So you should not use the ngram tokenizer (but one like "WhitespaceTokenizer" for example), apply all pre-ngram filters and then add this one:

<filter class="solr.NGramFilterFactory" minGramSize="2" maxGramSize="3" />
4

The solution turned out to be very simple: AND was set as the default operator, and if any of the ngrams didn't match, the whole query failed. So, it was sufficient to add:

<solrQueryParser defaultOperator="OR" />

in my schema definition.

2
  • 5
    Your answer makes it seem that a posteriori the question has nothing to do with ngrams. Am I right?
    – amit kumar
    Oct 20, 2011 at 5:14
  • 2
    @RyszardSzopa OR is defiantly not the same as n-gram analysis. OR gives lots of results but generally pretty bad results.
    – Adam Gent
    Nov 19, 2012 at 0:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.