2

I'l trying to figure out what I should do to index my keywords that contains "." .

ex : this.name

I want to index the terms : this and name in my index.

I use the StandardAnalyser. I try to extends the WhitespaceTokensizer or extends TokenFilter, but I'm not sure if I'm in the right direction.

if I use the StandardAnalyser, I'll obtain "this.name" as a keyword, and that's not what I want, but the analyser do the rest correctly for me.

2
  • do you have email names, website addresses, ip addresses, kind of things in your text that you don't want to be tokenized?
    – phanin
    Mar 5, 2013 at 22:25
  • oups, I forgot about that. Hummm, nice catch. I think, I'll try with the answer suggested (LetterTokenizer), but I still want to know how to split a token into multiple part, and index them all . Is not to hard to exclude a word, but didn't find how to keep the words that I want.. another example could be : lastName suppose that I want to index : last and Name. I was looking arund OffSetAttribute but didn't understand what to do with it. Mar 6, 2013 at 1:22

4 Answers 4

7

You can put a CharFilter in front of StandardTokenizer that converts periods and underscores to spaces. MappingCharFilter will work.

Here's MappingCharFilter added to a stripped-down StandardAnalyzer (see the original 4.1 version here):

import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.charfilter.MappingCharFilter;
import org.apache.lucene.analysis.charfilter.NormalizeCharMap;
import org.apache.lucene.analysis.core.LowerCaseFilter;
import org.apache.lucene.analysis.core.StopAnalyzer;
import org.apache.lucene.analysis.core.StopFilter;
import org.apache.lucene.analysis.standard.StandardFilter;
import org.apache.lucene.analysis.standard.StandardTokenizer;
import org.apache.lucene.analysis.util.StopwordAnalyzerBase;
import org.apache.lucene.util.Version;

import java.io.IOException;
import java.io.Reader;

public final class MyAnalyzer extends StopwordAnalyzerBase {
  private int maxTokenLength = 255;
  public MyAnalyzer() {
    super(Version.LUCENE_41, StopAnalyzer.ENGLISH_STOP_WORDS_SET);
  }

  @Override
  protected TokenStreamComponents createComponents
      (final String fieldName, final Reader reader) {
    final StandardTokenizer src = new StandardTokenizer(matchVersion, reader);
    src.setMaxTokenLength(maxTokenLength);
    TokenStream tok = new StandardFilter(matchVersion, src);
    tok = new LowerCaseFilter(matchVersion, tok);
    tok = new StopFilter(matchVersion, tok, stopwords);
    return new TokenStreamComponents(src, tok) {
      @Override
      protected void setReader(final Reader reader) throws IOException {
        src.setMaxTokenLength(MyAnalyzer.this.maxTokenLength);
        super.setReader(reader);
      }
    };
  }

  @Override
  protected Reader initReader(String fieldName, Reader reader) {
    NormalizeCharMap.Builder builder = new NormalizeCharMap.Builder();
    builder.add(".", " ");
    builder.add("_", " ");
    NormalizeCharMap normMap = builder.build();
    return new MappingCharFilter(normMap, reader);
  }
}

Here's a quick test to demonstrate it works:

import org.apache.lucene.analysis.Analyzer;
import org.apache.lucene.analysis.BaseTokenStreamTestCase;

public class TestMyAnalyzer extends BaseTokenStreamTestCase {
  private Analyzer analyzer = new MyAnalyzer();

  public void testPeriods() throws Exception {
    BaseTokenStreamTestCase.assertAnalyzesTo
    (analyzer, 
     "this.name; here.i.am; sentences ... end with periods.",
     new String[] { "name", "here", "i", "am", "sentences", "end", "periods" } );
  }

  public void testUnderscores() throws Exception {
    BaseTokenStreamTestCase.assertAnalyzesTo
        (analyzer,
         "some_underscore_term _and____ stuff that is_not in it",
         new String[] { "some", "underscore", "term", "stuff" } );
  }
}
2
  • Thanks, that's what I needed. If by curiosity, you know how to split manually a word in an Filter/Analyse, Tokenizer, just post it please. Suppose that I want to split words (I know that's stupid, but for educational purpose, it's perfect). How to create a Filter that will index words like that : thisIsFourWords : that will index : this Is Four Words. Mar 6, 2013 at 14:02
  • look like WordDelimiterFilter could do that. I'll try to understand how it works internaly. Mar 6, 2013 at 14:53
0

If I understand you correctly, you need to use a tokenizer that removes dots -- that is, any name that contains a dot should be split at that point ("here.i.am" becomes "here" + "i" + "am").

1
  • yes, that's right. I could extends the needs to "_" too. like : last_name Mar 6, 2013 at 2:07
0

you are getting caught by behavior documented here:

However, a dot that's not followed by whitespace is considered part of a token.

StandardTokenizer introduces some more complex to parsing rules than you may not be looking for. This one, in particular, is intended to prevent tokenization of URLs, IPs, idenifiers, etc. A simpler implementation might suit your needs, like LetterTokenizer.

If that doesn't really suit your needs (and it might well turn out to be throwing the baby out with the bathwater), then you may need to modify StandardTokenizer yourself, which is explicitly encouraged by the Lucene docs:

Many applications have specific tokenizer needs. If this tokenizer does not suit your application, please consider copying this source code directory to your project and maintaining your own grammar-based tokenizer.

2
  • thanks for LetterTokenizer. I missed that one in the source code :( I suppose that will do the job for my current test. but I hope to understand how to split myself a keyword. Suppose that I want to create a custom StandardAnalyser, I didn't understand how to split a word, do I have to parse the document char by char ? Mar 6, 2013 at 1:25
  • The StandardTokenizer is built using Jflex. If you need a full-featured grammar-based tokenizer, you might consider using it as well. I think an in-depth discussion of how to define your grammar might be a touch beyond the scope of this question.
    – femtoRgon
    Mar 6, 2013 at 8:41
0

Sebastien Dionne: I didn't understand how to split a word, do I have to parse the document char by char ?

Sebastien Dionne: I still want to know how to split a token into multiple part, and index them all


You may have to write a custom analyzer.

Analyzer is a combination of Tokenizer and possibly a chain of TokenFilter instances.

Tokenizer : Takes in the input text passed by you probably as a java.io.Reader. It
JUST breakdowns the text. Doesn't alter, just breaks it down.

TokenFilter : Takes in the token emitted by Tokenizer, adds / removes / alters tokens and emits the same one by one until all are finished.

If it replaces a token with multiple tokens based on requirements, buffers all, emits them one by one to the Indexer.

You may check following resource, unfortunately, you may have to sign-up for a trial membership.

By writing a custom analyzer, you can breakdown the text the way you want to. You may even use some existing components like LowercaseFilter. Fortunately, it is achievable with Lucene to come up with some Analyzer that serves your purpose if you couldn't find that as a built-in or on the web.

" Writing Custom Filters: Lucene in Action 2"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.