1

I currently have Zend_Search_Lucene set up as the search engine on the project I am working on.

It is working great at the default level (ie. searching all fields), however I have the need to now search a specific field.

The reason for this is because I am attempting to code in the capability of dealing with misspellings. I am therefore adding the soundex of each of the words in the document title.

For example:

$productArray['title'] = 'June Monthly Meat Box';  
$doc = new Zend_Search_Lucene_Document();  
$doc->addField(Zend_Search_Lucene_Field::text('product_title', $productArray['title']));  
$soundex = implode(' ',  array_map('soundex', array_map('trim', preg_split('/ /', $productArray['title'], NULL, PREG_SPLIT_NO_EMPTY))));  
$doc->addField(Zend_Search_Lucene_Field::keyword('soundex', $soundex));  
$index->addDocument($doc);

This adds 'J500 M534 M300 B200' as the soundex field.

This is how the search is performed:

$queryString = trim(urldecode($this->_request->getParam('q')));  
$words = array_map('trim', preg_split('/ /', $queryString, NULL, PREG_SPLIT_NO_EMPTY));    

$query = new Zend_Search_Lucene_Search_Query_Boolean();  
$subquery1 = new Zend_Search_Lucene_Search_Query_MultiTerm();  
foreach($words as $word) 
{  
    $subquery1->addTerm(new Zend_Search_Lucene_Index_Term($word));  
}  

$subquery2 = new Zend_Search_Lucene_Search_Query_MultiTerm();  
foreach($words as $word)
{  
        $subquery2->addTerm(new Zend_Search_Lucene_Index_Term(strtolower(soundex($word)), 'soundex'));  
}  
$query->addSubquery($subquery1);  
$query->addSubquery($subquery2);  

The variable $subquery1 stores each of the words of the original query (This works on it's own)
The variable $subquery2 stores the soundex of each word. The plan is to search the field for the soundex as well as the other fields for each word. Therefore if someone misspelt 'meat' with 'maet', it would return the result as the soundex would be the same at 'M300'.

I am using Luke to view the dataset and am seeing the correct terms. When I use Luke to search for the soundex (ie. soundex:M300) it returns no results, however if I search the entire field (ie. soundex:"J500 M534 M300 B200") it returns the correct document.

What is going wrong to prevent it searching within the field?

1 Answer 1

0

If I understand Zend_Search_Lucene_Field::keyword correctly (what you used for "soundex" above), it is designed to store a single value at a time (like a single date or a single URL).

I think for the "soundex" field you want to instead use a tokenizing storage method like Zend_Search_Lucene_Field::text, as it sounds like you want to search on individual tokens in the "soundex" field, not just the whole field value.

2
  • Hi Mark, Thanks for the info. I originally had it as a text field but changed it. The reason for the change was because Luke displayed the top ranking terms for the soundex field as one letter (such as: b, s, r, etc). Therefore if I search soundex:m does bring up all the results where there is a soundex MXXX. What strikes me as strange is that there are no numbers in the terms. Are numbers ignored for tokenized fields?
    – Mark Smith
    May 8, 2012 at 14:12
  • Sorted it, I had to change the default analyzer as I've learnt it doesn't treat numbers as part of terms. I added Zend_Search_Lucene_Analysis_Analyzer::setDefault( new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive() ); before the index was created and before the index was searched.
    – Mark Smith
    May 8, 2012 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.