3

I have 2 tables (projects and tasks) that both contain a name field. I want users to be able to search both tables at the same time when entering a new item. I want to rank results based on all the terms entered. A user should be able to enter text in any order he/she chooses.

For example, searching on:

office bmt

should yield these results:

PR BMT Time - Office
BMT Office - Development
BMT Office - Development
...

The following search should also work:

BMT canter

should contain this result:

Canterburry - BMT time

So partial matches need to work too.

Ideally if the user would type a small error like:

ofice bmt

The results should still appear.

I now use something like this:

where to_tsvector(projects.name || ' - ' || tasks.name) @@ to_tsquery('OFF:*&BMT:*')

I build the search string itself in the Ruby backend by splitting the user entry according to its spaces.

This works fine, however in some cases it doesn't and I believe that's because it interprets it like English and ignores some words like of, off, in, etc...

For example searching for:

off bmt

Gives results that don't contain Off at all because off is ignored completely.

Is there a way to avoid this but still have good performance and fuzzy search? I'm not keen on having to sync my PG with ElasticSearch for this.

I could do it by building a list of AND statements in the WHERE clause with LIKE '% ... %' but that would probably hurt performance and doesn't support fuzzysearch.

2 Answers 2

2

Ideally if the user would type a small error like: ofice bmt The results should still appear.

This could be very hard to do on more than a best-effort basis. If someone enters "Canter", how should the system know if they meant a shortening of Canterburry, or a misspelling of "cancer", or of "cantor", or if they really meant a horse's gait? Perhaps you can create a dictionary of common typos for your specific field? Also, without the specific knowledge that time zones are expected and common, "bmt" seems like a misspelling of, well, something.

This works fine, however in some cases it doesn't and I believe that's because it interprets it like English and ignores some words like of, off, in, etc...

Don't just believe, check and see!

select to_tsquery('english','OFF:*&BMT:*');
 to_tsquery 
------------
 'bmt':*

Yes indeed, to_tsquery does omit stop words, even with the :* thingy.

One option is to use 'simple' rather than 'english' as your configuration:

select to_tsquery('simple','OFF:*&BMT:*');
    to_tsquery     
-------------------
 'off':* & 'bmt':*

Another option is to write tsquery directly rather than processing through to_tsquery. Note that in this case, you have to lower-case it yourself:

select 'off:*&bmt:*'::tsquery;
      tsquery      
-------------------
 'off':* & 'bmt':*

Also note that if you do this with 'office:*', you will never get a match in an 'english' configuration, because 'office' in the document gets stemmed to 'offic', while no stemming occurs when you write 'office:*'::tsquery. So you could use 'simple' rather than 'english' to avoid both stemming and stop words. Or you could test each word in the query individually to see if it gets stemmed before deciding to add :* to it.

Is there a way to avoid this but still have good performance and fuzzy search? I'm not keen on having to sync my PG with ElasticSearch for this.

What do you mean by fuzzysearch? You don't seem to be using that now. You are just using prefix matching, and accidentally using stemming and stopwords. How large is your table to be searched, and what kind of performance is acceptable?

If did you use ElasticSearch, how would you then phrase your searches? If you explained how you would phrase the search in ES, maybe someone can help you do the same thing in PostgreSQL. I don't think we can take it as a given that switching to ES will just magically do the right thing.

I could do it by building a list of AND statements in the WHERE clause with LIKE '% ... %' but that would probably hurt performance and doesn't support fuzzysearch.

Have you looked into pg_trgm? It can make those types of queries quite fast. Also, LIKE '%...%' is lot more fuzzy than what you are currently doing, so I don't understand how you will lose that. pg_trgm also provides the '<->' operator which is even fuzzier, and might be your best bet. It can deal with typos fairly well when embedded in long strings, but in short strings they can really be a problem.

2
  • Thank you, a lot of valuable information in here. Switching to simple indeed solved a big part of the problem, now 'off' indeed works. With fuzzy search I meant to allow the user for small mistakes. pg_trgm looks good for this, however the data that needs to be searched is in two different tables joined together so I can't put an index to that. Maybe I need to create a new table for this and do the searches (and index) on that one and use triggers to keep it synced.
    – rept
    Nov 1, 2019 at 21:18
  • You could also use a materialized view and put an index on that. But they are not automatically refreshed in "real time", so triggers might be preferable.
    – jjanes
    Nov 2, 2019 at 0:47
1

In your case, to_tsquery() need to indicate that all words are required, you can use to_tsquery('english', 'off & bmt') and indicates a particular dictionary containing the 'off' word, listed in the link 4, below.

Some tips to use tsvector:

  1. Create a field on your table that contains all fields with terms that you want to search, this field should be the type tsvector

  2. Your search should use tsquery as you mentioned in your answer. In search, you can make some good tricks, like as follow:

    2.a. Create a rank, with ts_rank(), indicating the search priority, this indicates the priority and how much the tsquery approximates with original terms

    2.b. If you have specific words (like my case, search of chemical terms), you can create a dictionary with the commonly words used, this words can be used to extract radical or parts to compare the similarity.

    2.c. About the performance: The tsquery works very well with gin and gist indexes. I have used full text search in a table with +200k registers and the search returns in < 0.4secs.

If you need more fuzzy search in words, you can also use the fuzzy match. I used with tsquery, the levenshtein_less_equal search, using a distance of 3. The function searches words with 3 or minus letters differing from the search, for unique words is a good way to search.

  1. tsquery and tsvector: https://www.postgresql.org/docs/10/datatype-textsearch.html
  2. text search: https://www.postgresql.org/docs/10/textsearch-controls.html#TEXTSEARCH-RANKING
  3. Fuzzy: https://www.postgresql.org/docs/11/fuzzystrmatch.html#id-1.11.7.24.6
  4. Lexize: https://www.postgresql.org/docs/10/textsearch-dictionaries.html#TEXTSEARCH-SIMPLE-DICTIONARY

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.