12

I'm using pg_search for some text searching within my model. Among other attributes, I have an url field.

Unfortuantelly Postgres doesn't seem to identify / and . as word separators, therefore I cannot search within the url.

Example: searching for test in http://test.com yields no results.

Is there a way to fix this problem, perhaps using another gem or some inline SQL ?

1
  • 1
    as a work-around you could have another column/attribute with the url and recognized word separators, and search against that instead
    – AJcodez
    Jan 2, 2013 at 3:49

3 Answers 3

5
+150

As stated in the documentation (and noticed by AJcodez), there is a solution in creating a dedicated column for tsvector index. Then define a trigger that catches insertions to index urls properly:

CREATE test_url (url varchar NOT NULL, url_tsvector tsvector NOT NULL);

This method will transorm any non alpha characters into single space and turn the string into a tsvector:

CREATE OR REPLACE FUNCTION generate_url_tsvector(varchar) 
RETURNS tsvector 
LANGUAGE sql 
AS $_$
    SELECT to_tsvector(regexp_replace($1, '[^\w]+', ' ', 'gi'));
$_$;

Now create a trigger that calls this function:

CREATE OR REPLACE FUNCTION before_insert_test_url()
RETURNS TRIGGER
LANGUAGE plpgsql AS $_$
BEGIN;
  NEW.url_tsvector := generate_url_tsvector(NEW.url); 

  RETURN NEW;
END;
$_$
;

CREATE TRIGGER before_insert_test_url_trig 
BEFORE INSERT ON test_url 
FOR EACH ROW EXECUTE PROCEDURE before_insert_test_url();

Now, when url are inserted, the `url_tsvectorè field will be automatically populated.

INSERT INTO test_url (url) VALUES ('http://www.google.fr');
TABLE test_url;

 id          url                     url_tsvector            

  2  http://www.google.fr  'fr':4 'googl':3 'http':1 'www':2 

(1 row)

To FT search on URLs you only need to query against this field.

SELECT * FROM test_url WHERE url_tsvector @@ 'google'::tsquery;
2
  • 1
    You don't even need the extra column if you use a functional index. Further, you can create a view to avoid the need to specify the full form of the expression used to create the index when performing a query.
    – Phil Frost
    Jan 5, 2013 at 1:29
  • I hacked that gem to support arbitrary ts_vector calls, which will allow me to use regexp_replace, as you suggested. Using insert/update triggers might have the benefit of performance, but I find this solution a bit too cumbersome for my needs. Thanks for the help.
    – mihai
    Jan 6, 2013 at 10:45
1

I ended up modifying the pg_search gem to support arbitrary ts_vector expressions instead of just column names. The changes are here

Now I can write:

pg_search_scope :search, 
    against: [[:title , 'B'], ["to_tsvector(regexp_replace(url, '[^\\w]+', ' ', 'gi'))", 'A']],
    using: {tsearch: {dictionary: "simple"}}
0

Slightly simpler approach, add the protocol token type to the simple dictionary:

ALTER TEXT SEARCH CONFIGURATION simple
    ADD MAPPING FOR protocol
        WITH simple;

you can also add it to the english dictionary if you need stemming

https://www.postgresql.org/docs/13/textsearch-parsers.html

https://www.postgresql.org/docs/13/sql-altertsconfig.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.