2

I'm coding an Domain, URL and Regex (like SquidGuard) filter for Squid using the eCAP protocol and i want store all the domains into a postgresql database. The problem is that when i do a search with like, example:

SELECT website_groups.id,
       "name",
       description
FROM website_domains
   JOIN website_groups ON website_group_id = website_groups.id
WHERE (website_domains.domain = 'google.com'
    OR website_domains.domain LIKE '%.google.com')

the query for 1'605'923 tuples has a lag of 490ms, but this is bad for all the request to squid proxy.

My question is how optimize PostgreSQL to make that query more fast, or I need to use a NoSQL Database (I test with MongoDB and make the query in 609ms with less data).

I tried with FullText search, but this has english tokenize and the data are URLs (www.google.com/query?data1=3), Domains (bing.com) and Regex (.*.cu).

1 Answer 1

2

You may try to create a column for the reverse domain string and create an index on it:

ALTER TABLE website_domains ADD reverse_domain VARCHAR(100);
UPDATE website_domains SET reverse_domain = REVERSE(domain);
CREATE INDEX reverse_domain_index ON 
    website_domains (reverse_domain varchar_pattern_ops);

varchar_pattern_ops allows LIKE to use this index if possible.

The prefix search is done by reversing the pattern as well:

... OR website_domains.reverse_domain LIKE REVERSE('%.google.com')

You can probably avoid the extra column with a computed index

CREATE INDEX reverse_domain_index ON 
    website_domains (REVERSE(domain) varchar_pattern_ops);

and the following clause:

.. OR REVERSE(website_domains.domain) LIKE REVERSE('%.google.com')

But you should try.

2
  • 1
    Ohhhh!!! Now make the query in only 12ms Thankssss, finnaly i used the INDEX ufff is very quick, if you may explain me why is so quick in a short answer..... Jan 17, 2018 at 12:27
  • 1
    The index is based on a search tree, which are good for exact and for prefix matches as well. Searching a tree for an exact match is actually a prefix search. When you make a suffix search (like .. OR REVERSE(website_domains.domain) LIKE REVERSE('www.google.%')it should always be slow. Take a look at the query plans, too.
    – clemens
    Jan 17, 2018 at 12:31

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.