The BigQuery team strikes again: This question is not longer relevant, as the results with LOWER() are as fast as with REGEX() now.
Processing ~5GB of data with BigQuery should be super fast. For example the following query performs a case insensitive search in 18 seconds:
#standardSQL
SELECT COUNT(*) c
FROM `bigquery-public-data.hacker_news.full`
WHERE
LOWER(text) LIKE '%bigquery%' # 18s
Usually BigQuery is faster than this, but the real problem is that adding new search terms makes this query considerably slower (almost a minute with 3 search terms):
#standardSQL
SELECT COUNT(*) c
FROM `bigquery-public-data.hacker_news.full`
WHERE
LOWER(text) LIKE '%bigquery%' OR LOWER(text) LIKE '%big query%' # 34s
#standardSQL
SELECT COUNT(*) c
FROM `bigquery-public-data.hacker_news.full`
WHERE
LOWER(text) LIKE '%bigquery%' OR LOWER(text) LIKE '%big query%'
OR LOWER(text) LIKE '%google cloud%' # 52s
How can I improve my query performance?