I have a table id_vectors
that contains id
and their corresponding coordinates
. Each of the coordinates
is a repeated fields with 512 elements inside it.
I am looking for pairwise cosine similarity between all those vectors, e.g. If I have three ids
1,2 and 3 then I am looking for a table where I have cosine similarity between them (based on the calculation using 512 coordinates) like below:
id1 id2 similarity
1 2 0.5
1 3 0.1
2 3 0.99
Now in my table I have 424,970 unique ID
and their corresponding 512-dimension coordinates. Which means that basically I need to create around (424970 * 424969 / 2) unique pair of IDs and calculate their similarity.
I first tried with the following query using reference from here:
#standardSQL
with pairwise as
(SELECT t1.id as id_1, t1.coords as coord1, t2.id as id_2, t2.coords as coord2
FROM `project.dataset.id_vectors` t1
inner join `project.dataset.id_vectors` t2
on t1.id < t2.id)
SELECT id_1, id_2, (
SELECT
SUM(value1 * value2)/
SQRT(SUM(value1 * value1))/
SQRT(SUM(value2 * value2))
FROM UNNEST(coord1) value1 WITH OFFSET pos1
JOIN UNNEST(coord2) value2 WITH OFFSET pos2
ON pos1 = pos2
) cosine_similarity
FROM pairwise
But after running for 6 hrs I encountered the following error message
Query exceeded resource limits. 2.2127481953201417E7 CPU seconds were used, and this query must use less than 428000.0 CPU seconds.
Then I thought rather than using an intermediate table pairwise
, why don't I try to create that table first then do the cosine similarity calculation.
So I tried the following query:
SELECT t1.id as id_1, t1.coords as coord1, t2.id as id_2, t2.coords as coord2
FROM `project.dataset.id_vectors` t1
inner join `project.dataset.id_vectors` t2
on t1.id < t2.id
But this time the query could not be completed and I encountered the following message:
Error: Quota exceeded: Your project exceeded quota for total shuffle size limit. For more information, see https://cloud.google.com/bigquery/troubleshooting-errors
.
Then I tried to create even a smaller table, by just creating the combination pairs of the ids and stripping off the coordinates from it, using the following query:
SELECT t1.id as id_1, t2.id as id_2
FROM `project.dataset.id_vectors` t1
inner join `project.dataset.id_vectors` t2
on t1.id < t2.id
Again my query ends up with the error message Query exceeded resource limits. 610104.3843576935 CPU seconds were used, and this query must use less than 3000.0 CPU seconds. (error code: billingTierLimitExceeded)
I totally understand that this is a huge query and my stopping point is my billing quota.
What I am asking is that, is there a way to execute the query in a smarter way so that I do not exceed either of the resourceLimit
, shuffleSizeLimit
or billingTierLimit
?