1

I'm building a directory of users, where:

  • each user can have an account on one or more external services, and
  • each of these accounts can have one or more email addresses.

What I want to know is, how can I aggregate these accounts into single identities through common email addresses?

For example, let's say I have two services, A and B. For each service, I have a table that relates an account to one or more email addresses.

So if service A has these account email addresses:

account_id | email_address
-----------|--------------
1          | [email protected]
1          | [email protected]
2          | [email protected]

and service B has these account email addresses:

account_id | email_address
-----------|--------------
3          | [email protected]
3          | [email protected]
4          | [email protected]

I'd like to create a table that aggregates the email addresses of these accounts into a single user identity:

user_id | email_address
--------|--------------
X       | [email protected]
X       | [email protected]
X       | [email protected]
Y       | [email protected]
Z       | [email protected]

As you can see, account 1 from service A and account 2 from service B have been merged into a common user X, based on the common email address [email protected]. Here's an animated visual:

Animated visual

The closest answer I could find is this one, and I suspect the solution is a recursive CTE, but given the inputs and engine are different I'm having trouble implementing it.

Clarification: I'm looking for a solution that handles an arbitrary number of services, so perhaps the input table might be better off as:

service_id | account_id | email_address
-----------|------------|--------------
A          | 1          | [email protected]
A          | 1          | [email protected]
A          | 2          | [email protected]
B          | 3          | [email protected]
B          | 3          | [email protected]
B          | 4          | [email protected]
1
  • Just a bit unclear: guess your account 2 from service B is the 2nd row 3 | [email protected] of service B?
    – pirho
    Commented Nov 3, 2018 at 8:29

2 Answers 2

1

demo1:db<>fiddle, demo2:db<>fiddle

WITH combined AS (
    SELECT
        a.email as a_email,
        b.email as b_email,
        array_remove(ARRAY[a.id, b.id], NULL) as ids
    FROM 
        a
    FULL OUTER JOIN b ON (a.email = b.email)
), clustered AS (
    SELECT DISTINCT
        ids
    FROM (
        SELECT DISTINCT ON (unnest_ids) 
            *, 
            unnest(ids) as unnest_ids 
        FROM combined
        ORDER BY unnest_ids, array_length(ids, 1) DESC
    ) s
)
SELECT DISTINCT
    new_id, 
    unnest(array_cat) as email
FROM (
    SELECT
        array_cat(
            array_agg(a_email) FILTER (WHERE a_email IS NOT NULL), 
            array_agg(b_email) FILTER (WHERE b_email IS NOT NULL)
        ), 
        row_number() OVER () as new_id
    FROM combined co
    JOIN clustered cl
    ON co.ids <@ cl.ids
    GROUP BY cl.ids
) s

Step by step explanation:

For explanation I'll take this dataset. This is a little bit more complex than yours. It can illustrate my steps better. Some problems don't occur in your smaller set. Think about the characters as variables for email addresses.

Table A:

| id | email |
|----|-------|
|  1 |     a |
|  1 |     b |
|  2 |     c |
|  5 |     e |

Table B

| id | email |
|----|-------|
|  3 |     a |
|  3 |     d |
|  4 |     e |
|  4 |     f |
|  3 |     b |

CTE combined:

JOIN of both tables on same email addresses to get a touch point. IDs of same Ids will be concatenated in one array:

|   a_email |   b_email | ids |
|-----------|-----------|-----|
|    (null) | [email protected] |   3 |
| [email protected] | [email protected] | 1,3 |
| [email protected] |    (null) |   1 |
| [email protected] |    (null) |   2 |
|    (null) | [email protected] |   4 |

CTE clustered (sorry for the names...):

Goal is to get all elements exactly in only one array. In combined you can see, for example currently there are more arrays with the element 4: {5,4} and {4}.

First ordering the rows by the length of their ids arrays because the DISTINCT later should take the longest array (because holding the touch point {5,4} instead of {4}).

Then unnest the ids arrays to get a basis for filtering. This ends in:

| a_email | b_email | ids | unnest_ids |
|---------|---------|-----|------------|
|       b |       b | 1,3 |          1 |
|       a |       a | 1,3 |          1 |
|       c |  (null) |   2 |          2 |
|       b |       b | 1,3 |          3 |
|       a |       a | 1,3 |          3 |
|  (null) |       d |   3 |          3 |
|       e |       e | 5,4 |          4 |
|  (null) |       f |   4 |          4 |
|       e |       e | 5,4 |          5 |

After filtering with DISTINCT ON

| a_email | b_email | ids | unnest_ids |
|---------|---------|-----|------------|
|       b |       b | 1,3 |          1 |
|       c |  (null) |   2 |          2 |
|       b |       b | 1,3 |          3 |
|       e |       e | 5,4 |          4 |
|       e |       e | 5,4 |          5 |

We are only interested in the ids column with the generated unique id clusters. So we need all of them only once. This is the job of the last DISTINCT. So CTE clustered results in

| ids |
|-----|
|   2 |
| 1,3 |
| 5,4 |

Now we know which ids are combined and should share their data. Now we join the clustered ids against the origin tables. Since we have done this in the CTE combined we can reuse this part (that's the reason why it is outsourced into a single CTE by the way: We do not need another join of both tables in this step anymore). The JOIN operator <@ says: JOIN if the "touch point" array of combined is a subgroup of the id cluster of clustered. This yields in:

| a_email | b_email | ids | ids |
|---------|---------|-----|-----|
|       c |  (null) |   2 |   2 |
|       a |       a | 1,3 | 1,3 |
|       b |       b | 1,3 | 1,3 |
|  (null) |       d |   3 | 1,3 |
|       e |       e | 5,4 | 5,4 |
|  (null) |       f |   4 | 5,4 |

Now we are able to group the email addresses by using the clustered ids (rightmost column).

array_agg aggregates the mails of one column, array_cat concatenates the email arrays of both columns into one big email array.

Since there are columns where email is NULL we can filter these values out before clustering with the FILTER (WHERE...) clause.

Result so far:

| array_cat |
|-----------|
|         c |
| a,b,a,b,d |
|     e,e,f |

Now we group all email addresses for one single id. We have to generate new unique ids. That's what the window function row_number is for. It simply adds a row count to the table:

| array_cat | new_id |
|-----------|--------|
|         c |      1 |
| a,b,a,b,d |      2 |
|     e,e,f |      3 |

Last step is to unnest the array to get a row per email address. Since in the array are still some duplicates we can eliminate them in this step with a DISTINCT as well:

| new_id | email |
|--------|-------|
|      1 |     c |
|      2 |     a |
|      2 |     b |
|      2 |     d |
|      3 |     e |
|      3 |     f |
2
  • Wow, what a great answer, @S-Man! I'd like to generalize to not just two but n services, so I've added a possible input table above that removes the need to join multiple tables. Do you know how I'd adapt your answer to support this? Commented Nov 4, 2018 at 2:05
  • 1
    What about SELECT... FROM table WHERE serviceid = A JOIN SELECT... FROM table WHERE serviceid = B ON... in the combined. So split the table with WHERE into two tables that look like your two tables above. Next step would be trying that with a third join for a service C. Maybe you could build a more advanced data set so I could have a look at it.
    – S-Man
    Commented Nov 4, 2018 at 7:08
0

OK, provided you only have two 'services', and assuming that to begin with you are not overly concerned with how to best represent the new key (I've used text as the easiest to hand), then please try the below query. This works for me on Postgres 9.6:

WITH shared_addr AS 
(
SELECT foo.account_a, foo.account_b, row_number() OVER (ORDER BY foo.account_a) AS shared_id
FROM (
SELECT 
  a.account_id as account_a
, b.account_id as account_b
FROM
service_a a
JOIN
service_b b
ON 
a.email_address = b.email_address
GROUP BY a.account_id, b.account_id
) foo
)
SELECT
bar.account_id,
bar.email_address
FROM
(
SELECT
'A-' || service_a.account_id::text AS account_id,
service_a.email_address
FROM service_a
LEFT OUTER JOIN 
shared_addr
ON
shared_addr.account_a = service_a.account_id
WHERE shared_addr.account_b IS NULL

UNION ALL

SELECT
'B-' ||service_b.account_id::text,
service_b.email_address FROM service_b
LEFT OUTER JOIN 
shared_addr
ON
shared_addr.account_b = service_b.account_id
WHERE shared_addr.account_a IS NULL

UNION ALL

(
SELECT
'shared-' || shared_addr.shared_id::text,
service_b.email_address
FROM service_b
JOIN 
shared_addr
ON
shared_addr.account_b = service_b.account_id

UNION

SELECT
'shared-' || shared_addr.shared_id::text,
service_a.email_address
FROM service_a
JOIN 
shared_addr
ON
shared_addr.account_a = service_a.account_id
)
) bar
;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.