1

I have a table like this:

sport    personId
1        1
1        2
1        3
2        1
2        4
2        6
3        1
3        3
3        6

I would like to get a table with the count of the intersection of sports - something equivalent of the pseudo-code:

for sport1 in sports:
   for sport2 in sports:
       X[sport1,sport2] = count(intersect(sport1,sport2))

I know I to do it if I have only couple of sports:

select count(*) from (select * from table where sport = sport1) t1 join (select count(*) from table where sport = sport2) t2 on t1.personId = t2.personId;

But I would like to do generate a table for all my sports.

Note that I am on Hive 0.9, that doesn't support cross join.

Thanks!

1 Answer 1

1

You want a table that, for each pair of sports, lists the number of people who play both sports?

If so, you could do that in streaming hadoop and python:

  • Sort by personId
  • Pass sport, personId into Python
  • Cache the personId, and add all the sports to a list
  • Upon getting a different personId, do this:

.

for i, sport1 in enumerate(sports):
    for sport2 in sports[i:]:
        print '\t'.join([sport1, sport2, '1'])

Then select sport1, sport2, count(*) as c group by sport1, sport2 from the intermediate table output from the streaming job.

1
  • Yes, I did the double group by and it did the trick, although it didn't output a table. I'll investigate the python reducer later.
    – S4M
    Jan 26, 2013 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.