1

Thanks in advance..I need to create a term document matrix using apache spark ... Can any one tell me how I can to that using java mllib library of spark.Also can we implement affinity propagation algorithm using mllib.

2 Answers 2

2

For document classification in Spark have a look at Naive Bayes http://spark.apache.org/docs/latest/mllib-naive-bayes.html

You may also want to have a look at the TF-IDF method http://spark.apache.org/docs/latest/mllib-feature-extraction.html

2

Have a look at the following blog for details. Here's a summary:

sc = SparkContext('local', 'term_doc')
corpus = sc.parallelize([
    "It is the east, and Juliet is the sun.",
    "A dish fit for the gods.",
    "Brevity is the soul of wit."])

tokens = corpus.map(lambda raw_text: raw_text.split()).cache()   
local_vocab_map = tokens.flatMap(lambda token: token).distinct()\
                        .zipWithIndex().collectAsMap()

vocab_map = sc.broadcast(local_vocab_map)
vocab_size = sc.broadcast(len(local_vocab_map))

term_document_matrix = tokens \
 .map(Counter) \
 .map(lambda counts: {vocab_map.value[token]: float(counts[token]) for token in counts})\
 .map(lambda index_counts: SparseVector(vocab_size.value, index_counts))

for doc in term_document_matrix.collect():
    print doc`

This will produce the following output:

>>> tokens.first()
['It', 'is', 'the', 'east,', 'and', 'Juliet', 'is', 'the', 'sun.']

>>> local_vocab_map
{'and': 0, 'A': 1, 'fit': 14, 'for': 13, 'of': 3, 'is': 4, 'gods.': 7, 'It': 11,\
'Brevity': 10, 'soul': 12, 'sun.': 8, 'dish': 2, 'east,': 9, 'the': 5, 'wit.': 6, 'Juliet': 15}

>>> for doc in term_document_matrix.collect():
        print doc
(16,[0,4,5,8,9,11,15],[1.0,2.0,2.0,1.0,1.0,1.0,1.0])
(16,[1,2,5,7,13,14],[1.0,1.0,1.0,1.0,1.0,1.0])
(16,[3,4,5,6,10,12],[1.0,1.0,1.0,1.0,1.0,1.0])

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.