0

I am trying to train a naive bayes classifier and I am having troubles with the data. I plan to use it for extractive text summarization.

Example_Input: It was a sunny day. The weather was nice and the birds were singing.
Example_Output: The weather was nice and the birds were singing.

I have a dataset that I plan to use and in every document there is at least 1 sentence for summary.

I decided to use sklearn but I don't know how to represent the data that I have. Namely X and y.

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X, y)

The closest to my mind is to make it like this:

X = [
        'It was a sunny day. The weather was nice and the birds were singing.',
        'I like trains. Hi, again.'
    ]

y = [
        [0,1],
        [1,0]
    ]

where the target values mean 1 - included in the summary and 0 - not included. This unfortunately gives bad shape exception beacause y is expected to be 1-d array. I cannot think of a way of representing it as such so please help.

btw, I don't use the string values in X directly but represent them as vectors with CountVectorizer and TfidfTransformer from sklearn.

1 Answer 1

1

As per your requirement, you are classifying the data. That means, you need to separate each sentence to predict it's class.

for example:
Instead of using:

X = [
        'It was a sunny day. The weather was nice and the birds were singing.',
        'I like trains. Hi, again.'
    ]

Use it as following:

X = [
        'It was a sunny day.',
        'The weather was nice and the birds were singing.',
        'I like trains.',
        'Hi, again.'
    ]

Use sentence tokenizer of NLTK to achieve this.

Now, for labels, use two-classes. let say 1 for yes, 0 for no.

y = [
        [0,],
        [1,],
        [1,],
        [0,]
    ]

Now, use this data to fit and predict the way you want!

Hope it helps!

2
  • Thanks for your answer. It will work and is certainly better than mine but this way the classifier won't consider the place of the sentence in the document as everything will be considered as one. Is there a way I can include that as well.
    – Nikola
    Apr 5, 2017 at 12:23
  • @nikola take sentence with multiple lines as input and split it using nltk sentence tokenizer and predict each one but print only those sentences to the output which have prediction of class 1 i.e., yes
    – abhinav
    Apr 5, 2017 at 13:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.