I am trying to do the following in python:
Splitting a file of para of statements into sentences.
Splitting these sentences with contractions into words.
- Trying to remove stopwords from the set of words.
When I do the 2nd step I get the result [['Hello','World'], and so on. I understand ( if I am not wrong) I have got a list or a nested list so the probable error. But have got no idea to solve the error.
import nltk
from nltk.tokenize import WhitespaceTokenizer
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
file = open('C:/temp1/1.txt','r')
text = file.read() # read the contents of the text file into a variable
result1 = nltk.sent_tokenize(text)#split para into sentences
print "Split sentences are "
print result1
tokenizer=WhitespaceTokenizer()
result2 = [tokenizer.tokenize(sent) for sent in result1]#obtains the splitted sentences with contractions
print "Split words in each sentences are "
print result2
english_stops=set(stopwords.words('english'))
result3=[word for word in result2 if word not in english_stops]
print result3
Error:
Split sentences are
['Hello World.', "It's good to see you.", 'Thanks for buying this book.', "Can't is a contraction."]
Split words in each sentences are
[['Hello', 'World.'], ["It's", 'good', 'to', 'see', 'yTraceback (most recent call last):
File "D:\Learn NLTK\import nltk.py", line 34, in <module>
result3=[word for word in result2 if word not in english_stops]
TypeError: unhashable type: 'list'
ou.'], ['Thanks', 'for', 'buying', 'this', 'book.'], ["Can't", 'is', 'a', 'contraction.']]
Do I need to use a nested for loop so as to obtain the stopwords filtering? I have checked the related questions which had the same error but I am new to python so I am unable to grasp any idea from those related questions. Any help would be appreciable. Arc.
set(stopwords.words('english'))
works, I think this may be the error – EdChum Feb 16 '15 at 10:54