I'm using NLTK to perform kmeans clustering on my text file in which each line is considered as a document. So for example, my text file is something like this:

belong finger death punch
hasty
mike hasty walls jericho
jägermeister rules
rules bands follow performing jägermeister stage
approach

Now the demo code I'm trying to run is this: https://gist.github.com/xim/1279283

The error I receive is this:

Traceback (most recent call last):
File "cluster_example.py", line 40, in
words = get_words(job_titles)
File "cluster_example.py", line 20, in get_words
words.add(normalize_word(word))
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/nltk/decorators.py", line 183, in memoize
result = func(*args)
File "cluster_example.py", line 14, in normalize_word
return stemmer_func(word.lower())
File "/usr/local/lib/python2.7/dist-packages/nltk/stem/snowball.py", line 694, in stem
word = (word.replace(u"\u2019", u"\x27")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 13: ordinal not in range(128)

What is happening here?

share|improve this question
up vote 82 down vote accepted

The file is being read as a bunch of strs, but it should be unicodes. Python tries to implicitly convert, but fails. Change:

job_titles = [line.strip() for line in title_file.readlines()]

to explicitly decode the strs to unicode (here assuming UTF-8):

job_titles = [line.decode('utf-8').strip() for line in title_file.readlines()]

It could also be solved by importing the codecs module and using codecs.open rather than the built-in open.

share|improve this answer
    
running this line.decode('utf-8').strip().lower().split() also gives me the same error. I have added the .deocode('utf-8') – Aman Mathur Apr 22 '17 at 7:31

You can try this also:

import sys
reload(sys)
sys.setdefaultencoding('utf8')
share|improve this answer
    
What are the implications of this? It sounds like it's something global and not only applicable for this file. – simeg Aug 18 '17 at 20:41
    
Notice that the above is deprecated in Python 3. – gented Sep 6 '17 at 15:51

You can try this before using job_titles string:

source = unicode(job_titles, 'utf-8')
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.