-1

The codes below print the data:

f = codecs.open('scrapeddata.csv', 'r')
data = f.read()
print data

the data looks like this:

Foul by Fabian Sch�r (Switzerland).       Wayne Rooney (England) wins a free kick in the attacking half.       Attempt missed. Xherdan Shaqiri (Switzerland) right footed shot from outside the box is high and wide to the right. Assisted by Josip Drmic.       Booking       James Milner (England) is shown the yellow card for a bad foul.       Stephan Lichtsteiner (Switzerland) wins a free kick in the defensive half.       Foul by James Milner (England).       Offside, Switzerland. G�khan Inler tries a through ball, but Xherdan Shaqiri is caught offside.

Then, I am trying to do simple word frequency analysis with the following codes:

from nltk import FreqDist, sent_tokenize, word_tokenize

data = word_tokenize(data)
freq = FreqDist(data)

freq

This returns:

----> 3 data = word_tokenize(data)
UnicodeDecodeError: 'ascii' codec can't decode byte 0x94 in position 14: ordinal not in range(128)

Any help?

2
  • What encoding is your input file using? Python is assuming it’s ASCII and is choking when it realizes it isn’t.
    – icktoofay
    Sep 9, 2015 at 1:51
  • python 2's messy unicode support was enough of a motivator for me to move to python 3. If python 3 is an option then I would recommend you go that route.
    – user507484
    Sep 9, 2015 at 2:07

2 Answers 2

1

Provide an explicit encoding when you open your file. You said it’s UTF-8, so tell Python:

f = codecs.open('scrapeddata.csv', 'r', 'utf-8')
data = f.read()
0

The original data was collected through web scraping. So I changed the way I saved the original data in a csv and that fixed the ascii error.

data = [' scrapped data here']

w = csv.writer(open('scrapeddata.csv', 'wb'))

for sentence in data:
    w.writerow([sentence.encode('ascii','ignore')]) 

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.