5

I have input strings that are comprised of characters, including double and single quotes " and '

B@SS$*JU(PQ
AD&^%$^@!$
%()%@@DDSFD"*")(#
ABD*E@(%J^&@

however, when I open the above input from a text file and just print it, the double quotes " in the third line get printed as \xe2\x80\x9d

I am aiming to do a simple character count:

B 2
@ 3
S 2
$ 3
etc.

so I want to be able to output

" 3

in the above list. Should I replace the double quotes with something so I can count them and print out the count?

Thanks a lot.

1 Answer 1

12

\xe2\x80\x9d

Is a unicode value for "special" double quotes. You could decode from UTF-8 into Unicode to convert this into a "single" Unicode character.

>>> print "\xe2\x80\x9d".decode("utf-8")
”
>>> len("\xe2\x80\x9d".decode("utf-8"))
1

If you are using Python 3:

>>> print(b"\xe2\x80\x9d".decode('utf8'))
”
>>> len(b"\xe2\x80\x9d".decode("utf-8"))
1

So for your file that you are counting (in Python 2):

from collections import defaultdict
with open("filename", 'r') as f:
    for text in f:
        decoded = text.decode("utf-8")
        count = defaultdict(int)
        for i in decoded:
            count[i] += 1
5
  • thanks a lot, Martin - this definitely helps. Now when I decode each line to utf8, then do a character count, the double quotes appear as u'\u201d' and again they won't print. Is there a way around that? Jun 16, 2014 at 2:16
  • How are you printing it? Also, are you using Python2 or Python3? Jun 16, 2014 at 2:17
  • I'm using python 2.76 and I'm creating a dictionary of key-value pairs for the characters counts and then printing the keys. So the double quote gets stored as u'\u201d': 3, and when I try to print the keys it throws the error: UnicodeEncodeError: 'ascii' codec can't encode character u'\u201d' in position 0: ordinal not in range(128) Jun 16, 2014 at 2:20
  • Are you using str function somewhere by any chance? What does your print code look like? Jun 16, 2014 at 2:22
  • you're a genius - I must have put the str(key) in there at some point when I was playing around. Thank you so much!!! Jun 16, 2014 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.