1

I am creating some files using Python, with unicode text, and saving the content in sqlite3 DB. Later, I fetch these files and make some changes. Problem is that Python doesn't think these two strings are the same, even though they look the same.

Here's an example:

Str1 : "Copa América 2019"
Str2 : "Copa América 2019"

While both the strings look exactly the same, the program doesn't treat them the same.

I'm seeing the same behavior in other languages as well (Korean, Japanese etc.), and chose French example here as it's easier to read for me. Korean / Japanese I matched the characters by how they look.

I also did the comparisons online (for ex. https://www.quickdiff.com/) and found the same result... so it's not Python specific but some other issue.

What operation can I do on the strings so the code sees these two strings as same?

Tried explicit conversions to str / utf-8 encoding individually but none of it helped.

a = "Copa América 2019"
b = "Copa América 2019"
if a == b:
    print 'y'

type(a)
    <type 'str'>
type(b)
    <type 'str'>

I would like the String comparison for these strings to result in True.

4
  • Unicode can encode characters like é in different ways. Check len(a) and len(b). You could try normalizing both strings: stackoverflow.com/questions/16467479/normalizing-unicode
    – Blorgbeard
    Jul 22, 2019 at 23:59
  • Thank you. Yes, they are different. len(a) = 18; len(b) = 19. What I'm not able to understand is how to solve this problem. I tried b1=b.encode('utf-8') and a1=a.encode('utf-8'), but they are different as well (of course, because they are different characters).
    – sudcha
    Jul 23, 2019 at 0:01
  • 'Copa Ame\xcc\x81rica 2019' vs 'Copa Am\xc3\xa9rica 2019'
    – Selcuk
    Jul 23, 2019 at 0:02
  • 1
    Update: normalizing seems to work fine: repl.it/repls/RepentantRepulsiveHandwritingrecognition
    – Blorgbeard
    Jul 23, 2019 at 0:03

1 Answer 1

1

Full demonstration of @Blorgbeard's answer.

import unicodedata

a = "é"
b = "é"

print(len(a)) # 1
print(len(b)) # 2

print(len(unicodedata.normalize("NFC", a))) # 1
print(len(unicodedata.normalize("NFC", b))) # 1

print(len(unicodedata.normalize("NFD", a))) # 2
print(len(unicodedata.normalize("NFD", b))) # 2

This question may be flagged as a duplicate, Let this stay here just in case.

1
  • Thank you so much! I'm usually good at finding similar questions, but looks like in this case I didn't know what to search for, really. This solves my issue!
    – sudcha
    Jul 23, 2019 at 0:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.