0

I have tried to change the format of strings from latin1 to ascii, and most of the strings were changed well except for some characters, æ ø. Æ, and Ø.

I have checked the characters were changed correctly when using R package (stringi::stri_trans_general(loc1, "latin-ascii) but Python's unicodedata package did not work well.

Is there any way to convert them correctly in Python? I guess it may need an additional dictionary.

For information, I have applied the following function to change the format: unicodedata.normalize('NFKD', "Latin strings...").encode('latin1', 'ignore').decode('ascii')

4
  • 3
    "did not work well" is not a good problem description. Show some code, show some results, show some expected results. Mar 25, 2021 at 3:05
  • barentsøya should be changed to barentsoya (and the R package did it well) but Python changed it to barentsya (because of the 'ignore' option in the encode function)
    – Taehyun Ha
    Mar 25, 2021 at 3:13
  • 1
    I have to apologize, I thought unicodedata was the way to do this but I was getting it confused with unidecode which is not included with Python. Check it out. Mar 25, 2021 at 3:20
  • 1
    You might like to look at the module unidecode which does what I think you want, thought that is hard to tell, because you haven't actually said what you want to happen with characters like æ and ø and ß, only that the result of setting errors="ignore" isn't what you expect, which is what R does (and it's not sensible to expect that we will all know what R does). What unidecode does is ae, o, ss. That might not be what you want either.
    – BoarGules
    Jul 12, 2021 at 8:05

1 Answer 1

1

It's important to understand a) what encodings and decodings are; b) how text works; and c) what unicode normalization does.

Strings do not have a "format" in the sense that you describe, so talking about converting from latin1 to ascii format does not make sense. The string has representations (what it looks like when you print it out; or what the code looks like when you create it directly in your code; etc.), and it can be encoded. latin1, ascii etc. are encodings - that means, rules that explain how to store your string as a raw sequence of bytes.

So if you have a string, it is not "in latin1 format" just because the source data was in latin1 encoding - it is not in any format, because that concept doesn't apply. It's just a string.

Similarly, we cannot ask for a string "in ascii format" that we convert to. We can ask for an ascii encoding of the string - which is a sequence of bytes, and not text. (That "not" is one of the most important "not"s in all of computer science, because many people, tools and programs will lie to you about this.)

Of course, the problem here is that ascii cannot represent all possible text. There are over a million "code points" that can theoretically be used as elements of a string (this includes a lot of really weird things like emoji). The latin-1 and ascii encodings both use a single byte per code point in the string. Obviously, this means they can't represent everything. Latin-1 represents only the first 256 possible code points, and ascii represents only the first 128. So if we have data that comes from a latin-1 source, we can get a string with those characters like Æ in it, which cause a problem in our encoding step.

The 'ignore' option for .encode makes the encoder skip things that can't be handled by the encoding. So if you have the string 'barentsøya', since the ø cannot be represented in ascii, it gets skipped and you get the bytes b'barentsya' (using the unfortunately misleading way that Python displays bytes objects back to you).

When you normalize a string, you convert the code points into some plain format that's easier to work with, and treats distinct ways of writing a character - or distinct ways of writing very similar characters - the same way. There are a few different normalization schemes. The NFKD chooses decomposed representations for accented characters - that is, instead of using a single symbol to represent a letter with an accent, it will use two symbols, one that represents the plain letter, and one representing the "combining" version of the accent. That might seem useful - for example, it would turn an accented A into a plain A and an accent character. You might think that you can then just encode this as ascii, let the accent characters be ignored, and get the result you want. However, it turns out that this is not enough, because of how the normalization works.

Unfortunately, I think the best you can do is to either use a third-party library (and please note that recommendations are off-topic for Stack Overflow) or build the look-up table yourself and just translate each character. (Have a look at the built-in string methods translate and maketrans for help with this.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.