5

I have a text file that claims to be UTF-8 encoded. That is, when i call file -I $file it prints $file: text/plain; charset=utf-8. But when I open it with UTF-8 encoding some characters seem corrupted. That is, the file is suppose to be german but the special german characters like ö are displayed as ö.

I guessed that the claim to be UTF-8 is wrong and executed the enca script to guess the real encoding. But sadly enca tells me that the language de (german) is not supported.

Is there another way to fix the file?

3 Answers 3

4

The UTF-8 encoded form of “ö” U+00F6 is 0xC3 0xB6, and if these bytes are interpreted in ISO-8859-1 they are “ö” (U+00C3 U+00B6). So either the file is actually being read and interprered as ISO-8859-1, even though you expect otherwise, or there has been a double encoding: previously, the file or part thereof has been read as if it were ISO-8859-1 (even though it was UTF-8), and the misinterpreted data has then been written out as UTF-8 encoded.

2
  • It actually could be the case, that there was a double encoding. Is there a way to fix that?
    – katosh
    Feb 12, 2014 at 19:40
  • Have you tried opening the file, doing :set fenc=latin1, and saving the file? That seems to convert ö to ö correctly, and file reports utf-8 both before and after. Feb 12, 2014 at 21:29
3

To get a file to read properly in a given encoding, you need three things:

  1. 'encoding' which controls the characters Vim can store and display must be able to represent all the characters in your file.
  2. 'fileencodings' which controls which encodings Vim will attempt to recognize must be set in a way that your file encoding is recognized
  3. 'fileencoding' must be set properly, normally by being automatically detected by the 'fileencodings' setting, to the encoding your file is stored in.

Note that (2) is not strictly necessary, but if the file encoding is detected improperly, you will need to manually re-read the file in the correct encoding. For example, using :e ++enc=utf-8 for a utf-8 file that was not detected as such.

See http://vim.wikia.com/wiki/Working_with_Unicode for getting all three of these concepts correct.

1
  • Thank you, this :e ++enc=utf-8 did the trick for me; had to run that command, save, exit, and re-open the file and everything was fixed.
    – kjones
    Nov 9, 2020 at 18:02
2

You can also check the encoding with :set encoding, and set it accordingly with :set encoding=utf-8. If you still see incorrect characters, that means those where not written in the file as utf-8 and you'll need to convert them.

EDIT : if you could submit your file it would help

3
  • I think that the 'fileencoding' option (short form 'fenc') may matter more than the 'encoding' ('enc') option. Feb 12, 2014 at 19:32
  • Yes, I tried setting the encoding to several types, including utf-8, but nothing helped. :set encoding prints encoding=utf-8but that seems to be wrong. Unfortunately I am not allowed to publish the whole file but here is just a word of it: http://userpage.fu-berlin.de/katosh/problem.txt
    – katosh
    Feb 12, 2014 at 19:38
  • 1
    The 'encoding' option in Vim has very little to do with the encoding of the file when it is written. 'encoding' controls how bytes are interpreted and stored deep within the bowels of Vim. You want the 'fileencoding' option. Now, 'encoding' does matter a little bit, because it must be able to represent all characters in the 'fileencoding' of your file for the conversion to work properly.
    – Ben
    Feb 13, 2014 at 3:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.