390

What is the difference between the Unicode, UTF8, UTF7, UTF16, UTF32, ASCII, and ANSI encodings?

In what way are these helpful for programmers?

4

2 Answers 2

536

Going down your list:

  • "Unicode" isn't an encoding, although unfortunately, a lot of documentation imprecisely uses it to refer to whichever Unicode encoding that particular system uses by default. On Windows and Java, this often means UTF-16; in many other places, it means UTF-8. Properly, Unicode refers to the abstract character set itself, not to any particular encoding.
  • UTF-16: 2 bytes per "code unit". This is the native format of strings in .NET, and generally in Windows and Java. Values outside the Basic Multilingual Plane (BMP) are encoded as surrogate pairs. These used to be relatively rarely used, but now many consumer applications will need to be aware of non-BMP characters in order to support emojis.
  • UTF-8: Variable length encoding, 1-4 bytes per code point. ASCII values are encoded as ASCII using 1 byte.
  • UTF-7: Usually used for mail encoding. Chances are if you think you need it and you're not doing mail, you're wrong. (That's just my experience of people posting in newsgroups etc - outside mail, it's really not widely used at all.)
  • UTF-32: Fixed width encoding using 4 bytes per code point. This isn't very efficient, but makes life easier outside the BMP. I have a .NET Utf32String class as part of my MiscUtil library, should you ever want it. (It's not been very thoroughly tested, mind you.)
  • ASCII: Single byte encoding only using the bottom 7 bits. (Unicode code points 0-127.) No accents etc.
  • ANSI: There's no one fixed ANSI encoding - there are lots of them. Usually when people say "ANSI" they mean "the default locale/codepage for my system" which is obtained via Encoding.Default, and is often Windows-1252 but can be other locales.

There's more on my Unicode page and tips for debugging Unicode problems.

The other big resource of code is unicode.org which contains more information than you'll ever be able to work your way through - possibly the most useful bit is the code charts.

14
  • 8
    The term "ANSI" when applied to Microsoft's 8-bit code pages is a misnomer. They were based on drafts submitted for ANSI standardization, but ANSI itself never standardized them. Windows-1252 (the code page most commonly referred to as "ANSI") is similar to ISO 8859-1 (Latin-1), except that Windows-1252 has printable characters in the range 0x80..0x9F, where ISO 8859-1 has control characters in that range. Unicode also has control characters in that range. en.wikipedia.org/wiki/Windows_code_page Jun 15, 2015 at 23:59
  • 1
    @jp2code: I wouldn't - but you need to distinguish between "content that is sent back via HTTP from the web server" and "content that is sent via email". It's not the web page content that sends the email - it's the app behind it, presumably. The web content would be best in UTF-8; the mail content could be in UTF-7, although I suspect that it's fine to keep that in UTF-8 these days.
    – Jon Skeet
    Oct 1, 2015 at 13:39
  • 2
    For UTF-16, IMHO, I would say "2 bytes per code unit" since a code point outside the BMP will be encoded in surrogate pairs as 2 code units (4 bytes). Dec 14, 2015 at 14:04
  • 1
    Misses the differences between UTF-16LE (within .NET) and BE as well as the notion of the BOM. Apr 20, 2016 at 15:47
  • 2
    @Andrew: No, there's no (general) encoding marker. Windows 1252 can't represent the Unicode BOM, and it wouldn't make sense as it's only a one-byte-per-char encoding anyway.
    – Jon Skeet
    Jan 3, 2018 at 21:39
77

Some reading to get you started on character encodings: Joel on Software: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

By the way - ASP.NET has nothing to do with it. Encodings are universal.

2
  • 12
    Answered here 6 years after the article was written. I read it 8 years after the post was written. 14 years later and it's still a good read. That's more than half my life ago. Incredible.
    – Dave Knise
    Aug 1, 2017 at 23:42
  • Another similar useful resource: youtube.com/watch?v=MijmeoH9LT4 Dec 15, 2020 at 8:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.