41

In some Windows 10 builds (insiders starting April 2018 and also "normal" 1903) there is a new option called "Beta: Use Unicode UTF-8 for worldwide language support".

You can see this option by going to Settings and then: All Settings -> Time & Language -> Language -> "Administrative Language Settings"

This is what it looks like:

enter image description here

When this checkbox is checked I observe some irregularities (below) and I would like to know what exactly this checkbox does and why the below happens.

Create a brand new Windows Forms application in your Visual Studio 2019. On the main form specify the Paint even handler as follows:

private void Form1_Paint(object sender, PaintEventArgs e)
{
    Font buttonFont = new Font("Webdings", 9.25f);
    TextRenderer.DrawText(e.Graphics, "0r", buttonFont, new Point(), Color.Black);
}

Run the program, here is what you will see if the checkbox is NOT checked:

enter image description here

However, if you check the checkbox (and reboot as asked) this changes to:

enter image description here

You can look up Webdings font on Wikipedia. According to character table given, the codes for these two characters are "\U0001F5D5\U0001F5D9". If I use them instead of "0r" it works with the checkbox checked but without the checkbox checked it now looks like this:

enter image description here

I would like to find a solution that always works that is regardless whether the box checked or unchecked.

Can this be done?

6
  • 4
    The system locale determines the ANSI and OEM codepages. The checkbox forces them to UTF-8 (codepage 65001). Apparently this has a secondary effect that causes DrawText to not render "0r" using the selected font. I'd guess it's because symbol fonts such as Webdings and Wingding don't claim any Unicode ranges or legacy codepages in the font's OS/2 table. Instead they map codes to arbitrary glyphs. Continuing to guess, maybe "\U0001F5D5\U0001F5D9" will work if you select a regular font. Apparently font fallback can find the needed font(s).
    – Eryk Sun
    Jun 3, 2019 at 15:47
  • Other than the first two sentences, I was just speculating. You're closer to the problem and its solution than I am. I'd prefer for you to answer this yourself if you don't mind.
    – Eryk Sun
    Dec 5, 2019 at 11:50
  • What encoding is your source code in? Changing the Windows setting will change CP_ACP which will change the interpretation of characters in text files that do not have an explicit encoding (via BOM). You source appears to contain extended characters. In source code I recommend using unicode escape codes in string literals for any characters that aren't ASCII, otherwise you are depending on ambient settings which may vary (are varying in this scenario). Jan 31, 2020 at 21:28
  • 1
    @PaulDempsey all information to reproduce it is given in the question. No special encoding changes were made that were not mentioned in the OP. Adittionally, my source does not appear to contain extended characters other than unicode escape codes in string literals, which are marked as such in the OP. Jan 31, 2020 at 22:42

4 Answers 4

16

You can see it in ProcMon. It seems to set the REG_SZ values ACP, MACCP, and OEMCP in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage to 65001.

I'm not entirely sure but it might be related to the variable gAnsiCodePage in KernelBase.dll, which GetACP reads. If you really want to, you might be able to change it dynamically for your program regardless of the system setting by dynamically disassembling GetACP to find the instruction sequence that reads gAnsiCodePage and obtaining a pointer to it, then updating the variable directly.

(Actually, I see references to an undocumented function named SetCPGlobal that would've done the job, but I can't find that function on my system. Not sure if it still exists.)

2
  • Thank you, indeed the values 'ACPT', 'MACCP' and 'OEMCP' change to 65001 if UTF-8 is ticked. How did you find out in the first place that these are the registry-values that are modified by the UTF-8 checkbox? Apr 1, 2020 at 9:09
  • @JoakimThorén: You can see it in ProcMon?
    – user541686
    Apr 1, 2020 at 9:10
7

Please look at this question to see what it solves when it is enabled: How to save to file non-ascii output of program in Powershell?

Also I found explanation written by Ghisler helpful (source):

If you check this option, Windows will use codepage 65001 (Unicode UTF-8) instead of the local codepage like 1252 (Western Latin1) for all plain text files. The advantage is that text files created in e.g. Russian locale can also be read in other locale like Western or Central Europe. The downside is that ANSI-Only programs (most older programs) will show garbage instead of accented characters.


I leave here two ways to enable it, I think they will be helpful for many users:

  1. Win+R -> intl.cpl
  2. Administrative tab
  3. Click the Change system locale button.
  4. Enable Beta: Use Unicode UTF-8 for worldwide language support
  5. Reboot

or alternatively via reg file:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage]
"ACP"="65001"
"OEMCP"="65001"
"MACCP"="65001"
1
  • 1
    Nice; also worth mentioning that a reboot is required (if you do it via Control Panel, you'll be prompted). Note that the setting applies to both the OEM code page (as used in console applications; e.g. 437 (CP437) on US-English systems) and the ANSI code page (as used in GUI-subsystem applications; e.g., 1252 (Windows-1252). Using this still-in-beta setting has far-reaching consequences, so it may not be for everyone - see this answer.
    – mklement0
    Jun 21, 2021 at 20:12
6

Most Windows C APIs come in two different variants:

  • "A" variant that uses 8-bit strings with whatever the systems configured encoding is. This varies depending on the configured country/language. (Microsoft calls the configured encoding the "ANSI Code Page", but it's not really anything to do with ANSI).
  • "W" variant that uses 16-bit strings in a fixed almost-UTF-16 encoding. (The "almost" is because "unpaired surrogates" are allowed; if you don't know what those are then don't worry about them).

The official Microsoft advice is not to use the "A" versions, but to ensure your code always use uses the "W" variants. That way you're supposed to get consistent behaviour no matter what the user's country/language is configured as.

However, it looks like that checkbox is doing more than one thing. It's clear it's supposed to change the "ANSI Code Page" to 65001, which means UTF-8. It looks like it's also changing font rendering to be more Unicody.

I suggest you detect if GetACP() == 65001, then draw the Unicode version of your strings, otherwise draw the old "0r" version. I'm not sure how you do that from .NET...

7
  • 1
    The "A" variant worked with ANSI C, C89 and C90. The "W" variant did NOT work with ANSI C, and was not compatible with portable ('ANSI') C until the standards of C95 were widely adopted.
    – david
    Nov 21, 2020 at 3:33
  • 1
    @david: I'm talking about the Windows-specific APIs such as CreateFileA / CreateFileW. They are completely independent of what language you're using, whether that's some revision of C, or C++, or Pascal, or whatever. You're talking about the changes Microsoft proposed to the official C specification to add better wide-char support, which are a different thing (and are part of the C runtime, not the core OS API).
    – user9876
    Dec 5, 2020 at 21:09
  • And I was commenting on your assertion that the ANSI API (available for use with ANSI c) had nothing to do with ANSI.
    – david
    Dec 7, 2020 at 0:43
  • 1
    learn.microsoft.com/en-us/windows/win32/intl/code-pages explains the origin of the "A" suffix: "Windows code pages, commonly called "ANSI code pages", are [encodings] ... Originally, Windows code page 1252, the code page commonly used for English and other Western European languages, was based on an American National Standards Institute (ANSI) draft. ... but Windows code page 1252 was implemented before the standard became final, and is not exactly the same ... The "A" version handles text based on Windows code pages".
    – user9876
    May 5, 2021 at 0:34
  • I.e. the suffix is "A" because Windows supported a bunch of 8-bit encodings, the one commonly used by Microsoft was based on an ANSI draft code page, but most of the others had nothing to do with ANSI. So the entire encoding mechanism got wrongly called "ANSI Code Page" by western programmers, and the name stuck. It's nothing to do with ANSI C.
    – user9876
    May 5, 2021 at 0:37
1

On my windows, When I checked the Beta: Use Unicode UTF-8 for worldwide language support. The following regedit values in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Nls\CodePage changed.

ACP: 936 -> 65001
MACCP: 10008 -> 65001
OEMCP : 936 -> 65001

If I do not checked, then the visual studio compilation failed with Exception: Bad UTF-8 encoding (U+FFFD; REPLACEMENT CHARACTER) found while decoding string: ..., If I checked, then the compilation successed, but the os is full with unreadable code.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.