8

The Subversion API has a number of functions for converting from "natively-encoded" strings to strings that are encoded in UTF-8. My question is: what is this native encoding on Windows? Does it depend on locale?

1
  • 2
    There is only one native encoding in Windows: UTF-16. But probably that documentation rather talks about the legacy MBCS encoding.
    – Philipp
    Jan 10, 2011 at 18:46

4 Answers 4

7

"Natively encoded" strings are strings written in whatever code page the user is using. That is, they are numbers that are translated to the appropriate glyphs based on the correct code page. Assuming the file was saved that way and not as a UTF-8 file.

This is a candidate question for Joel's article on Unicode.

Specifically:

Eventually this OEM free-for-all got codified in the ANSI standard. In the ANSI standard, everybody agreed on what to do below 128, which was pretty much the same as ASCII, but there were lots of different ways to handle the characters from 128 and on up, depending on where you lived. These different systems were called code pages. So for example in Israel DOS used a code page called 862, while Greek users used 737. They were the same below 128 but different from 128 up, where all the funny letters resided. The national versions of MS-DOS had dozens of these code pages, handling everything from English to Icelandic and they even had a few "multilingual" code pages that could do Esperanto and Galician on the same computer! Wow! But getting, say, Hebrew and Greek on the same computer was a complete impossibility unless you wrote your own custom program that displayed everything using bitmapped graphics, because Hebrew and Greek required different code pages with different interpretations of the high numbers.

2

Windows 1252. Jukka Korpela has an excellent page on character encodings, with an extensive discussion of the Windows character set.

2
  • 1
    Excellent page indeed! I don't think I could ever want a more detailed and comprehensive explanation of character encodings. Jan 10, 2011 at 17:47
  • 1
    that page is nice but not as detailed as the Unicode standard. For example, it makes no distinction between abstract and encoded characters, or between encoding forms and encoding schemes.
    – Philipp
    Jan 10, 2011 at 18:50
2

From the header svn_string.h you can see that the relevant svn_strings are just plain old const char* + a length element.

I would guess that the "natively encoded" svn strings are interpreted according to your system locale (I do not know this for sure, but this is the convention). On Windows 7 you can check your locale by selecting "Start-->Control Panel-->Region and Language-->Administrative-->Change system locale" where any value of English would probably entail the character encoding Windows 1252. However, a different system locale, for example Hebrew (Israel), would entail a different character encoding (Windows 1255 for the case of Hebrew).

1

Sadly the MSVC version of the C library does not support UTF-8 and uses legacy codepages only, but cygwin provides a UTF-8 locale as part of its emulation layer. If your svn is built on cygwin, you should be able to use UTF-8 just fine.

2
  • Only the standard functions use the legacy encoding, many non-standard extensions use UTF-16.
    – Philipp
    Jan 10, 2011 at 18:47
  • Or an #ifdef that calls the UTF-16 functions on Windows and the standard functions elsewhere.
    – dan04
    Jan 11, 2011 at 0:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.