3

Hello I have to convert a string column into its uppercase version, but when 'ß' is present in the string, it gets changed to 'SS' while doing uppercase I understand that this is because,earlier 'SS' was considered the uppercase of 'ß'. BUt in 2017, both 'SS' and the uppercase 'ß' is allowed.

and its unicode version is also available.

I have the following questions on this:

  1. Why is python not converting it to the uppercase 'ß'.

  2. Is it because of the unicode standard that is embedded in python? How to know which unicode standard python/jupyter notebook is using?

  3. Is there anyway to get the uppercase 'ß' instead of 'SS' in python?

Is i

6
  • Uppercase ß was added in Unicode 5.1, Python 3.7 should be using Unicode 10. Which Python version are you using?
    – paxdiablo
    Commented Jul 7, 2020 at 6:25
  • Hmm, even Python 3.8 incorrectly converts ß to SS.
    – paxdiablo
    Commented Jul 7, 2020 at 6:35
  • It's defined by Unicode. Note: this is a single character that only looks like two "S", it's not "SS".
    – lenz
    Commented Jul 7, 2020 at 6:39
  • i am using Python 3.7.1 which uses unicode version 11.0.0. while the unicode official code chart lists the uppercase ẞ value as 1E9E, it is still converting it to SS
    – ROma
    Commented Jul 7, 2020 at 6:57
  • Unicode tend to be backward compatible. If it was SS, it will remain probably SS, with implementation note and flags, but the default will remain SS. OTOH uppercase is locale dependent, and python uses OS defined locales (and uppercase). So you may want to check locale, and if you have an alternate form Commented Jul 7, 2020 at 7:49

3 Answers 3

5

Various Python versions use specific Unicode versions. For example, I think the original Python 3.7 used Unicode 10.0.0 which, while it has the letter available (it has had it since Unicode 5.1, I believe), still lists the old upper/lower mapping:

00DF ß LATIN SMALL LETTER SHARP S
    = Eszett
    - German
    - uppercase is "SS"
    - nonstandard uppercase is 1E9E ẞ
1E9E ẞ LATIN CAPITAL LETTER SHARP S
    - lowercase is 00DF ß

Even the latest standard at the time of this answer, 13.0.0 (though this change was made in 11.0.0), appears to allow discretion as to how to convert lower to upper:

00DF ß LATIN SMALL LETTER SHARP S
    = Eszett
    - German
    - not used in Swiss High German
    - uppercase is "SS" or 1E9E ẞ
1E9E ẞ LATIN CAPITAL LETTER SHARP S
    - not used in Swiss High German
    - lowercase is 00DF ß

The following table maps some Python version to Unicode version:

 Python     Unicode
--------    -------
   3.5.9      8.0.0
  3.6.11      9.0.0
   3.7.8     11.0.0
3.8.4rc1     12.1.0
 3.9.0b4     13.0.0
3.10.0a0     13.0.0

So you may well have to wait for a later version of Unicode (and a Python that uses that Unicode version) where the mapping is a little less wishy-washy than uppercase is "SS" or 1E9E ẞ". But this may actually be precluded by the Unicode stability policy which states, in part:

If two characters form a case pair in a version of Unicode, they will remain a case pair in each subsequent version of Unicode. If two characters do not form a case pair in a version of Unicode, they will never become a case pair in any subsequent version of Unicode.

You can make a case pair from a newly introduced character, assuming that the one you want to pair with is not already paired but that's not allowed here since:

  • this "new" character was introduced way back in Unicode 5.1; and
  • the character we'd want to pair it with is already paired.

My reading of that leads me to believe that the only way to fix this without violating that policy, would be to introduce two new characters in a case pair, something like:

ß LATIN SMALL LETTER SHARP S THAT IS LOWER OF ẞ
ẞ LATIN CAPITAL LETTER SHARP S THAT IS UPPER OF ß

However, I'm not sure that'll ever get past the Unicode consortium silliness filters :-)

For an immediate fix, you can simply force that specific character to whatever you want it to be, before applying the inbuilt case change, something like:

to_be_uppered.replace('ß', 'ẞ').upper()
to_be_lowered.replace('ẞ', 'ß').lower()

The latter appears to be unnecessary, at least on my version, Python 3.8.2. I include it just in case an earlier Python version may need it. It may even be worth putting these into a custom my_upper() and my_lower() function, if it turns out there are more cases like this that you need to handle.

3
  • 1
    Actually, there will never be a Unicode version where the simple uppercase mapping of “ß” is “ẞ”. Stability policy (unicode.org/policies/stability_policy.html#Case_Pair) forbids two characters that do not form a case pair in one version of the standard from becoming a case pair in a future version. Commented Jul 7, 2020 at 21:45
  • @CharlotteBuff: good call, I've added a note to the answer to incorporate this information.
    – paxdiablo
    Commented Jul 8, 2020 at 0:42
  • Thank you for the info. This helped clear things up a bit
    – ROma
    Commented Jul 9, 2020 at 7:24
2

That is the behaviour in many other languages as well, you may work around it like this:

my_string.replace('ß', 'ẞ').upper()
1
  • FYI, 'ẞß'.ToUpper([cultureinfo]::GetCultureInfo('de-DE')) in famous PowerShell fails and returns unchanged string ẞß… :)
    – JosefZ
    Commented Jul 7, 2020 at 9:24
0

Apply the unil's workaround (+1):

my_string.replace('ß', 'ẞ').upper()

I can't see any other solution due some kind of political correctness found in Unicode documents:

  • from Character Properties, Case Mappings & Names FAQ:

  • Q: Is all of the Unicode case mapping information in UnicodeData.txt?

    A: No. The UnicodeData.txt file includes all of the one-to-one case mappings. Since many parsers were built with the expectation that UnicodeData.txt would have at most a single character in each case mapping field, the file SpecialCasing.txt was added to provide the one-to-many mappings, such as the one needed for uppercasing ß (U+00DF LATIN SMALL LETTER SHARP S). In addition, CaseFolding.txt contains additional mappings used in case folding and caseless matching. For more information, see Section 5.18, Case Mappings in The Unicode Standard.

  • Q: Why does ß (U+00DF LATIN SMALL LETTER SHARP S) not uppercase to U+1E9E LATIN CAPITAL LETTER SHARP S by default?

    A: In standard German orthography, the sharp s ("ß") used to be exclusively uppercased to a sequence of two capital S characters. This longstanding practice is reflected in the default case mappings in Unicode. A capital form of ß is sometimes preferred for typographic reasons or to avoid ambiguity, such as in uppercase names as found in passports. It is encoded in the Unicode Standard as U+1E9E. While this character is not widely used, is now recognized in the official orthography as an optional uppercase form of ß in addition to "SS". Because it is only an optional alternative, the original mapping to "SS" is retained in the Unicode character properties.

  • from SpecialCasing.txt

    The German es-zed is special--the normal mapping is to SS.

  • from UnicodeData.txt (see Uppercase mapping and Lowercase mapping fields as defined in UnicodeData File Format): lowercase mapping is defined for Latin Capital Letter Sharp S while uppercase mapping for Latin Small Letter Sharp S is not…

 00DF;LATIN SMALL LETTER SHARP S;Ll;0;L;;;;;N;;;;;
 1E9E;LATIN CAPITAL LETTER SHARP S;Lu;0;L;;;;;N;;;;00DF;

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.