35

I want to use the the following function in the MemoryExtensions namespace

public static int ToUpper(this ReadOnlySpan<char> source, Span<char> destination, CultureInfo? culture)

My question now is: am I always safe when destination Span has the length of the source span? e.g.

destination = stackalloc char[source.Length];

If no, can someone provide an example which string converts to a larger string when calling ToUpper on in (including which culture)?

9
  • 7
    I want to say yes, but I know some character sets/cultures do weird things when certain letters are capitalized. I'd expect you'd be fine 99.99%+ of the time, but that extra 0.01% could be killer. Or maybe you are perfectly fine... I'm not an expert here, and I'm interested to see if anyone could prove otherwise. Sep 20 at 19:18
  • 20
    The one example that I could think of, which might do this, was German ""viel spaß", possibly turning the ß character into SS, but it turns out that it is left alone by string.ToUpper(new CultureInfo("de-DE")). Sep 20 at 19:24
  • 4
    ß could also turn into ẞ, the former having 2 utf-8 bytes and the latter having 3 utf-8 bytes, though as @500-InternalServerError pointed out german culture appears to just ignore ß
    – Skgland
    Sep 20 at 19:47
  • 2
    @Narish stackalloc will allocate that many <sizeInBytesOfYourType> which in this case is 2 * source.Length Sep 20 at 20:55
  • 2
    So a ToUpper would only change the length (in UTF-16 code units, which are what matter) if it either changed the number of code points, or if the mapping involved code points beyond the basic multilingual plane.
    – plugwash
    Sep 21 at 17:15

1 Answer 1

35

MemoryExtensions.ToUpper returns -1 if the destination is too small.

The source code for ToUpper has this gem:

// Assuming that changing case does not affect length
if (destination.Length < source.Length)
    return -1;

There is no other point where -1 is returned, the function finishes with return source.Length;

So they've assumed it can't happen. Whether they're right is another question: if you find a counter-example I suggest you file a bug report on GitHub.

The docs for TextInfo (used later on in the code) say:

The returned string might differ in length from the input string. For more information on casing, refer to the Unicode Technical Report #21 "Case Mappings," published by the Unicode Consortium (https://www.unicode.org/). The current implementation preserves the length of the string. However, this behavior is not guaranteed and could change in future implementations.


To clarify further, we are talking about code points in UTF-16. We have not considered UTF-8 or UTF-32, as char is strictly UTF-16.

Unicode defines Case Mapping as follows:

  • Simple (Single-Character) Case Mapping

    The general case mapping in ICU is non-language based and a 1 to 1 generic character map.

    A character is considered to have a lowercase, uppercase, or title case equivalent if there is a respective “simple” case mapping specified for the character in the Unicode Character Database (UnicodeData.txt). If a character has no mapping equivalent, the result is the character itself.

    The APIs provided for the general case mapping, located in uchar.h file, handles only single characters of type UChar32 and returns only single characters. To convert a string to a non-language based specific case, use the APIs in either the unistr.h or ustring.h files with a NULL argument locale.

  • Full (Language-Specific) Case Mapping

    There are different case mappings for different locales. For instance, unlike English, the character Latin small letter ‘i’ in Turkish has an equivalent Latin capital letter ‘I’ with dot above ( \u0130 ‘İ’).

    Similar to the simple case mapping API, a character is considered to have a lowercase, uppercase or title case equivalent if there is a respective mapping specified for the character in the Unicode Character database (UnicodeData.txt). In the case where a character has no mapping equivalent, the result is the character itself.

    To convert a string to a language based specific case, use the APIs in ustring.h and unistr.h with an intended argument locale.

    ICU implements full Unicode string case mappings.

    In general:

    • case mapping can change the number of code points and/or code units of a string,
    • is language-sensitive (results may differ depending on language), and
    • is context-sensitive (a character in the input string may map differently depending on surrounding characters).

TL;DR;

In theory, the number of code points could change (this is separate from the number of bytes). But .NET does not currently implement this. That could change without notice, but that's unlikely until there is a way to calculate the number of code points, due to interdependencies on Span.

6
  • 2
    What the doc says about it: The returned string might differ in length from the input string. For more information on casing, refer to the Unicode Technical Report #21 "Case Mappings," published by the Unicode Consortium (unicode.org). The current implementation preserves the length of the string. However, this behavior is not guaranteed and could change in future implementations.
    – Ben Voigt
    Sep 20 at 21:02
  • 2
    True but I think at that point there would have to be a new API surface which would return the required length, as there are a number of places that this assumption is made. There isn't even such a function buried privately in TextInfo and associated classes. Sep 20 at 21:36
  • I think everybody is thinking in bytes. In bytes, it is almost certainly true that the strings would be different lengths in some circumstances, but the code sample given specifies char as the data type. A char is always a char. Depending on the character set in use and the version and flavor of language employed, that might translate to 1, 2, 3 or even more bytes, but for the purposes of this discussion, that's irrelevant. It's still one char. In C#, it will be one char, regardless of how many bytes it occupies.
    – Yrth
    Sep 28 at 13:38
  • @Yrth I don't think that is strictly true. According to the Unicode Case Mappings "In general: * case mapping can change the number of code points and/or code units of a string, * is language-sensitive (results may differ depending on language), and * is context-sensitive (a character in the input string may map differently depending on surrounding characters)." Sep 28 at 13:48
  • So the fact that .NET does it this way doesn't necessarily mean it's right, or that it won't change in the future. As far as bytes is concerned, I haven't discussed that at all, the discussion is only if a single char could map to multiple char codepoints, or vv, and that could be the case according to the Unicode. Sep 28 at 13:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.