43

My std::string is UTF-8 encoded so obviously, str.length() returns the wrong result.

I found this information but I'm not sure how I can use it to do this:

The following byte sequences are used to represent a character. The sequence to be used depends on the UCS code number of the character:

   0x00000000 - 0x0000007F:
       0xxxxxxx

   0x00000080 - 0x000007FF:
       110xxxxx 10xxxxxx

   0x00000800 - 0x0000FFFF:
       1110xxxx 10xxxxxx 10xxxxxx

   0x00010000 - 0x001FFFFF:
       11110xxx 10xxxxxx 10xxxxxx 10xxxxxx

How can I find the actual length of a UTF-8 encoded std::string? Thanks

5
  • 2
    C++ knows nothing about encodings, so you can't expect to use a standard function to do this. Some Operating Systems (e.g., Windows) may offer functions to help with this, if you don't want to write one from scratch. Oct 31, 2010 at 12:56
  • I understand that which is why I tagged this also algorithm, I do want to write one from scratch
    – jmasterx
    Oct 31, 2010 at 12:57
  • 1
    Please note that while Michael's claim was true when he wrote it, since C++11 the Standard Library does know about encodings. See stackoverflow.com/questions/16863937/…
    – Ben Voigt
    May 31, 2013 at 19:08
  • 6
    What do you mean by "actual length"? Number of code points? In NFC, NFD, NFKC, or NFKD? Number of composite characters? Number of graphemes? Number of grapheme clusters? Number of pixels on screen in a given font? Jan 15, 2018 at 16:56
  • 3
    Indeed, most of the answers here assume that "actual length" means "number of code points".
    – plugwash
    Jan 15, 2018 at 20:44

12 Answers 12

74

Count all first-bytes (the ones that don't match 10xxxxxx).

int len = 0;
while (*s) len += (*s++ & 0xc0) != 0x80;
3
  • 4
    Note that this returns the number of code points - assumed this is meant by "actual length". Plus: Only use this algorithm on strings that are verified to be a valid UTF-8 sequence, since it doesn't take invalid byte sequences into consideration that may lead to an incorrect result.
    – Florian S.
    Sep 25, 2019 at 16:36
  • 2
    This would be length in codepoints as opposed to glyphs. For estimating screen space, even in a fixed length font, this won't be adequate. Also, it won't work for calculating the size of the corresponding UTF-16 buffer, although as long as you stay in the Basic Multilingual Plane, this will work (but watch out for those pesky emoji). May 18, 2020 at 21:19
  • 2
    For people from now and beyond, who're trying to align items in the command line (with monospaced font), this won't work properly for symbols in unicode range 1F300–1F5FF as they are printed as 2 character long symbols Jul 30, 2021 at 11:18
23

C++ knows nothing about encodings, so you can't expect to use a standard function to do this.

The standard library indeed does acknowledge the existence of character encodings, in the form of locales. If your system supports a locale, it is very easy to use the standard library to compute the length of a string. In the example code below I assume your system supports the locale en_US.utf8. If I compile the code and execute it as "./a.out ソニーSony", the output is that there were 13 char-values and 7 characters. And all without any reference to the internal representation of UTF-8 character codes or having to use 3rd party libraries.

#include <clocale>
#include <cstdlib>
#include <iostream>
#include <string>

using namespace std;

int main(int argc, char *argv[])
{
  string str(argv[1]);
  unsigned int strLen = str.length();
  cout << "Length (char-values): " << strLen << '\n';
  setlocale(LC_ALL, "en_US.utf8");
  unsigned int u = 0;
  const char *c_str = str.c_str();
  unsigned int charCount = 0;
  while(u < strLen)
  {
    u += mblen(&c_str[u], strLen - u);
    charCount += 1;
  }
  cout << "Length (characters): " << charCount << endl; 
}
8
  • 2
    right, and use std::mbrlen if you want to be safe(ish) Mar 23, 2015 at 20:41
  • are you sure about the locale name? on ubuntu the correct one should be "en_US.utf8".
    – qed
    Nov 15, 2018 at 11:59
  • @user2781185, did you intend to change the code to have "en_US.utf8" in response to qed's comment? Because I still see "en_US.UTF-8" in the code above. Sep 4, 2019 at 15:17
  • Hmm, I indeed thought I had changed it... But let's try again. Hopefully it is now correct. Sep 5, 2019 at 16:29
  • Can you please explain why it returns 5 (characters) for this string: std::string str(u8"Fünf"); on Windows? I expect to get 4, which is so when I run the same code on Linux (gcc).
    – vahancho
    Jul 10, 2020 at 11:17
8

This is a naive implementation, but it should be helpful for you to see how this is done:

std::size_t utf8_length(std::string const &s) {
  std::size_t len = 0;
  std::string::const_iterator begin = s.begin(), end = s.end();
  while (begin != end) {
    unsigned char c = *begin;
    int n;
    if      ((c & 0x80) == 0)    n = 1;
    else if ((c & 0xE0) == 0xC0) n = 2;
    else if ((c & 0xF0) == 0xE0) n = 3;
    else if ((c & 0xF8) == 0xF0) n = 4;
    else throw std::runtime_error("utf8_length: invalid UTF-8");

    if (end - begin < n) {
      throw std::runtime_error("utf8_length: string too short");
    }
    for (int i = 1; i < n; ++i) {
      if ((begin[i] & 0xC0) != 0x80) {
        throw std::runtime_error("utf8_length: expected continuation byte");
      }
    }
    len += n;
    begin += n;
  }
  return len;
}
1
  • This is basically the same as Marcelo Cantos' answer above and only works for very simple strings that don't have multi-codepoint characters
    – phuclv
    Feb 25 at 4:16
5

You should probably take the advice of Omry and look into a specialized library for this. That said, if you just want to understand the algorithm to do this, I'll post it below.

Basically, you can convert your string into a wider-element format, such as wchar_t. Note that wchar_t has a few portability issues, because wchar_t is of varying size depending on your platform. On Windows, wchar_t is 2 bytes, and therefore ideal for representing UTF-16. But on UNIX/Linux, it's four-bytes and is therefore used to represent UTF-32. Therefore, for Windows this will only work if you don't include any Unicode codepoints above 0xFFFF. For Linux you can include the entire range of codepoints in a wchar_t. (Fortunately, this issue will be mitigated with the C++0x Unicode character types.)

With that caveat noted, you can create a conversion function using the following algorithm:

template <class OutputIterator>
inline OutputIterator convert(const unsigned char* it, const unsigned char* end, OutputIterator out) 
{
    while (it != end) 
    {
        if (*it < 192) *out++ = *it++; // single byte character
        else if (*it < 224 && it + 1 < end && *(it+1) > 127) { 
            // double byte character
            *out++ = ((*it & 0x1F) << 6) | (*(it+1) & 0x3F);
            it += 2;
        }
        else if (*it < 240 && it + 2 < end && *(it+1) > 127 && *(it+2) > 127) { 
            // triple byte character
            *out++ = ((*it & 0x0F) << 12) | ((*(it+1) & 0x3F) << 6) | (*(it+2) & 0x3F);
            it += 3;
        }
        else if (*it < 248 && it + 3 < end && *(it+1) > 127 && *(it+2) > 127 && *(it+3) > 127) { 
            // 4-byte character
            *out++ = ((*it & 0x07) << 18) | ((*(it+1) & 0x3F) << 12) |
                ((*(it+2) & 0x3F) << 6) | (*(it+3) & 0x3F);
            it += 4;
        }
        else ++it; // Invalid byte sequence (throw an exception here if you want)
    }

    return out;
}

int main()
{
    std::string s = "\u00EAtre";
    cout << s.length() << endl;

    std::wstring output;
    convert(reinterpret_cast<const unsigned char*> (s.c_str()), 
        reinterpret_cast<const unsigned char*>(s.c_str()) + s.length(), std::back_inserter(output));

    cout << output.length() << endl; // Actual length
}

The algorithm isn't fully generic, because the InputIterator needs to be an unsigned char, so you can interpret each byte as having a value between 0 and 0xFF. The OutputIterator is generic, (just so you can use an std::back_inserter and not worry about memory allocation), but its use as a generic parameter is limited: basically, it has to output to an array of elements large enough to represent a UTF-16 or UTF-32 character, such as wchar_t, uint32_t or the C++0x char32_t types. Also, I didn't include code to convert character byte sequences greater than 4 bytes, but you should get the point of how the algorithm works from what's posted.

Also, if you just want to count the number of characters, rather than output to a new wide-character buffer, you can modify the algorithm to include a counter rather than an OutputIterator. Or better yet, just use Marcelo Cantos' answer to count the first-bytes.

2
  • on the nitpicking front, what makes you think the "être" string will use UTF8 encoding ? I believe it's non-standard in C/C++ to use non-ascii in source code (and indeed, some compilers will choose another encoding).
    – Bahbar
    Oct 31, 2010 at 16:15
  • @Bahbar, good point. It should in fact use \u hex notation. Oct 31, 2010 at 16:46
4

I recommend you use UTF8-CPP. It's a header-only library for working with UTF-8 in C++. With this lib, it would look something like this:

int LenghtOfUtf8String( const std::string &utf8_string ) 
{
    return utf8::distance( utf8_string.begin(), utf8_string.end() ); 
}

(Code is from the top of my head.)

2
  • 1
    is it cross-platform solution?
    – JavaRunner
    May 31, 2013 at 16:30
  • 1
    A quick look at the link tells: "The library was designed to be: ... Portable: ... across different platforms and compilers. The only non-portable code is a small section that declares unsigned integers of different sizes: three typedefs. ... can be changed ... if they don't match [the] platform. The default ... should work for Windows (both 32 and 64 bit), and most 32 bit and 64 bit Unix derivatives."
    – Sz.
    Jul 24, 2019 at 23:28
3

Most of my personal C library code has only been really tested in English, but here is how I've implemented my utf-8 string length function. I originally based it on the bit pattern described in this wiki page table. Now this isn't the most readable code, but my intent was to remove any branching from the loop. Also sorry for this being C code when asking for C++, it should translate over to std::string in C++ pretty easily though with some slight modifications. The below functions are copied from my website if you're interested.

size_t utf8len(const char* const str) {
    size_t len = 0;
    for (size_t i = 0; *str != 0; ++len) {
        int v0 = (*str & 0x80) >> 7;
        int v1 = (*str & 0x40) >> 6;
        int v2 = (*str & 0x20) >> 5;
        int v3 = (*str & 0x10) >> 4;
        str += 1 + v0 * v1 + v0 * v1 * v2 + v0 * v1 * v2 * v3;
    }
    return len;
}

Note that this does not validate any of the bytes (much like all the other suggested answers here). Personally I would separate string validation out of my string length function as that is not it's responsibility. If we were to move string validation to another function we could have the validation done something like the following.

bool utf8valid(const char* const str) {
    if (str == NULL)
        return false;
    const char* c = str;
    bool valid = true;
    for (size_t i = 0; c[0] != 0 && valid;) {
        valid = (c[0] & 0x80) == 0
            || ((c[0] & 0xE0) == 0xC0 && (c[1] & 0xC0) == 0x80)
            || ((c[0] & 0xF0) == 0xE0 && (c[1] & 0xC0) == 0x80 && (c[2] & 0xC0) == 0x80)
            || ((c[0] & 0xF8) == 0xF0 && (c[1] & 0xC0) == 0x80 && (c[2] & 0xC0) == 0x80 && (c[3] & 0xC0) == 0x80);
        int v0 = (c[0] & 0x80) >> 7;
        int v1 = (c[0] & 0x40) >> 6;
        int v2 = (c[0] & 0x20) >> 5;
        int v3 = (c[0] & 0x10) >> 4;
        i += 1 + v0 * v1 + v0 * v1 * v2 + v0 * v1 * v2 * v3;
        c = str + i;
    }
    return valid;
}

If you are going for readability, I'll admit that other suggestions are a quite bit more readable haha!

1
  • I could improve the execution speed of your formula when I do this: const auto v01 = v0 & v1 and str += 1 + ((v01 << v2) | (v01 & v3));. It's less readable but uses less and faster instructions.
    – Max Brauer
    Mar 28 at 9:00
1

try to use an encoding library like iconv. it probably got the api you want.

an alternative is to implement your own utf8strlen which determines the length of each codepoint and iterate codepoints instead of characters.

1

A slightly lazy approach would be to only count lead bytes, but visit every byte. This saves the complexity of decoding the various lead byte sizes, but obviously you pay to visit all the bytes, though there usually aren't that many (2x-3x):

size_t utf8Len(std::string s)
{
  return std::count_if(s.begin(), s.end(),
    [](char c) { return (static_cast<unsigned char>(c) & 0xC0) != 0x80; } );
}

Note that certain code values are illegal as lead bytes, those that represent bigger values than the 20 bits needed for extended unicode, for example, but then the other approach would not know how to deal with that code, anyway.

0

UTF-8 CPP library has a function that does just that. You can either include the library into your project (it is small) or just look at the function. http://utfcpp.sourceforge.net/

char* twochars = "\xe6\x97\xa5\xd1\x88";
size_t dist = utf8::distance(twochars, twochars + 5);
assert (dist == 2);
0

This code I'm porting from php-iconv to c++, you need use iconv first, hope usefull:

// porting from PHP
// http://lxr.php.net/xref/PHP_5_4/ext/iconv/iconv.c#_php_iconv_strlen
#define GENERIC_SUPERSET_NBYTES 4
#define GENERIC_SUPERSET_NAME   "UCS-4LE"

UInt32 iconvStrlen(const char *str, size_t nbytes, const char* encode)
{
    UInt32 retVal = (unsigned int)-1;

    unsigned int cnt = 0;

    iconv_t cd = iconv_open(GENERIC_SUPERSET_NAME, encode);
    if (cd == (iconv_t)(-1))
        return retVal;

    const char* in;
    size_t  inLeft;

    char *out;
    size_t outLeft;

    char buf[GENERIC_SUPERSET_NBYTES * 2] = {0};

    for (in = str, inLeft = nbytes, cnt = 0; inLeft > 0; cnt += 2) 
    {
        size_t prev_in_left;
        out = buf;
        outLeft = sizeof(buf);

        prev_in_left = inLeft;

        if (iconv(cd, &in, &inLeft, (char **) &out, &outLeft) == (size_t)-1) {
            if (prev_in_left == inLeft) {
                break;
            }
        }
    }
    iconv_close(cd);

    if (outLeft > 0)
        cnt -= outLeft / GENERIC_SUPERSET_NBYTES;

    retVal = cnt;
    return retVal;
}

UInt32 utf8StrLen(const std::string& src)
{
    return iconvStrlen(src.c_str(), src.length(), "UTF-8");
}
0

Just another naive implementation to count chars in UTF-8 string

int utf8_strlen(const string& str)
{
    int c,i,ix,q;
    for (q=0, i=0, ix=str.length(); i < ix; i++, q++)
    {
        c = (unsigned char) str[i];
        if      (c>=0   && c<=127) i+=0;
        else if ((c & 0xE0) == 0xC0) i+=1;
        else if ((c & 0xF0) == 0xE0) i+=2;
        else if ((c & 0xF8) == 0xF0) i+=3;
        //else if (($c & 0xFC) == 0xF8) i+=4; // 111110bb //byte 5, unnecessary in 4 byte UTF-8
        //else if (($c & 0xFE) == 0xFC) i+=5; // 1111110b //byte 6, unnecessary in 4 byte UTF-8
        else return 0;//invalid utf8
    }
    return q;
}
0

Most (if not all) of the other answers only give the number of code points and completely fail for combining characters, emojis or more complex scripts. For example here's an example output from user2781185's solution above after modifying slightly for a demo on Godbolt:

Length (char-values):  5, length (code points):  4. String: café
Length (char-values):  6, length (code points):  5. String: café
Length (char-values): 15, length (code points):  5. String: 가각
Length (char-values): 24, length (code points):  8. String: ဂ︀င︀⋚︀丸︀
Length (char-values): 47, length (code points): 13. String: 🏳️‍🌈👨‍👩‍👦‍👦🇪🇺
Length (char-values): 74, length (code points): 21. String: 👨‍👩‍👦‍👦😶‍🌫️👩🏻‍❤️‍💋‍👩🏿
Length (char-values): 21, length (code points):  7. String: ফোল্ডার
Length (char-values): 18, length (code points):  8. String: dര്‍g1️⃣
Length (char-values): 18, length (code points):  6. String: Xല്‍🇺🇳

As you can see, the lengths returned are just the number of code points and made no relation whatsoever to what users see (“user-perceived characters”). Even the 2 café strings are different

To get the actual number of visible characters (called glyphs) you have to use a proper library like Boost.Unicode/Boost.Text/Boost.Locale or the official ICU from the Unicode Consortium to normalize the string to a non-combining form like NFC or NFKC first, then count the length in glyph

This is the sample code on how to do that:

#include <unicode/schriter.h>
#include <unicode/brkiter.h>

#include <iostream>
#include <cassert>
#include <memory>

int main()
{
    const UnicodeString str(L"नमस्ते café café 😶‍🌫️🏃🏻‍♀️");
    UErrorCode errorCode;
    nfkc.normalize(str, errorCode); // ALWAYS NORMALIZE THE STRINGS FIRST

    {
        UErrorCode err = U_ZERO_ERROR;
        std::unique_ptr<BreakIterator> iter(
            BreakIterator::createCharacterInstance(Locale::getDefault(), err));
        assert(U_SUCCESS(err));
        iter->setText(str);

        int count = 0;
        while(iter->next() != BreakIterator::DONE) ++count;
        std::cout << count << std::endl;
    }

    return 0;
}

Another probably simply library for that purpose is yhirose/cpp-unicodelib

std::u32string s = U"hello☺😆";
auto normalized = unicode::to_nfkc(s.c_str(), s.length());
std::cout << "Length: "
          << unicode::grapheme_count(normalized.c_str(), normalized.length())

See also:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.