1

I am fetching text from a utf-8 text file, and doing it by chunks to increase performance.

std::ifstream.read(myChunkBuff_str, myChunkBuff_str.length())

Here is a more detailed example

I am getting around 16 thousand characters with each chunk. My next step is to convert this std::string into something that can allow me to work on these "complex characters" individually, thus converting that std::string into std::wstring.

I am using the following function for converting, taken from here:

#include <string>
#include <codecvt>
#include <locale>

std::string narrow (const std::wstring& wide_string)
{
    std::wstring_convert <std::codecvt_utf8 <wchar_t>, wchar_t> convert;
    return convert.to_bytes (wide_string);
}

std::wstring widen (const std::string& utf8_string)
{
    std::wstring_convert <std::codecvt_utf8 <wchar_t>, wchar_t> convert;
    return convert.from_bytes (utf8_string);
}

However, at its end of the chunk one of the Russian characters might be cut-off, and the conversion will fail, with an std::range_error exception.

For example, in UTF-8 "привет" takes 15 chars and "приве" takes 13 chars. So, if my chunk was hypothetically 14, the 'т' would be partially missing, and the conversion would throw exception.

Question:

How to detect these partially-loaded character? ('т' in this case) This would allow me to convert without it, and perhaps shift the next chunk a bit earlier than planned, to include this problematic 'т' next time?

I don't want to try or catch around these functions, as try/catch might slow me down the program. It also doesn't tell me "how much of character was missing for the conversion to actually succeed".

I know about wstring_convert::converted() but it's not really useful if my program crashes before I get to it

7
  • 1
    I would just write my own conversion routine, UTF-8 is easy to interpret. That way I'd have full control over this issue, and all the other little consdierations there are when converting UTF-8 to UTF-16 (i.e. what to do about the various kinds of illegal sequences)
    – john
    Sep 14, 2018 at 20:39
  • ugh, that sounds too hard for me )) I would assume there are many edgecases
    – Kari
    Sep 14, 2018 at 20:39
  • 1
    @Kan really it's only about 20 to 30 lines of code, wikipedia will tell you all the technical details. But only a suggestion, someone may have a better answer to your question.
    – john
    Sep 14, 2018 at 20:43
  • You could use something like Qt's QString which handles this nicely (assuming you feed it complete characters - so just read everything (memory is cheap ;) ). Btw, detecting an incomplete utf-8 character and backing up to the previous complete character is not hard. Sep 14, 2018 at 20:45
  • 1
    I guess you could use std::codecvt_utf8<wchar_t> directly.
    – Galik
    Sep 14, 2018 at 22:46

1 Answer 1

4

You could do this using a couple of functions. UTF-8 has a way to detect the beginning of a multibyte character and (from the beginning) the size of the multibyte character.

So two functions:

// returns zero if this is the first byte of a UTF-8 char
// otherwise non-zero.
static unsigned is_continuation(char c)
{
    return (c & 0b10000000) && !(c & 0b01000000);
}

// if c is the *first* byte of a UTF-8 multibyte character, returns 
// the total number of bytes of the character.
static unsigned size(const unsigned char c)
{
    constexpr static const char u8char_size[] =
    {
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
        , 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
        , 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
        , 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
        , 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
        , 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
        , 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
        , 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
        , 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
        , 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
        , 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
        , 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
        , 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2
        , 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2
        , 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3
        , 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 6, 6, 0, 0
    };
    return u8char_size[(unsigned char)c];
}

You could track back from the end of your buffer until is_continuation(c) is false. Then check if size(c) of the current UTF-8 char is longer than the end of the buffer.

Disclaimer - last time I looked these functions were working but have not used them in a while.

Edit: to add.

If you feel like doing th whole thing manually I may as well post the code to convert a UTF-8 multibyte character to a UTF-16 multibyte or a UTF-32 char.

UTF-32 Is easy:

// returns a UTF-32 char from a `UTF-8` multibyte
// character pointed to by cp
static char32_t char32(const char* cp)
{
    auto sz = size(*cp); // function above

    if(sz == 1)
        return *cp;

    char32_t c32 = (0b0111'1111 >> sz) & (*cp);

    for(unsigned i = 1; i < sz; ++i)
        c32 = (c32 << 6) | (cp[i] & 0b0011'1111);

    return c32;
}

UTF-16 Is a little more tricky:

// UTF-16 characters can be 1 or 2 characters wide...
using char16_pair = std::array<char16_t, 2>;

// outputs a UTF-16 char in cp16 from a `UTF-8` multibyte
// character pointed to by cp
//
// returns the number of characters in this `UTF-16` character
// (1 or 2).
static unsigned char16(const char* cp, char16_pair& cp16)
{
    char32_t c32 = char32(cp);

    if(c32 < 0xD800 || (c32 > 0xDFFF && c32 < 0x10000))
    {
        cp16[0] = char16_t(c32);
        cp16[1] = 0;
        return 1;
    }

    c32 -= 0x010000;

    cp16[0] = ((0b1111'1111'1100'0000'0000 & c32) >> 10) + 0xD800;
    cp16[1] = ((0b0000'0000'0011'1111'1111 & c32) >> 00) + 0xDC00;

    return 2;
}
3
  • note that 5 and 6-character UTF-8 sequences are prohibited
    – phuclv
    Sep 15, 2018 at 4:24
  • inside UTF-32 conversion function char32() I think you forgot to dereference the cp pointer at line auto sz = size(cp); // function above
    – Kari
    Sep 15, 2018 at 13:03
  • 1
    @Kari Thanks! In the code I took this from I have both overloads taking the pointer and the direct value so I missed that one.
    – Galik
    Sep 15, 2018 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.