I'm testing how some of my code handles bad data, and I need a few series of bytes that are invalid UTF-8.
Can you post some, and ideally, an explanation of why they are bad/where you got them?
In PHP:
$examples = array(
'Valid ASCII' => "a",
'Valid 2 Octet Sequence' => "\xc3\xb1",
'Invalid 2 Octet Sequence' => "\xc3\x28",
'Invalid Sequence Identifier' => "\xa0\xa1",
'Valid 3 Octet Sequence' => "\xe2\x82\xa1",
'Invalid 3 Octet Sequence (in 2nd Octet)' => "\xe2\x28\xa1",
'Invalid 3 Octet Sequence (in 3rd Octet)' => "\xe2\x82\x28",
'Valid 4 Octet Sequence' => "\xf0\x90\x8c\xbc",
'Invalid 4 Octet Sequence (in 2nd Octet)' => "\xf0\x28\x8c\xbc",
'Invalid 4 Octet Sequence (in 3rd Octet)' => "\xf0\x90\x28\xbc",
'Invalid 4 Octet Sequence (in 4th Octet)' => "\xf0\x28\x8c\x28",
'Valid 5 Octet Sequence (but not Unicode!)' => "\xf8\xa1\xa1\xa1\xa1",
'Valid 6 Octet Sequence (but not Unicode!)' => "\xfc\xa1\xa1\xa1\xa1\xa1",
);
From http://www.php.net/manual/en/reference.pcre.pattern.modifiers.php#54805
The idea of patterns of ill-formed byte-sequences can be gotten from the table of well-formed byte sequences. See "Table 3-7. Well-Formed UTF-8 Byte Sequences" in the Unicode Standard 6.2.
Code Points First Byte Second Byte Third Byte Fourth Byte
U+0000 - U+007F 00 - 7F
U+0080 - U+07FF C2 - DF 80 - BF
U+0800 - U+0FFF E0 A0 - BF 80 - BF
U+1000 - U+CFFF E1 - EC 80 - BF 80 - BF
U+D000 - U+D7FF ED 80 - 9F 80 - BF
U+E000 - U+FFFF EE - EF 80 - BF 80 - BF
U+10000 - U+3FFFF F0 90 - BF 80 - BF 80 - BF
U+40000 - U+FFFFF F1 - F3 80 - BF 80 - BF 80 - BF
U+100000 - U+10FFFF F4 80 - 8F 80 - BF 80 - BF
Here are the examples generated from U+24B62. I used them for a bug report: Bug #65045 mb_convert_encoding breaks well-formed character
// U+24B62: "\xF0\xA4\xAD\xA2"
"\xF0\xA4\xAD" ."\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD\xA2"
"\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD\xA2"."\xF0\xA4\xAD"
The oversimplification of range of trailing bytes([0x80, 0xBF]) can be seen in the various libraries.
// U+0800 - U+0FFF
\xE0\x80\x80
// U+D000 - U+D7FF
\xED\xBF\xBF
// U+10000 - U+3FFFF
\xF0\x80\x80\x80
// U+100000 - U+10FFFF
\xF4\xBF\xBF\xBF
Invalid one byte sequences
Just to give this simplest example more clearly, if you look at the table at: https://stackoverflow.com/a/17199164/895245 then any single byte outside of 0x00 - 0x7F is invalid following single bytes seu are invalid because:
You can check if some bytes are valid UTF-8 as per: https://superuser.com/questions/649834/is-there-a-linux-command-to-find-out-if-a-file-is-utf-8 iconv is a good one e.g.:
$ printf '\x80' | iconv -f utf8 -t utf8
iconv: illegal input sequence at position 0
$ printf '\xc2' | iconv -f utf8 -t utf8
iconv: incomplete character or shift sequence at end of buffer
$ printf '\xc0' | iconv -f utf8 -t utf8
iconv: illegal input sequence at position 0
Tested on Ubuntu 25.04.
This might not be exactly what OP asked but it's somewhat related :
if you happen to already have byte ordinance values (0 - 255) and wanna know whether a byte# is a valid UTF-8 starting point byte or not, I came up with this strange unified formula that returns a 1 (true) or 0 (false) :
fn newUTF8start_v1(_) {
return 118^(_ < 194) < (246 - _) # "^" := exponentiation
}
if you don't wanna deal with taking powers of anything, then this alternative expression yields identical results :
fn newUTF8start_v2(_) {
return 117 * (_ < 194) < (245 - _)
}
byte 194 => 0xC2 \302 (first UTF-8 multi-byte lead byte)byte 245 => 0xF5 \365 (first UTF-8 invalid byte greater than all lead bytes)
As for
117
…. that in itself isn't a particular byte, but simply the unsigned byte ordinal difference between
0xF4 \364 (largest UTF-8 lead byte)
and
0x7F \177 (largest ASCII byte - "DEL")
Fuzz Testing - generate a random sequence of octets. Most likely you'll get some illegal sequences sooner than later.
srand() advice is a good idea, it might help other people here.
Commented
Nov 22, 2017 at 8:20