Questions tagged [surrogate-pairs]
Unicode characters with code above 0xFFFF, are encoded in UTF-16 by pairs of 16-bit code units called **surrogate pairs**.
112
questions
0
votes
1
answer
75
views
How can the a surrogate code point (0xD800-0xDFFF) in C++ cause program errors?
The following is a description of the cppreference:
cppreference
Range of universal character names
...
If a universal character name corresponds surrogate code point (the range 0xD800-0xDFFF, ...
0
votes
0
answers
72
views
python json5 and json package inconsistent deal with surrogate pair
#!/opt/homebrew/bin/python3
#-*-coding:utf8-*-
#python version 3.11.4
import json5
import json
d = json5.loads('{"\\ud83d\\ude03": "ok"}')
print(d) # output: {'\ud83d\ude03': 'ok'...
0
votes
0
answers
45
views
Python MySQL doesn't encode surrogates for query parameters
Running tried this with both Python3.7 and Python3.8, with mysql-connector-python 8.0.13 and 8.1.0
MySQL 5.7.42
Collation on the database is set to 'utf8mb4_unicode_520_ci'
Connection from Python is:
...
0
votes
0
answers
44
views
Why does a keyboard-entered super Unicode string change after being passed to a method?
Why is the string "🍷Hello" entered by Scanner(System.in) printed incorrectly after calling func(str2)?
The java code is as follows:
package test;
import java.util.Scanner;
public class ...
1
vote
1
answer
38
views
How to chunk a string containing characters with high code point?
I have this string : "397"
The special character has the code point : 1114111
When I chunk the string : "397".match(/.{2}/g)
I have this result : ['3\uDBFF', '\uDFFF9']
But I ...
1
vote
1
answer
81
views
How to convert surrogate pairs into hexadecimal, and vice-versa in Python?
How would I convert characters which are surrogate pairs into hexadecimal?
I've found that using hex() and ord() works for characters with a single code point, such as emojis like "😀". For ...
2
votes
2
answers
328
views
Replace all unmatched surrogate pairs with replacement character in JavaScript string
I have a JavaScript string that I'm writing to a file. I need to replace any unmatched surrogate pairs with the replacement character. Is there some regex character class that only matches unpaired ...
0
votes
3
answers
174
views
XSL Transformation, emoji and attributes
I'm encountering issue with emojis when trying to generate html output using xsl transformation under certain circumstances.
For instance, I've tested following xsl with different transformation ...
1
vote
0
answers
195
views
Can Windows' std::wcout display UTF-16 wchar_t surrogate pairs as a unicode character?
[LIVE]
\U0001F34C\U0002008A on cout : 🍌𠂊
\U0001F34C\U0002008A on wcout :
wcsnlen : 4
1 : 0xd83c
2 : 0xdf4c
3 : 0xd840
4 : 0xdc8a
wcsrtombs to char on cout ...
0
votes
1
answer
117
views
Unicode conversion in XML failed
In response of a webservice call, I am getting an XML
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns1:executeAPI xmlns:ns1="http://...
-4
votes
1
answer
248
views
c++: how to remove surrogate unicode values from string? [closed]
how do you remove surrogate values from a std::string in c++?
looking for regular expression like this:
string pattern = u8"[\uD800-\uDFFF]";
regex regx(pattern);
name = regex_replace(name, ...
0
votes
0
answers
5k
views
Bulding Python 3 command from bash gives UnicodeEncodeError: 'utf-8' codec can't encode characters in position surrogates not allowed
This is a silly example of the problem, I have a bash variable I am building inside the string to run with python -c "$string", but there are some characters as 🇯🇵 which are breaking my ...
0
votes
1
answer
386
views
XML marshalling of surrogate pairs
I have came across a strange behavior of marshaller when surrogate pairs are involved. Why JAXB marshaller adds unnecessary (and invalid) XML entity? When I try to marshall the following:
\uD83D\...
-1
votes
1
answer
169
views
Can the the string with 2 different codepoints, which form 1 character, consider a surrogate pair?
As far as I see, surrogate pair is two 16-bit codepoints for 1 character. Surrogate pair uses for "big" codepoints, which can't be written in 16 bits.
So, my question is... Can this consider ...
2
votes
1
answer
419
views
unicode utf-16 surrogate pair print problem
In Visual Studio /C++ declared a wstring c and filled it with a surrogate pair ( Unicode 0001F01C = Mahong tile )
std::cout << std::hex << 16;
std::cout << "...
2
votes
0
answers
171
views
Why are utf16 / surrogate pairs emojis not displayed correctly in console.log() when run through yarn on Windows? (npm and node directly are fine)
Somehow running console.log('Thunderbirds are go! 🚀') via yarn / node does not display the emoji but outputs � instead.
I'm on Windows Terminal / Git Bash which in itself has no problems displaying ...
1
vote
1
answer
427
views
What unicode character (emoji) it was?
I have that string in my text file: ├░┬č┬Ź┬ć
What is known is that it was emoji or at least some surrogate character/character created by javascript string of length 2 or 4
Because of some reason it ...
0
votes
2
answers
208
views
python function to transform data to JSON
Can I check how do we convert the below to a dictionary?
code.py
message = event['Records'][0]['Sns']['Message']
print(message)
# this gives the below and the type is <class 'str'>
{
"...
-1
votes
1
answer
3k
views
C# UNITY Unicode Emoji Display
I am receiving emoji in the below format from a source system.
'\u2764' for ❤
'\ud83d\udc4d' for 😂
I need to know how i can convert this, so it displays as Proper emoji in Unity Debug log and in ...
0
votes
1
answer
649
views
Why does the UTF-16 bytes for emoji smiley and emoji flag together looks different than a sequence of their individual UTF-16 bytes?
Following is from the Visual Studio's C# Interactive Compiler:
> BitConverter.ToString(Encoding.BigEndianUnicode.GetBytes("😀"))
"D8-3D-DE-00"
> BitConverter.ToString(...
0
votes
1
answer
102
views
How to print surrogate chars as ints in Java
I have this:
char c = "\ud804\udef4".charAt(0);
char d = "\ud804\udef4".charAt(1);
How will I print c and d as hex Strings ?
I want d804 for c and def4 for d.
3
votes
3
answers
3k
views
How to display Unicode character names and their hexadecimal codes with PowerShell?
Since the default Windows PowerShell console fonts don't support Emojis, I'd like to display their surrogate pair hexadecimal codes and ideally also their Unicode character names for debugging ...
1
vote
0
answers
87
views
How to convert surrogate pairs read from txt files back to emojis in python 3? [duplicate]
I have a few txt files to read where there are string such as:
"Yes! Sardines in a can distancing! \uD83E\uDD23"
Problem is that : when I'm doing
"Yes! Sardines in a can distancing! \...
1
vote
1
answer
493
views
Spliting an emoji sequence in powershell
I have a text box that will be filled with emoji only. No spaces or characters of any kind. I need to split these emoji in order to identify them. This is what I have tried:
function emoji_to_unicode(...
0
votes
0
answers
182
views
How to detect if string contains any supplementary characters in PHP?
From what I understand so far, supplementary characters (or "surrogate pairs") are defined in a range from 0xd800 to 0xdbff for the first char, and from 0xdc00 and 0xdfff for the second char.
So I'm ...
1
vote
0
answers
330
views
Conversion of UTF16 to UTF32 - Invalid surrogate pair
While converting an array of UTF16 to UTF32, if I come across a high surrogate and if the next value is neither a high surrogate nor a low surrogate, should we invalidate both the values in UTF16 ...
1
vote
0
answers
60
views
How to Convert Surrogate Pairs in Servlet Response? How can I read Surrogate Pairs?
I have emoji as Surrogate Pairs received in the response.
I want to change these Surrogates Pairs to a Unicode so that the WebSphere Portal 7 will understand the Unicodes.
Added a filter to modify ...
2
votes
1
answer
523
views
How to Convert UTF-16 Surrogate Decimal to UNICODE in Java
I have some string data like
� ;� ;
These are surrogate pairs in UTF 16 in decimal format.
How can I convert them to Unicode Code Points in Java, so that my client can ...
0
votes
1
answer
2k
views
Converting surrogate pairs to emoji - python3
I found the solution to similar question on the other topic, but unfortunately it's not working for me.
Here is my problem:
I'm making dataframe from the surrogatepairs unicodes which I'd like to ...
7
votes
1
answer
2k
views
Emojis to/from codepoints in Javascript
In a hybrid Android/Cordova game that I am creating I let users provide an identifier in the form of an Emoji + an alphanumeric - i.e. 0..9,A..Z,a..z - name. For example
🙋️Stackoverflow
Server-...
1
vote
1
answer
423
views
Python Unicode - What Characters Can Be Printed in Windows Console?
Which Unicode characters can be printed in a Windows console from Python?
The following code
for code in range(1114112):
print(chr(code), end=",")
gives unimpressive results, including an error:...
3
votes
1
answer
1k
views
PHP json_decode does not work with single unpaired surrogate caused by Node 12 well-formed JSON.stringify
Node 12 now uses Well-formed JSON.stringify which outputs escape sequences for lone surrogates. PHP is not able to json_decode this when there is a lone surrogate that is escaped.
Take the following ...
0
votes
0
answers
279
views
How to encode unicode surrogate pairs, to write to file
I'm creating an compression / encryption program that utilises (nearly) all unicode characters, and then writes the data to a file. However, to write to the file, I need to encode the characters to ...
2
votes
3
answers
987
views
How to reverse strings that contain surrogate pairs in Dart?
I was playing with algorithms using Dart and as I actually followed TDD, I realized that my code has some limitations.
I was trying to reverse strings as part of an interview problem, but I couldn't ...
0
votes
1
answer
579
views
Replacing emoji from a string
I am new to vb net. i am trying to process a string containing emoji but I cannot do it. my string looks like this:💛💛💙💛 I want to replace 💙 with 💛 what i am doing is using string.remove and ...
0
votes
1
answer
211
views
Should StringIO(HighSurrogate) throw an error in Python 2.7?
When I run this Python 2.7 code (edit: updated the code)
import io
x = io.StringIO(u'\ud801')
CPython runs fine, but IronPython throws the following error:
UnicodeEncodeError:
Unable to translate ...
2
votes
2
answers
290
views
Python unicode indexing shows different character
I have a Unicode string in a "narrow" build of Python 2.7.10 containing a Unicode character. I'm trying to use that Unicode character as a lookup in a dictionary, but when I index the string to get ...
3
votes
1
answer
2k
views
C# - Regular expression to find a surrogate pair of a unicode codepoint from any string?
I am trying to parse a message that possibly contains emojis in it. An example message that could be received looks like:
{"type":"chat","msg":"UserName:\u00a0\ud83d\ude0b \n"}
What should match is \...
3
votes
2
answers
2k
views
Unicode surrogate pairs and String.fromCodePoint() — JavaScript
I'm dealing with raw strings containing escape sequences for surrogate halves of UTF astral symbols. (I think I got that lingo right…)
console.log("\uD83D\uDCA9")
// => 💩
Let's use the above ...
2
votes
1
answer
389
views
Python 2.7: Strange Unicode behavior
I am experiencing the following behavior in Python 2.7:
>>> a1 = u'\U0001f04f' #1
>>> a2 = u'\ud83c\udc4f' #2
>>> a1 == a2 #3
False
>>> a1.encode('utf8') == a2....
1
vote
1
answer
92
views
Write surrogate pairs to file using Haskell
This is the code I have:
import qualified System.IO as IO
writeSurrogate :: IO ()
writeSurrogate = do
IO.writeFile "/home/sibi/surrogate.txt" ['\xD800']
Executing the above code gives error:
...
0
votes
0
answers
47
views
get unicode graphemes as unsplitted item with python2.7
any idea, if it is possible with regex (python 2.7) to get uniq chars unspitted into surrogate pairs for unicode graphemes?
According This Example this is possible with python 3.x.
See here:
>>...
0
votes
1
answer
482
views
PYTHON RE Dont split UNICODE Chars into surrogate pairs while matching
who know, if it is possible to forbidden regex while macthing splitting code points into surrogate pairs.
See the following example:
How it is now:
$ te = u'\U0001f600\U0001f600'
$ flags1 = regex....
2
votes
1
answer
2k
views
Convert from '\ud835' format to "𝓛" in c# [UWP]
I have a string with some wonky characters (for example) "𝓛𝓮𝓪𝓭 𝓑𝓪𝓬𝓴𝓮𝓷𝓭". I need to check if a List contains the first item in the string. But if I index it, it always becomes \ud835. After ...
0
votes
0
answers
18
views
How to extract a list of all 18 character entries after specific phrase in string using RegExr?
I managed to extract a list of the text within square brackets within an emoji list I have here:
https://regexr.com/3sqk1
But now I need to extract the equivalent decimalSurrogateHtml pairs for each ...
10
votes
1
answer
12k
views
What are surrogate characters in UTF-8?
I have a strange validation program that validates wheather a utf-8 string is a valid host name(Zend Framework Hostname valdiator in PHP). It allows IDNs(internationalized domain names). It will ...
1
vote
1
answer
951
views
What is a Unicode safe replica of String.IndexOf(string input) that can handle Surrogate Pairs?
I am trying to figure out an equivalent to C# string.IndexOf(string) that can handle surrogate pairs in Unicode characters.
I am able to get the index when only comparing single characters, like in ...
2
votes
1
answer
404
views
Is String.Replace(string,string) Unicode Safe in regards to Surrogate Pairs?
I am trying to figure out the best way to create a function that is equivalent to String.Replace("oldValue","newValue");
that can handle surrogate pairs.
My concern is that if there are surrogate ...
1
vote
1
answer
444
views
How to reveal the surrogate pairs in String in perl
I am working a perl code base to validate customer input, my goal is to block surrogate characters.
My thought is first encoding the customer input as UTF-16 and
foreach my $messageChar (@...
0
votes
0
answers
9k
views
UnicodeEncodeError: 'utf-8' codec can't encode character '\udc43' in position 1: surrogates not allowed
I have a list containing placenames and I want to create another array, initially empty, and then iterate the list of placenames and fill up my empty array with these placenames.
For example, my ...