Questions tagged [surrogate-pairs]

Unicode characters with code above 0xFFFF, are encoded in UTF-16 by pairs of 16-bit code units called **surrogate pairs**.

Filter by
Sorted by
Tagged with
0 votes
1 answer
75 views

How can the a surrogate code point (0xD800-0xDFFF) in C++ cause program errors?

The following is a description of the cppreference: cppreference Range of universal character names ... If a universal character name corresponds surrogate code point (the range 0xD800-0xDFFF, ...
lidajun's user avatar
0 votes
0 answers
72 views

python json5 and json package inconsistent deal with surrogate pair

#!/opt/homebrew/bin/python3 #-*-coding:utf8-*- #python version 3.11.4 import json5 import json d = json5.loads('{"\\ud83d\\ude03": "ok"}') print(d) # output: {'\ud83d\ude03': 'ok'...
Green's user avatar
  • 121
0 votes
0 answers
45 views

Python MySQL doesn't encode surrogates for query parameters

Running tried this with both Python3.7 and Python3.8, with mysql-connector-python 8.0.13 and 8.1.0 MySQL 5.7.42 Collation on the database is set to 'utf8mb4_unicode_520_ci' Connection from Python is: ...
TechnoSwiss's user avatar
0 votes
0 answers
44 views

Why does a keyboard-entered super Unicode string change after being passed to a method?

Why is the string "🍷Hello" entered by Scanner(System.in) printed incorrectly after calling func(str2)? The java code is as follows: package test; import java.util.Scanner; public class ...
Ken.Zhang's user avatar
1 vote
1 answer
38 views

How to chunk a string containing characters with high code point?

I have this string : "3􏿿97" The special character has the code point : 1114111 When I chunk the string : "3􏿿97".match(/.{2}/g) I have this result : ['3\uDBFF', '\uDFFF9'] But I ...
mbourd's user avatar
  • 35
1 vote
1 answer
81 views

How to convert surrogate pairs into hexadecimal, and vice-versa in Python?

How would I convert characters which are surrogate pairs into hexadecimal? I've found that using hex() and ord() works for characters with a single code point, such as emojis like "😀". For ...
radiantsoup0827's user avatar
2 votes
2 answers
328 views

Replace all unmatched surrogate pairs with replacement character in JavaScript string

I have a JavaScript string that I'm writing to a file. I need to replace any unmatched surrogate pairs with the replacement character. Is there some regex character class that only matches unpaired ...
user avatar
0 votes
3 answers
174 views

XSL Transformation, emoji and attributes

I'm encountering issue with emojis when trying to generate html output using xsl transformation under certain circumstances. For instance, I've tested following xsl with different transformation ...
morbac's user avatar
  • 311
1 vote
0 answers
195 views

Can Windows' std::wcout display UTF-16 wchar_t surrogate pairs as a unicode character?

[LIVE] \U0001F34C\U0002008A on cout : 🍌𠂊 \U0001F34C\U0002008A on wcout : wcsnlen : 4 1 : 0xd83c 2 : 0xdf4c 3 : 0xd840 4 : 0xdc8a wcsrtombs to char on cout ...
sandthorn's user avatar
  • 2,800
0 votes
1 answer
117 views

Unicode conversion in XML failed

In response of a webservice call, I am getting an XML <soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns1:executeAPI xmlns:ns1="http://...
keku keku's user avatar
-4 votes
1 answer
248 views

c++: how to remove surrogate unicode values from string? [closed]

how do you remove surrogate values from a std::string in c++? looking for regular expression like this: string pattern = u8"[\uD800-\uDFFF]"; regex regx(pattern); name = regex_replace(name, ...
Rashid Mousavy Khoshrou's user avatar
0 votes
0 answers
5k views

Bulding Python 3 command from bash gives UnicodeEncodeError: 'utf-8' codec can't encode characters in position surrogates not allowed

This is a silly example of the problem, I have a bash variable I am building inside the string to run with python -c "$string", but there are some characters as 🇯🇵 which are breaking my ...
Evandro Coan's user avatar
  • 8,720
0 votes
1 answer
386 views

XML marshalling of surrogate pairs

I have came across a strange behavior of marshaller when surrogate pairs are involved. Why JAXB marshaller adds unnecessary (and invalid) XML entity? When I try to marshall the following: \uD83D\...
Giron's user avatar
  • 350
-1 votes
1 answer
169 views

Can the the string with 2 different codepoints, which form 1 character, consider a surrogate pair?

As far as I see, surrogate pair is two 16-bit codepoints for 1 character. Surrogate pair uses for "big" codepoints, which can't be written in 16 bits. So, my question is... Can this consider ...
Heaven's user avatar
  • 41
2 votes
1 answer
419 views

unicode utf-16 surrogate pair print problem

In Visual Studio /C++ declared a wstring c and filled it with a surrogate pair ( Unicode 0001F01C = Mahong tile ) std::cout << std::hex << 16; std::cout << "...
user3443063's user avatar
  • 1,497
2 votes
0 answers
171 views

Why are utf16 / surrogate pairs emojis not displayed correctly in console.log() when run through yarn on Windows? (npm and node directly are fine)

Somehow running console.log('Thunderbirds are go! 🚀') via yarn / node does not display the emoji but outputs � instead. I'm on Windows Terminal / Git Bash which in itself has no problems displaying ...
Meldon's user avatar
  • 740
1 vote
1 answer
427 views

What unicode character (emoji) it was?

I have that string in my text file: ├░┬č┬Ź┬ć What is known is that it was emoji or at least some surrogate character/character created by javascript string of length 2 or 4 Because of some reason it ...
ElSajko's user avatar
  • 1,622
0 votes
2 answers
208 views

python function to transform data to JSON

Can I check how do we convert the below to a dictionary? code.py message = event['Records'][0]['Sns']['Message'] print(message) # this gives the below and the type is <class 'str'> { "...
Adam's user avatar
  • 1,183
-1 votes
1 answer
3k views

C# UNITY Unicode Emoji Display

I am receiving emoji in the below format from a source system. '\u2764' for ❤ '\ud83d\udc4d' for 😂 I need to know how i can convert this, so it displays as Proper emoji in Unity Debug log and in ...
Tamil Ninja's user avatar
0 votes
1 answer
649 views

Why does the UTF-16 bytes for emoji smiley and emoji flag together looks different than a sequence of their individual UTF-16 bytes?

Following is from the Visual Studio's C# Interactive Compiler: > BitConverter.ToString(Encoding.BigEndianUnicode.GetBytes("😀")) "D8-3D-DE-00" > BitConverter.ToString(...
Social Developer's user avatar
0 votes
1 answer
102 views

How to print surrogate chars as ints in Java

I have this: char c = "\ud804\udef4".charAt(0); char d = "\ud804\udef4".charAt(1); How will I print c and d as hex Strings ? I want d804 for c and def4 for d.
Ankur Agarwal's user avatar
3 votes
3 answers
3k views

How to display Unicode character names and their hexadecimal codes with PowerShell?

Since the default Windows PowerShell console fonts don't support Emojis, I'd like to display their surrogate pair hexadecimal codes and ideally also their Unicode character names for debugging ...
Nemo XXX's user avatar
  • 644
1 vote
0 answers
87 views

How to convert surrogate pairs read from txt files back to emojis in python 3? [duplicate]

I have a few txt files to read where there are string such as: "Yes! Sardines in a can distancing! \uD83E\uDD23" Problem is that : when I'm doing "Yes! Sardines in a can distancing! \...
Aniruddha's user avatar
1 vote
1 answer
493 views

Spliting an emoji sequence in powershell

I have a text box that will be filled with emoji only. No spaces or characters of any kind. I need to split these emoji in order to identify them. This is what I have tried: function emoji_to_unicode(...
birojow's user avatar
  • 53
0 votes
0 answers
182 views

How to detect if string contains any supplementary characters in PHP?

From what I understand so far, supplementary characters (or "surrogate pairs") are defined in a range from 0xd800 to 0xdbff for the first char, and from 0xdc00 and 0xdfff for the second char. So I'm ...
c00000fd's user avatar
  • 21.4k
1 vote
0 answers
330 views

Conversion of UTF16 to UTF32 - Invalid surrogate pair

While converting an array of UTF16 to UTF32, if I come across a high surrogate and if the next value is neither a high surrogate nor a low surrogate, should we invalidate both the values in UTF16 ...
Mounika's user avatar
  • 371
1 vote
0 answers
60 views

How to Convert Surrogate Pairs in Servlet Response? How can I read Surrogate Pairs?

I have emoji as Surrogate Pairs received in the response. I want to change these Surrogates Pairs to a Unicode so that the WebSphere Portal 7 will understand the Unicodes. Added a filter to modify ...
Prashanth Tiramareddi's user avatar
2 votes
1 answer
523 views

How to Convert UTF-16 Surrogate Decimal to UNICODE in Java

I have some string data like &#55357 ;&#56842 ; These are surrogate pairs in UTF 16 in decimal format. How can I convert them to Unicode Code Points in Java, so that my client can ...
Prashanth Tiramareddi's user avatar
0 votes
1 answer
2k views

Converting surrogate pairs to emoji - python3

I found the solution to similar question on the other topic, but unfortunately it's not working for me. Here is my problem: I'm making dataframe from the surrogatepairs unicodes which I'd like to ...
maliniaki's user avatar
7 votes
1 answer
2k views

Emojis to/from codepoints in Javascript

In a hybrid Android/Cordova game that I am creating I let users provide an identifier in the form of an Emoji + an alphanumeric - i.e. 0..9,A..Z,a..z - name. For example 🙋‍️Stackoverflow Server-...
DroidOS's user avatar
  • 8,620
1 vote
1 answer
423 views

Python Unicode - What Characters Can Be Printed in Windows Console?

Which Unicode characters can be printed in a Windows console from Python? The following code for code in range(1114112): print(chr(code), end=",") gives unimpressive results, including an error:...
Robin Andrews's user avatar
3 votes
1 answer
1k views

PHP json_decode does not work with single unpaired surrogate caused by Node 12 well-formed JSON.stringify

Node 12 now uses Well-formed JSON.stringify which outputs escape sequences for lone surrogates. PHP is not able to json_decode this when there is a lone surrogate that is escaped. Take the following ...
Alex G's user avatar
  • 151
0 votes
0 answers
279 views

How to encode unicode surrogate pairs, to write to file

I'm creating an compression / encryption program that utilises (nearly) all unicode characters, and then writes the data to a file. However, to write to the file, I need to encode the characters to ...
user avatar
2 votes
3 answers
987 views

How to reverse strings that contain surrogate pairs in Dart?

I was playing with algorithms using Dart and as I actually followed TDD, I realized that my code has some limitations. I was trying to reverse strings as part of an interview problem, but I couldn't ...
Vince Varga's user avatar
  • 6,281
0 votes
1 answer
579 views

Replacing emoji from a string

I am new to vb net. i am trying to process a string containing emoji but I cannot do it. my string looks like this:💛💛💙💛 I want to replace 💙 with 💛 what i am doing is using string.remove and ...
sebastian's user avatar
0 votes
1 answer
211 views

Should StringIO(HighSurrogate) throw an error in Python 2.7?

When I run this Python 2.7 code (edit: updated the code) import io x = io.StringIO(u'\ud801') CPython runs fine, but IronPython throws the following error: UnicodeEncodeError: Unable to translate ...
user541686's user avatar
  • 206k
2 votes
2 answers
290 views

Python unicode indexing shows different character

I have a Unicode string in a "narrow" build of Python 2.7.10 containing a Unicode character. I'm trying to use that Unicode character as a lookup in a dictionary, but when I index the string to get ...
Tim's user avatar
  • 2,763
3 votes
1 answer
2k views

C# - Regular expression to find a surrogate pair of a unicode codepoint from any string?

I am trying to parse a message that possibly contains emojis in it. An example message that could be received looks like: {"type":"chat","msg":"UserName:\u00a0\ud83d\ude0b \n"} What should match is \...
TEEBQNE's user avatar
  • 6,114
3 votes
2 answers
2k views

Unicode surrogate pairs and String.fromCodePoint() — JavaScript

I'm dealing with raw strings containing escape sequences for surrogate halves of UTF astral symbols. (I think I got that lingo right…) console.log("\uD83D\uDCA9") // => 💩 Let's use the above ...
Audun Olsen's user avatar
2 votes
1 answer
389 views

Python 2.7: Strange Unicode behavior

I am experiencing the following behavior in Python 2.7: >>> a1 = u'\U0001f04f' #1 >>> a2 = u'\ud83c\udc4f' #2 >>> a1 == a2 #3 False >>> a1.encode('utf8') == a2....
FireAphis's user avatar
  • 6,650
1 vote
1 answer
92 views

Write surrogate pairs to file using Haskell

This is the code I have: import qualified System.IO as IO writeSurrogate :: IO () writeSurrogate = do IO.writeFile "/home/sibi/surrogate.txt" ['\xD800'] Executing the above code gives error: ...
Sibi's user avatar
  • 47.7k
0 votes
0 answers
47 views

get unicode graphemes as unsplitted item with python2.7

any idea, if it is possible with regex (python 2.7) to get uniq chars unspitted into surrogate pairs for unicode graphemes? According This Example this is possible with python 3.x. See here: >>...
Egor Savin's user avatar
0 votes
1 answer
482 views

PYTHON RE Dont split UNICODE Chars into surrogate pairs while matching

who know, if it is possible to forbidden regex while macthing splitting code points into surrogate pairs. See the following example: How it is now: $ te = u'\U0001f600\U0001f600' $ flags1 = regex....
Egor Savin's user avatar
2 votes
1 answer
2k views

Convert from '\ud835' format to "𝓛" in c# [UWP]

I have a string with some wonky characters (for example) "𝓛𝓮𝓪𝓭 𝓑𝓪𝓬𝓴𝓮𝓷𝓭". I need to check if a List contains the first item in the string. But if I index it, it always becomes \ud835. After ...
Adam Dernis's user avatar
0 votes
0 answers
18 views

How to extract a list of all 18 character entries after specific phrase in string using RegExr?

I managed to extract a list of the text within square brackets within an emoji list I have here: https://regexr.com/3sqk1 But now I need to extract the equivalent decimalSurrogateHtml pairs for each ...
deeve's user avatar
  • 113
10 votes
1 answer
12k views

What are surrogate characters in UTF-8?

I have a strange validation program that validates wheather a utf-8 string is a valid host name(Zend Framework Hostname valdiator in PHP). It allows IDNs(internationalized domain names). It will ...
Gherman's user avatar
  • 6,997
1 vote
1 answer
951 views

What is a Unicode safe replica of String.IndexOf(string input) that can handle Surrogate Pairs?

I am trying to figure out an equivalent to C# string.IndexOf(string) that can handle surrogate pairs in Unicode characters. I am able to get the index when only comparing single characters, like in ...
Ibrennan208's user avatar
  • 1,345
2 votes
1 answer
404 views

Is String.Replace(string,string) Unicode Safe in regards to Surrogate Pairs?

I am trying to figure out the best way to create a function that is equivalent to String.Replace("oldValue","newValue"); that can handle surrogate pairs. My concern is that if there are surrogate ...
Ibrennan208's user avatar
  • 1,345
1 vote
1 answer
444 views

How to reveal the surrogate pairs in String in perl

I am working a perl code base to validate customer input, my goal is to block surrogate characters. My thought is first encoding the customer input as UTF-16 and foreach my $messageChar (@...
Dengke Liu's user avatar
0 votes
0 answers
9k views

UnicodeEncodeError: 'utf-8' codec can't encode character '\udc43' in position 1: surrogates not allowed

I have a list containing placenames and I want to create another array, initially empty, and then iterate the list of placenames and fill up my empty array with these placenames. For example, my ...
Mr Moose's user avatar