Questions tagged [utf]
Unicode Transformation Format (8/16/32/...) used for encoding Unicode code points
851
questions
2
votes
3
answers
44
views
Problem to process visually identical looking characters (umlauts)
This may probably be a more general issue related to character encoding, but since I came across the issue while coding an outer join of two dataframes, I post it with a Python code example.
On the ...
0
votes
1
answer
38
views
Get Unicode of a character in Groovy
Is there any way in Groovy so that I can get Unicode equivalent of any character? e.g.
Suppose a method getUnicode(char c). A call getUnicode('÷') should return \u00f7.
3
votes
4
answers
153
views
How to Store Emojis in char8_t and Print Them Out in C++20?
I just now heard about the existence of char8_t, char16_t and char32_t and I am testing it out. When I try to compile the code below, g++ throws the following error:
error: use of deleted function ‘...
0
votes
0
answers
33
views
Why french accents are not represented correctly after Matlab2011b?
I am mantaining a piece of software developed on Matlab R2011b which uses several languages, french among them.
I am now compiling the code on Matlab R2022 and the non ASCII characters are represented ...
0
votes
3
answers
90
views
I have difficulties with putwchar() in c
Certainly, my problem is not new...., so I apologize if my error is simply too stupid.
I just wanted to become familiar with putwchar and simply wrote the following little piece of code:
#include <...
0
votes
1
answer
47
views
Is UTF-16 encoding handles data compression by default?
I've unicode char த
When I convert id to data,
UTF 8 -> Size: 3 bytes Array: [224, 174, 164]
UTF 16 -> Size: 4 bytes Array: [2980]
Seems pretty simple UTF8 tooks 1 byte per code and UTF16 ...
0
votes
1
answer
169
views
Delphi THttpClient: post's IHttpResponse giving encoded string seen in fiddler webview
I have a THTTPClient giving a strange response (apparently UTF-16 encoding?) when invoking lHttpResp.ContentAsString().
The string comes through as this:
㰀㼀砀洀氀 瘀攀爀猀椀漀渀㴀∀⸀ ∀ 攀渀挀漀搀椀渀最㴀∀唀吀䘀ⴀ㘀∀ ...
0
votes
2
answers
179
views
Neither ASCII or UTF8 can decode French characters, what should I do?
I have the following function:
private void ReceivedData(byte[] data)
{
string info = Encoding.ASCII.GetString(data);
When I use this, then the data, containing an é character, replace that ...
0
votes
0
answers
35
views
Python3 Unicode Character Space
So I'm trying to find a way for python3 to tell me how much space a character takes up
I have found a method that works for some characters, it looks like this
from wcwidth import wcswidth
length = ...
2
votes
2
answers
105
views
How to combine two code points to get one?
I know that unicode code point for Á is U+00C1. I read on internet and many forums and articles that I can also make an Á by combining characters ´ (unicode: U+00B4) and A (unicode: U+0041).
My ...
0
votes
1
answer
12
views
How to represent Unicode codepoints in string literals in Memgraph?
What is the correct way to represent Unicode codepoints in string literals in Memgraph? Does memgraph support UTF-16 or UTF-32 codepoints?
3
votes
3
answers
406
views
Why does my html email appear differently when sent by AWS SES?
I've been working on some custom html email templates. I'm having some trouble with my emails appearing differently when they are sent by different email services. I'm using AWS SES to send these ...
0
votes
2
answers
792
views
Decode UUID 4 as a Python string
I would like to generate a UUID v4 string starting from the uuid import from the Python standard library.
I know I can cast a UUID to str by doing str(uuid.uuid4()), however I am trying to understand ...
0
votes
0
answers
48
views
remove � question mark characters from columns
I have a dataset in which some row values have � (question mark). How can I remove these? This symbol however for some reason does not show up in MS-Excel (so maybe � is space?).
Please note I am new ...
2
votes
0
answers
69
views
Arabic words aren't displayed properly in DrRacket
I work on Arabic scripted texts in DrRacket but the characters stand seprate, they have to be attached to each other.
The second problem is that DrRacket reads them left-to-right like in Latin script.
...
0
votes
1
answer
62
views
Character decoding issue after Azure AD authentication
In my Java 11, Spring Boot 2.7.3 application, I am making a REST call to Hashicorp Vault in order to fetch my app config. Here is the code snippet: -
HttpHeaders headers = new HttpHeaders();
...
1
vote
1
answer
334
views
How to convert VS code from UTF-16 to UTF-8
I don't know why but my VS code in the bottom line report UTF-16 LE and I want UTF-8.
This gives me many problems with JS file.
What do I have to do to change that value?
2
votes
1
answer
148
views
Check if file/blob object is valid UTF-8
I need a function that can check if a file or blob object is valid UTF-8. I can get the text and check for � characters, but if the string has that character to begin with, the function would mark it ...
0
votes
1
answer
2k
views
SyntaxError: Non-UTF-8 code starting with '\xca' in file /usr/local/bin/python3 on line 2, but no encoding declared
I am trying to schedule a crontab job to run a python script and referring to python in /usr/local/bin/python3 but getting this error
SyntaxError: Non-UTF-8 code starting with '\xca' in file /usr/...
0
votes
1
answer
244
views
With the release of Java 18, UTF-8 is the default charset now. Does it imply that char data type is no more UTF-16?
I understand that Java follows UTF-16 for the char data type. But the new update in Java 18 - Default charset for the standard Java APIs is UTF-8.
Does this update have any impact on the char data ...
1
vote
3
answers
759
views
While reading a CSV I get a question mark at the beginning
I'm trying to do a small school practice about Java Text I/O and while trying to read a CSV file with name prefixes (a Dutch thing) and surnames I got a question mark in the beginning.
It's a small ...
1
vote
1
answer
348
views
Python multiple special characters in csv file
I have an csv file with some data that contains string like this:
string = 'übung Überprüfung'
and I want my programm to read it or convert it into the right format:
'übung Überprüfung'
My current ...
0
votes
1
answer
248
views
Transforming encoding string to bytes and then writing PDF file not working - What kind of encoding is this?
EDIT: This is solved. Thank you Jeremy Lakeman for your comment, it put me on the right track.
I have a PDF file content string that I am trying to use to produce a PDF file.
EDIT: The encoded string ...
0
votes
1
answer
347
views
How to support Cyrillic alphabet in Eclipse?
How to make it so that when the Russian language is output, Russian letters are output, and not �� ����� �������� �����! It does not work only in the console, but everything is ok in the terminal
I've ...
0
votes
0
answers
148
views
How do I use a unicode character in a font family in HTML?
I am developing a game using HTML. I found a font I want to use. I installed it correctly on the computer, and I imported it into the CSS file, however; I am unable to use the character that I ...
0
votes
1
answer
76
views
Any way to generate SHA 1 hash of a sequence of 1s and 0s in Python
I want to generate a random sequence of 1s and 0s and input it into the SHA1 hash calculator in Python.
The hashlib library (doc link) for generating hashes accepts byte-like objects as input in its ...
0
votes
1
answer
106
views
How to convert UTF8 to UTF16 in Firebird
What ways do you know to convert UTF8 to UTF16. Maybe even through UDF functions. Well, you can also write in general about how to use UTF16 in Firebird?
0
votes
1
answer
158
views
Why String.protoptype.charCodeAt() can convert binary string into an Uint8Array?
Suppose I have a base64 encoded string and I want to convert it into an ArrayBuffer, I can do it in this way:
// base64 decode the string to get the binary data
const binaryString = window.atob(...
0
votes
0
answers
46
views
Comparing utf decoded value with a string value
I have been trying to compare the UTF-decoded data from pyserial with a value of a similar data type. However, it never returns true.
Please refer to the Python 3 code below:
import serial.tools....
0
votes
0
answers
187
views
Save file as Unicode in C# stopped working as expected
I have demo file Encoded UTF-8, which includes special characters:
I had solution which was converting this file to Unicode (This conversion is necessary so data was displayed by MS Excel correctly):
...
1
vote
3
answers
287
views
Why does utf-16 only support 2^20 code points?
Well, I'm starting to study unicode now, and I had several doubts, at this moment I'm learning what a plane is, I saw that a plane is a set of 2^16 code points, and that utf-16 encoding supports 17 ...
0
votes
1
answer
106
views
Cannot deserialize a class only when sent by the server-to-client, works vice versa
The streams used to work, but suddenly just stopped two days ago, and I can't get it fixed.
The client creates and successfully sends a User object and a Message object (serialize by the client and ...
0
votes
0
answers
45
views
How to calculate the distance of one string across a defined range of strings?
Given an interval defined by two strings, [x, y], and third string s between them, is there a way to calculate the percentage of the whole interval from x to s. Preferably which honors collation (case ...
0
votes
1
answer
43
views
MYSQL: best utf setting for ACCENT sensitive but CASE insensitive
For example, ἐν or Ἐν are the same, but should be distinguished from ἕν/Ἓν. I've tried utf8_bin which seems to be the closest, but is also case sensitive.
2
votes
2
answers
735
views
Maximum number of codepoints in a grapheme cluster
I am using the C++ ICU library. I wish to split a utf-8 string into approximately equal chunks. However, I want the chunks to be demarcated at grapheme cluster boundaries. I do not wish to convert my ...
-1
votes
1
answer
53
views
How to read 2 bytes in flex?
I want to read utf-16 by 2 bytes in flex and use it.
I will use 2 byte data converted to multibyte.
I opened the file with "rb" and read it.
But I can't read the following.
4C AE C0 C9 20 00 ...
0
votes
0
answers
16
views
How can I read \x1a from a file? [duplicate]
The contents of the file I need to read are as follows.
This file is in utf-16 LE format.
테스트갚켚테스트
in utf-16 LE
ff fe 4c d1 a4 cd b8 d2 1a ac 1a cf 4c d1 a4 cd b8 d2
I can't read the whole file ...
0
votes
0
answers
92
views
AS3: How to show Hindi number in Input and Dynamic text?
I have a project where I calculate the Area of a triangle.
Problem:
The problem is the number appear as Arabic and I want it to be shown as a Hindi (Devanagari) font's number?
What I tried:
I have ...
2
votes
2
answers
365
views
C++ test for validation UTF-8
I need to write unit tests for UTF-8 validation, but I don't know how to write incorrect UTF-8 cases in C++:
TEST(validation, Tests)
{
std::string str = "hello";
EXPECT_TRUE(...
0
votes
1
answer
183
views
nano 4.3 won't accept special characters, while nano 2.0.6 will
I can copy and paste UTF-8 etc. characters fine from a browser to a Mac terminal. I can also do this in nano version 2.0.6. See how it accepts the apostrophe from Chrome:
The problem is that in nano ...
0
votes
1
answer
1k
views
Copy data from Excel, paste to .txt file
I am trying to figure out how to copy a range in Excel and paste to a .txt file. Note: This involves multiple languages which adds complications.
I currently have the following for copying the exact ...
0
votes
1
answer
138
views
Displaying accented characters using VBA in PowerPoint
I have created a PowerPoint to display 5 random words from a .txt file following this excellent tutorial.
The problem is, the words in my .txt file are Spanish, so have accents on them. When ...
0
votes
1
answer
192
views
How is UTF-16 with big endian BOM (FEFF) stored in a little endian machine?
From my understanding, the endianness of UTF-16 is independent from the endianness of the host architecture.
Let's take the character 0x0042 (the letter "D") saved in a file for example. In ...
0
votes
0
answers
48
views
Unicode Character Outside of Quote Python
Im using python3, and i'm trying to print a unicode character called hangul filler, when i assign just that character to a variable, and I check the variable, or print it with something else, it ...
1
vote
0
answers
146
views
Linux utf16 to utf8 on c++
I have std::string utf16 containing text in utf16. This string I get from third-party library.
For example std::string utf16("\x04\x1f\x04\x40\x04\x38\x04\x32\x04\x35\x04\x42"). This text is ...
1
vote
1
answer
548
views
Reading UTF-8 input
I am making a program, like flashcards but console based. At the start of the program, I read from a file containing UTF-8 encoded Japanese characters (such as "ひらがな, カタカナ, 患者"). However, ...
0
votes
1
answer
309
views
Redshift copy statement losing spanish characters
I have a csv file which containt the following value in one column:
Some words: camión, coche, moto ...
0
votes
2
answers
372
views
Access dictionary value using Spark dataframe column as key
I have a dictionary below called sample.
sample = {
10:1
20:2
30:3
}
I have a dataframe column called age for example and the column contains 10. I want to use this dataframe column as ...
0
votes
1
answer
123
views
Which is the first multi-byte UTF codepoint?
I just wanted to know which unicode blocks can be safely used when being limited to single-byte codepoints only.
So, which is the last single-byte codepoint, and which is the first multi-byte ...
0
votes
1
answer
313
views
Contents within a .bat file not running properly when running said .bat file
So I've encountered a very odd problem while trying to run this java application I've just compiled. I wrote this very simple .bat file to run the program with just one parameter:
@echo off
java –...