Questions tagged [utf]

Unicode Transformation Format (8/16/32/...) used for encoding Unicode code points

Filter by
Sorted by
Tagged with
2 votes
3 answers
44 views

Problem to process visually identical looking characters (umlauts)

This may probably be a more general issue related to character encoding, but since I came across the issue while coding an outer join of two dataframes, I post it with a Python code example. On the ...
  • 822
0 votes
1 answer
38 views

Get Unicode of a character in Groovy

Is there any way in Groovy so that I can get Unicode equivalent of any character? e.g. Suppose a method getUnicode(char c). A call getUnicode('÷') should return \u00f7.
3 votes
4 answers
153 views

How to Store Emojis in char8_t and Print Them Out in C++20?

I just now heard about the existence of char8_t, char16_t and char32_t and I am testing it out. When I try to compile the code below, g++ throws the following error: error: use of deleted function ‘...
  • 370
0 votes
0 answers
33 views

Why french accents are not represented correctly after Matlab2011b?

I am mantaining a piece of software developed on Matlab R2011b which uses several languages, french among them. I am now compiling the code on Matlab R2022 and the non ASCII characters are represented ...
0 votes
3 answers
90 views

I have difficulties with putwchar() in c

Certainly, my problem is not new...., so I apologize if my error is simply too stupid. I just wanted to become familiar with putwchar and simply wrote the following little piece of code: #include <...
0 votes
1 answer
47 views

Is UTF-16 encoding handles data compression by default?

I've unicode char த When I convert id to data, UTF 8 -> Size: 3 bytes Array: [224, 174, 164] UTF 16 -> Size: 4 bytes Array: [2980] Seems pretty simple UTF8 tooks 1 byte per code and UTF16 ...
  • 13
0 votes
1 answer
169 views

Delphi THttpClient: post's IHttpResponse giving encoded string seen in fiddler webview

I have a THTTPClient giving a strange response (apparently UTF-16 encoding?) when invoking lHttpResp.ContentAsString(). The string comes through as this: 㰀㼀砀洀氀 瘀攀爀猀椀漀渀㴀∀㄀⸀ ∀ 攀渀挀漀搀椀渀最㴀∀唀吀䘀ⴀ㄀㘀∀ ...
0 votes
2 answers
179 views

Neither ASCII or UTF8 can decode French characters, what should I do?

I have the following function: private void ReceivedData(byte[] data) { string info = Encoding.ASCII.GetString(data); When I use this, then the data, containing an é character, replace that ...
  • 15.8k
0 votes
0 answers
35 views

Python3 Unicode Character Space

So I'm trying to find a way for python3 to tell me how much space a character takes up I have found a method that works for some characters, it looks like this from wcwidth import wcswidth length = ...
2 votes
2 answers
105 views

How to combine two code points to get one?

I know that unicode code point for Á is U+00C1. I read on internet and many forums and articles that I can also make an Á by combining characters ´ (unicode: U+00B4) and A (unicode: U+0041). My ...
  • 311
0 votes
1 answer
12 views

How to represent Unicode codepoints in string literals in Memgraph?

What is the correct way to represent Unicode codepoints in string literals in Memgraph? Does memgraph support UTF-16 or UTF-32 codepoints?
  • 876
3 votes
3 answers
406 views

Why does my html email appear differently when sent by AWS SES?

I've been working on some custom html email templates. I'm having some trouble with my emails appearing differently when they are sent by different email services. I'm using AWS SES to send these ...
  • 550
0 votes
2 answers
792 views

Decode UUID 4 as a Python string

I would like to generate a UUID v4 string starting from the uuid import from the Python standard library. I know I can cast a UUID to str by doing str(uuid.uuid4()), however I am trying to understand ...
  • 4,275
0 votes
0 answers
48 views

remove � question mark characters from columns

I have a dataset in which some row values have � (question mark). How can I remove these? This symbol however for some reason does not show up in MS-Excel (so maybe � is space?). Please note I am new ...
  • 1,783
2 votes
0 answers
69 views

Arabic words aren't displayed properly in DrRacket

I work on Arabic scripted texts in DrRacket but the characters stand seprate, they have to be attached to each other. The second problem is that DrRacket reads them left-to-right like in Latin script. ...
  • 21
0 votes
1 answer
62 views

Character decoding issue after Azure AD authentication

In my Java 11, Spring Boot 2.7.3 application, I am making a REST call to Hashicorp Vault in order to fetch my app config. Here is the code snippet: - HttpHeaders headers = new HttpHeaders(); ...
  • 366
1 vote
1 answer
334 views

How to convert VS code from UTF-16 to UTF-8

I don't know why but my VS code in the bottom line report UTF-16 LE and I want UTF-8. This gives me many problems with JS file. What do I have to do to change that value?
2 votes
1 answer
148 views

Check if file/blob object is valid UTF-8

I need a function that can check if a file or blob object is valid UTF-8. I can get the text and check for � characters, but if the string has that character to begin with, the function would mark it ...
  • 1,411
0 votes
1 answer
2k views

SyntaxError: Non-UTF-8 code starting with '\xca' in file /usr/local/bin/python3 on line 2, but no encoding declared

I am trying to schedule a crontab job to run a python script and referring to python in /usr/local/bin/python3 but getting this error SyntaxError: Non-UTF-8 code starting with '\xca' in file /usr/...
  • 43
0 votes
1 answer
244 views

With the release of Java 18, UTF-8 is the default charset now. Does it imply that char data type is no more UTF-16?

I understand that Java follows UTF-16 for the char data type. But the new update in Java 18 - Default charset for the standard Java APIs is UTF-8. Does this update have any impact on the char data ...
1 vote
3 answers
759 views

While reading a CSV I get a question mark at the beginning

I'm trying to do a small school practice about Java Text I/O and while trying to read a CSV file with name prefixes (a Dutch thing) and surnames I got a question mark in the beginning. It's a small ...
1 vote
1 answer
348 views

Python multiple special characters in csv file

I have an csv file with some data that contains string like this: string = 'übung Überprüfung' and I want my programm to read it or convert it into the right format: 'übung Überprüfung' My current ...
  • 9
0 votes
1 answer
248 views

Transforming encoding string to bytes and then writing PDF file not working - What kind of encoding is this?

EDIT: This is solved. Thank you Jeremy Lakeman for your comment, it put me on the right track. I have a PDF file content string that I am trying to use to produce a PDF file. EDIT: The encoded string ...
0 votes
1 answer
347 views

How to support Cyrillic alphabet in Eclipse?

How to make it so that when the Russian language is output, Russian letters are output, and not �� ����� �������� �����! It does not work only in the console, but everything is ok in the terminal I've ...
  • 1
0 votes
0 answers
148 views

How do I use a unicode character in a font family in HTML?

I am developing a game using HTML. I found a font I want to use. I installed it correctly on the computer, and I imported it into the CSS file, however; I am unable to use the character that I ...
0 votes
1 answer
76 views

Any way to generate SHA 1 hash of a sequence of 1s and 0s in Python

I want to generate a random sequence of 1s and 0s and input it into the SHA1 hash calculator in Python. The hashlib library (doc link) for generating hashes accepts byte-like objects as input in its ...
0 votes
1 answer
106 views

How to convert UTF8 to UTF16 in Firebird

What ways do you know to convert UTF8 to UTF16. Maybe even through UDF functions. Well, you can also write in general about how to use UTF16 in Firebird?
0 votes
1 answer
158 views

Why String.protoptype.charCodeAt() can convert binary string into an Uint8Array?

Suppose I have a base64 encoded string and I want to convert it into an ArrayBuffer, I can do it in this way: // base64 decode the string to get the binary data const binaryString = window.atob(...
  • 699
0 votes
0 answers
46 views

Comparing utf decoded value with a string value

I have been trying to compare the UTF-decoded data from pyserial with a value of a similar data type. However, it never returns true. Please refer to the Python 3 code below: import serial.tools....
  • 145
0 votes
0 answers
187 views

Save file as Unicode in C# stopped working as expected

I have demo file Encoded UTF-8, which includes special characters: I had solution which was converting this file to Unicode (This conversion is necessary so data was displayed by MS Excel correctly): ...
1 vote
3 answers
287 views

Why does utf-16 only support 2^20 code points?

Well, I'm starting to study unicode now, and I had several doubts, at this moment I'm learning what a plane is, I saw that a plane is a set of 2^16 code points, and that utf-16 encoding supports 17 ...
0 votes
1 answer
106 views

Cannot deserialize a class only when sent by the server-to-client, works vice versa

The streams used to work, but suddenly just stopped two days ago, and I can't get it fixed. The client creates and successfully sends a User object and a Message object (serialize by the client and ...
  • 11
0 votes
0 answers
45 views

How to calculate the distance of one string across a defined range of strings?

Given an interval defined by two strings, [x, y], and third string s between them, is there a way to calculate the percentage of the whole interval from x to s. Preferably which honors collation (case ...
  • 5,608
0 votes
1 answer
43 views

MYSQL: best utf setting for ACCENT sensitive but CASE insensitive

For example, ἐν or Ἐν are the same, but should be distinguished from ἕν/Ἓν. I've tried utf8_bin which seems to be the closest, but is also case sensitive.
  • 67
2 votes
2 answers
735 views

Maximum number of codepoints in a grapheme cluster

I am using the C++ ICU library. I wish to split a utf-8 string into approximately equal chunks. However, I want the chunks to be demarcated at grapheme cluster boundaries. I do not wish to convert my ...
-1 votes
1 answer
53 views

How to read 2 bytes in flex?

I want to read utf-16 by 2 bytes in flex and use it. I will use 2 byte data converted to multibyte. I opened the file with "rb" and read it. But I can't read the following. 4C AE C0 C9 20 00 ...
  • 1
0 votes
0 answers
16 views

How can I read \x1a from a file? [duplicate]

The contents of the file I need to read are as follows. This file is in utf-16 LE format. 테스트갚켚테스트 in utf-16 LE ff fe 4c d1 a4 cd b8 d2 1a ac 1a cf 4c d1 a4 cd b8 d2 I can't read the whole file ...
  • 1
0 votes
0 answers
92 views

AS3: How to show Hindi number in Input and Dynamic text?

I have a project where I calculate the Area of a triangle. Problem: The problem is the number appear as Arabic and I want it to be shown as a Hindi (Devanagari) font's number? What I tried: I have ...
2 votes
2 answers
365 views

C++ test for validation UTF-8

I need to write unit tests for UTF-8 validation, but I don't know how to write incorrect UTF-8 cases in C++: TEST(validation, Tests) { std::string str = "hello"; EXPECT_TRUE(...
  • 225
0 votes
1 answer
183 views

nano 4.3 won't accept special characters, while nano 2.0.6 will

I can copy and paste UTF-8 etc. characters fine from a browser to a Mac terminal. I can also do this in nano version 2.0.6. See how it accepts the apostrophe from Chrome: The problem is that in nano ...
0 votes
1 answer
1k views

Copy data from Excel, paste to .txt file

I am trying to figure out how to copy a range in Excel and paste to a .txt file. Note: This involves multiple languages which adds complications. I currently have the following for copying the exact ...
  • 1
0 votes
1 answer
138 views

Displaying accented characters using VBA in PowerPoint

I have created a PowerPoint to display 5 random words from a .txt file following this excellent tutorial. The problem is, the words in my .txt file are Spanish, so have accents on them. When ...
0 votes
1 answer
192 views

How is UTF-16 with big endian BOM (FEFF) stored in a little endian machine?

From my understanding, the endianness of UTF-16 is independent from the endianness of the host architecture. Let's take the character 0x0042 (the letter "D") saved in a file for example. In ...
0 votes
0 answers
48 views

Unicode Character Outside of Quote Python

Im using python3, and i'm trying to print a unicode character called hangul filler, when i assign just that character to a variable, and I check the variable, or print it with something else, it ...
1 vote
0 answers
146 views

Linux utf16 to utf8 on c++

I have std::string utf16 containing text in utf16. This string I get from third-party library. For example std::string utf16("\x04\x1f\x04\x40\x04\x38\x04\x32\x04\x35\x04\x42"). This text is ...
  • 11
1 vote
1 answer
548 views

Reading UTF-8 input

I am making a program, like flashcards but console based. At the start of the program, I read from a file containing UTF-8 encoded Japanese characters (such as "ひらがな, カタカナ, 患者"). However, ...
  • 23
0 votes
1 answer
309 views

Redshift copy statement losing spanish characters

I have a csv file which containt the following value in one column: Some words: camión, coche, moto ...
  • 2,677
0 votes
2 answers
372 views

Access dictionary value using Spark dataframe column as key

I have a dictionary below called sample. sample = { 10:1 20:2 30:3 } I have a dataframe column called age for example and the column contains 10. I want to use this dataframe column as ...
  • 197
0 votes
1 answer
123 views

Which is the first multi-byte UTF codepoint?

I just wanted to know which unicode blocks can be safely used when being limited to single-byte codepoints only. So, which is the last single-byte codepoint, and which is the first multi-byte ...
  • 1,028
0 votes
1 answer
313 views

Contents within a .bat file not running properly when running said .bat file

So I've encountered a very odd problem while trying to run this java application I've just compiled. I wrote this very simple .bat file to run the program with just one parameter: @echo off java –...

1
2 3 4 5
18