I have some UTF-8 encoded data living in a range of Uint8Array elements in Javascript. Is there an efficient way to decode these out to a regular javascript string (I believe Javascript uses 16 bit Unicode)? I dont want to add one character at the time as the string concaternation would become to CPU intensive.
TextEncoder
and TextDecoder
from the Encoding standard, which is polyfilled by the stringencoding library, converts between strings and ArrayBuffers:
var uint8array = new TextEncoder("utf-8").encode("ยข");
var string = new TextDecoder("utf-8").decode(uint8array);
-
41For anyone lazy like me,
npm install text-encoding
,var textEncoding = require('text-encoding'); var TextDecoder = textEncoding.TextDecoder;
. No thanks. – Evan Hu Nov 29 '16 at 6:11 -
18beware the npm text-encoding library, webpack bundle analyzer shows the library is HUGE – wayofthefuture Nov 19 '17 at 1:42
-
5@VincentScheib Browsers removed the support for any other formats except
utf-8
. So, theTextEncoder
argument is unnecessary! – bsound Mar 17 '19 at 17:00 -
1nodejs.org/api/string_decoder.html from the example: const { StringDecoder } = require('string_decoder'); const decoder = new StringDecoder('utf8'); const cent = Buffer.from([0xC2, 0xA2]); console.log(decoder.write(cent)); – curist Jul 24 '19 at 15:43
-
6Note that Node.js added the
TextEncoder
/TextDecoder
APIs in v11, so no need to install any extra packages if you only target current Node versions. – Loilo Sep 26 '19 at 15:38
This should work:
// http://www.onicos.com/staff/iz/amuse/javascript/expert/utf.txt
/* utf.js - UTF-8 <=> UTF-16 convertion
*
* Copyright (C) 1999 Masanao Izumo <iz@onicos.co.jp>
* Version: 1.0
* LastModified: Dec 25 1999
* This library is free. You can redistribute it and/or modify it.
*/
function Utf8ArrayToStr(array) {
var out, i, len, c;
var char2, char3;
out = "";
len = array.length;
i = 0;
while(i < len) {
c = array[i++];
switch(c >> 4)
{
case 0: case 1: case 2: case 3: case 4: case 5: case 6: case 7:
// 0xxxxxxx
out += String.fromCharCode(c);
break;
case 12: case 13:
// 110x xxxx 10xx xxxx
char2 = array[i++];
out += String.fromCharCode(((c & 0x1F) << 6) | (char2 & 0x3F));
break;
case 14:
// 1110 xxxx 10xx xxxx 10xx xxxx
char2 = array[i++];
char3 = array[i++];
out += String.fromCharCode(((c & 0x0F) << 12) |
((char2 & 0x3F) << 6) |
((char3 & 0x3F) << 0));
break;
}
}
return out;
}
It's somewhat cleaner as the other solutions because it doesn't use any hacks nor depends on Browser JS functions, e.g. works also in other JS environments.
Check out the JSFiddle demo.
-
6This seems kinda slow. But the only snippet in the universe I found that works. Good find+adoption! – Redsandro May 26 '14 at 13:25
-
6I don't understand why this doesn't have more upvotes. It seems eminently sensible to crank through the UTF-8 convention for small snippets. Async Blob + Filereader works great for big texts as others have indicated. – DanHorner Feb 2 '15 at 21:31
-
2
-
5Works great, except it doesn't handle 4+ byte sequences, e.g.
fromUTF8Array([240,159,154,133])
turns out empty (whilefromUTF8Array([226,152,131])โ"โ"
) – unhammer Nov 14 '16 at 11:55 -
1Why cases 8, 9, 10 and 11 are excluded? Any particular reason? And case 15 is also possible, right? 15(1111) will denote 4 bytes are used, isn't it? – RaR Oct 22 '19 at 5:38
Here's what I use:
var str = String.fromCharCode.apply(null, uint8Arr);
-
7
-
32This will throw
RangeError
on bigger texts. "Maximum call stack size exceeded" – Redsandro Mar 27 '14 at 0:18 -
1If you are converting large Uint8Arrays to binary strings and are getting RangeError, see the Uint8ToString function from stackoverflow.com/a/12713326/471341. – yonran Oct 31 '14 at 18:46
-
IE 11 throws
SCRIPT28: Out of stack space
when I feed it 300+k chars, orRangeError
for Chrome 39. Firefox 33 is ok. 100+k runs ok with all three. – Sheepy Nov 29 '14 at 4:30 -
This does not produce the correct result from the example unicode characters on en.wikipedia.org/wiki/UTF-8. e.g. String.fromCharCode.apply(null, new Uint8Array([0xc2, 0xa2])) does not produce ¢. – Vincent Scheib Apr 30 '16 at 0:27
In Node "Buffer
instances are also Uint8Array
instances", so buf.toString()
works in this case.
-
1Works great for me. And so simple ! But in fact Uint8Array have toString() method. – doom Nov 3 '17 at 11:34
-
1Simple and elegant, was not aware
Buffer
is also Uint8Array. Thanks! – LeOn - Han Li Oct 21 '18 at 1:33 -
4@doom On the browser side, Uint8Array.toString() will not compile a utf-8 string, it will list the numeric values in the array. So if what you have is a Uint8Array from another source that does not happen to also be a Buffer, you will need to create one to do the magic:
Buffer.from(uint8array).toString('utf-8')
– Joachim Lous Aug 7 '19 at 12:35 -
How about โBuffer.prototype.toString.call(uint8array, โutf8โ)โ to avoid creating a new buffer instance. – user11390576 Nov 23 '20 at 8:49
Found in one of the Chrome sample applications, although this is meant for larger blocks of data where you're okay with an asynchronous conversion.
/**
* Converts an array buffer to a string
*
* @private
* @param {ArrayBuffer} buf The buffer to convert
* @param {Function} callback The function to call when conversion is complete
*/
function _arrayBufferToString(buf, callback) {
var bb = new Blob([new Uint8Array(buf)]);
var f = new FileReader();
f.onload = function(e) {
callback(e.target.result);
};
f.readAsText(bb);
}
-
2As you said, this would perform terribly unless the buffer to convert is really really huge. The synchronous UTF-8 to wchar converstion of a simple string (say 10-40 bytes) implemented in, say, V8 should be much less than a microsecond whereas I would guess that your code would require a hundreds times that. Thanks all the same. – Jack Wester Jan 25 '13 at 9:20
The solution given by Albert works well as long as the provided function is invoked infrequently and is only used for arrays of modest size, otherwise it is egregiously inefficient. Here is an enhanced vanilla JavaScript solution that works for both Node and browsers and has the following advantages:
โข Works efficiently for all octet array sizes
โข Generates no intermediate throw-away strings
โข Supports 4-byte characters on modern JS engines (otherwise "?" is substituted)
var utf8ArrayToStr = (function () {
var charCache = new Array(128); // Preallocate the cache for the common single byte chars
var charFromCodePt = String.fromCodePoint || String.fromCharCode;
var result = [];
return function (array) {
var codePt, byte1;
var buffLen = array.length;
result.length = 0;
for (var i = 0; i < buffLen;) {
byte1 = array[i++];
if (byte1 <= 0x7F) {
codePt = byte1;
} else if (byte1 <= 0xDF) {
codePt = ((byte1 & 0x1F) << 6) | (array[i++] & 0x3F);
} else if (byte1 <= 0xEF) {
codePt = ((byte1 & 0x0F) << 12) | ((array[i++] & 0x3F) << 6) | (array[i++] & 0x3F);
} else if (String.fromCodePoint) {
codePt = ((byte1 & 0x07) << 18) | ((array[i++] & 0x3F) << 12) | ((array[i++] & 0x3F) << 6) | (array[i++] & 0x3F);
} else {
codePt = 63; // Cannot convert four byte code points, so use "?" instead
i += 3;
}
result.push(charCache[codePt] || (charCache[codePt] = charFromCodePt(codePt)));
}
return result.join('');
};
})();
-
2Best solution here, as it also handles 4-byte-characters (e.g. emojis) Thank you! – fiffy Mar 8 '19 at 13:36
-
1
Do what @Sudhir said, and then to get a String out of the comma seperated list of numbers use:
for (var i=0; i<unitArr.byteLength; i++) {
myString += String.fromCharCode(unitArr[i])
}
This will give you the string you want, if it's still relevant
-
Sorry, haven't noticed the last sentense in which you said you don't want to add one character at a time. Hope this helps others who doesn't have a problem with CPU usage however. – shuki Apr 3 '12 at 13:54
-
14
-
Even shorter:
String.fromCharCode.apply(null, unitArr);
. As mentioned, it doesn't handle UTF8 encoding, but sometimes this is simple enough if you only need ASCII support but don't have access to TextEncoder/TextDecoder. – Ravenstine Apr 17 '18 at 21:51 -
The answer mentions a @Sudhir but I searched the page and found now such answer. So would be better to inline whatever he said – Joakim Apr 23 '18 at 18:32
-
This will have terrible performance on longer strings. Don't use the + operator on strings. – Max May 9 '18 at 20:07
I was frustrated to see that people were not showing how to go both ways or showing that things work on none trivial UTF8 strings. I found a post on codereview.stackexchange.com that has some code that works well. I used it to turn ancient runes into bytes, to test some crypo on the bytes, then convert things back into a string. The working code is on github here. I renamed the methods for clarity:
// https://codereview.stackexchange.com/a/3589/75693
function bytesToSring(bytes) {
var chars = [];
for(var i = 0, n = bytes.length; i < n;) {
chars.push(((bytes[i++] & 0xff) << 8) | (bytes[i++] & 0xff));
}
return String.fromCharCode.apply(null, chars);
}
// https://codereview.stackexchange.com/a/3589/75693
function stringToBytes(str) {
var bytes = [];
for(var i = 0, n = str.length; i < n; i++) {
var char = str.charCodeAt(i);
bytes.push(char >>> 8, char & 0xFF);
}
return bytes;
}
The unit test uses this UTF-8 string:
// http://kermitproject.org/utf8.html
// From the Anglo-Saxon Rune Poem (Rune version)
const secretUtf8 = `แ แแปแซแแฆแฆแซแ แฑแฉแ แขแฑแซแ แแฑแชแซแทแแปแนแฆแแณแขแ
แแณแแชแแซแฆแแชแปแซแแชแพแพแชแซแทแแปแนแฆแแณแซแแแณแแขแพแซแปแฆแแซแแซแแชแพ
แทแแ แซแปแแซแนแแแแซแ แฉแฑแซแแฑแแปแแพแแซแแฉแแแแซแปแแแแชแพแฌ`;
Note that the string length is only 117 characters but the byte length, when encoded, is 234.
If I uncomment the console.log lines I can see that the string that is decoded is the same string that was encoded (with the bytes passed through Shamir's secret sharing algorithm!):
-
String.fromCharCode.apply(null, chars)
will error ifchars
is too big. – Marc J. Schmidt Jul 31 '20 at 11:40 -
is that everywhere or just some browsers and is it documented at all? – simbo1905 Aug 11 '20 at 9:09
-
e.g. here developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/…
But beware: by using apply this way, you run the risk of exceeding the JavaScript engine's argument length limit. The consequences of applying a function with too many arguments (that is, more than tens of thousands of arguments) varies across engines. (The JavaScriptCore engine has hard-coded argument limit of 65536.
– Marc J. Schmidt Aug 11 '20 at 10:12 -
thanks. in my case i was doing crypto over smallish strings so not a problem. do you have a fix for long strings? :-) – simbo1905 Aug 12 '20 at 10:30
-
1
If you can't use the TextDecoder API because it is not supported on IE:
- You can use the FastestSmallestTextEncoderDecoder polyfill recommended by the Mozilla Developer Network website;
- You can use this function also provided at the MDN website:
function utf8ArrayToString(aBytes) {
var sView = "";
for (var nPart, nLen = aBytes.length, nIdx = 0; nIdx < nLen; nIdx++) {
nPart = aBytes[nIdx];
sView += String.fromCharCode(
nPart > 251 && nPart < 254 && nIdx + 5 < nLen ? /* six bytes */
/* (nPart - 252 << 30) may be not so safe in ECMAScript! So...: */
(nPart - 252) * 1073741824 + (aBytes[++nIdx] - 128 << 24) + (aBytes[++nIdx] - 128 << 18) + (aBytes[++nIdx] - 128 << 12) + (aBytes[++nIdx] - 128 << 6) + aBytes[++nIdx] - 128
: nPart > 247 && nPart < 252 && nIdx + 4 < nLen ? /* five bytes */
(nPart - 248 << 24) + (aBytes[++nIdx] - 128 << 18) + (aBytes[++nIdx] - 128 << 12) + (aBytes[++nIdx] - 128 << 6) + aBytes[++nIdx] - 128
: nPart > 239 && nPart < 248 && nIdx + 3 < nLen ? /* four bytes */
(nPart - 240 << 18) + (aBytes[++nIdx] - 128 << 12) + (aBytes[++nIdx] - 128 << 6) + aBytes[++nIdx] - 128
: nPart > 223 && nPart < 240 && nIdx + 2 < nLen ? /* three bytes */
(nPart - 224 << 12) + (aBytes[++nIdx] - 128 << 6) + aBytes[++nIdx] - 128
: nPart > 191 && nPart < 224 && nIdx + 1 < nLen ? /* two bytes */
(nPart - 192 << 6) + aBytes[++nIdx] - 128
: /* nPart < 127 ? */ /* one byte */
nPart
);
}
return sView;
}
let str = utf8ArrayToString([50,72,226,130,130,32,43,32,79,226,130,130,32,226,135,140,32,50,72,226,130,130,79]);
// Must show 2Hโ + Oโ โ 2HโO
console.log(str);
Try these functions,
var JsonToArray = function(json)
{
var str = JSON.stringify(json, null, 0);
var ret = new Uint8Array(str.length);
for (var i = 0; i < str.length; i++) {
ret[i] = str.charCodeAt(i);
}
return ret
};
var binArrayToJson = function(binArray)
{
var str = "";
for (var i = 0; i < binArray.length; i++) {
str += String.fromCharCode(parseInt(binArray[i]));
}
return JSON.parse(str)
}
source: https://gist.github.com/tomfa/706d10fed78c497731ac, kudos to Tomfa
In NodeJS, we have Buffers available, and string conversion with them is really easy. Better, it's easy to convert a Uint8Array to a Buffer. Try this code, it's worked for me in Node for basically any conversion involving Uint8Arrays:
let str = Buffer.from(uint8arr.buffer).toString();
We're just extracting the ArrayBuffer from the Uint8Array and then converting that to a proper NodeJS Buffer. Then we convert the Buffer to a string (you can throw in a hex or base64 encoding if you want).
If we want to convert back to a Uint8Array from a string, then we'd do this:
let uint8arr = new Uint8Array(Buffer.from(str));
Be aware that if you declared an encoding like base64 when converting to a string, then you'd have to use Buffer.from(str, "base64")
if you used base64, or whatever other encoding you used.
This will not work in the browser without a module! NodeJS Buffers just don't exist in the browser, so this method won't work unless you add Buffer functionality to the browser. That's actually pretty easy to do though, just use a module like this, which is both small and fast!
class UTF8{
static encode(str:string){return new UTF8().encode(str)}
static decode(data:Uint8Array){return new UTF8().decode(data)}
private EOF_byte:number = -1;
private EOF_code_point:number = -1;
private encoderError(code_point) {
console.error("UTF8 encoderError",code_point)
}
private decoderError(fatal, opt_code_point?):number {
if (fatal) console.error("UTF8 decoderError",opt_code_point)
return opt_code_point || 0xFFFD;
}
private inRange(a:number, min:number, max:number) {
return min <= a && a <= max;
}
private div(n:number, d:number) {
return Math.floor(n / d);
}
private stringToCodePoints(string:string) {
/** @type {Array.<number>} */
let cps = [];
// Based on http://www.w3.org/TR/WebIDL/#idl-DOMString
let i = 0, n = string.length;
while (i < string.length) {
let c = string.charCodeAt(i);
if (!this.inRange(c, 0xD800, 0xDFFF)) {
cps.push(c);
} else if (this.inRange(c, 0xDC00, 0xDFFF)) {
cps.push(0xFFFD);
} else { // (inRange(c, 0xD800, 0xDBFF))
if (i == n - 1) {
cps.push(0xFFFD);
} else {
let d = string.charCodeAt(i + 1);
if (this.inRange(d, 0xDC00, 0xDFFF)) {
let a = c & 0x3FF;
let b = d & 0x3FF;
i += 1;
cps.push(0x10000 + (a << 10) + b);
} else {
cps.push(0xFFFD);
}
}
}
i += 1;
}
return cps;
}
private encode(str:string):Uint8Array {
let pos:number = 0;
let codePoints = this.stringToCodePoints(str);
let outputBytes = [];
while (codePoints.length > pos) {
let code_point:number = codePoints[pos++];
if (this.inRange(code_point, 0xD800, 0xDFFF)) {
this.encoderError(code_point);
}
else if (this.inRange(code_point, 0x0000, 0x007f)) {
outputBytes.push(code_point);
} else {
let count = 0, offset = 0;
if (this.inRange(code_point, 0x0080, 0x07FF)) {
count = 1;
offset = 0xC0;
} else if (this.inRange(code_point, 0x0800, 0xFFFF)) {
count = 2;
offset = 0xE0;
} else if (this.inRange(code_point, 0x10000, 0x10FFFF)) {
count = 3;
offset = 0xF0;
}
outputBytes.push(this.div(code_point, Math.pow(64, count)) + offset);
while (count > 0) {
let temp = this.div(code_point, Math.pow(64, count - 1));
outputBytes.push(0x80 + (temp % 64));
count -= 1;
}
}
}
return new Uint8Array(outputBytes);
}
private decode(data:Uint8Array):string {
let fatal:boolean = false;
let pos:number = 0;
let result:string = "";
let code_point:number;
let utf8_code_point = 0;
let utf8_bytes_needed = 0;
let utf8_bytes_seen = 0;
let utf8_lower_boundary = 0;
while (data.length > pos) {
let _byte = data[pos++];
if (_byte == this.EOF_byte) {
if (utf8_bytes_needed != 0) {
code_point = this.decoderError(fatal);
} else {
code_point = this.EOF_code_point;
}
} else {
if (utf8_bytes_needed == 0) {
if (this.inRange(_byte, 0x00, 0x7F)) {
code_point = _byte;
} else {
if (this.inRange(_byte, 0xC2, 0xDF)) {
utf8_bytes_needed = 1;
utf8_lower_boundary = 0x80;
utf8_code_point = _byte - 0xC0;
} else if (this.inRange(_byte, 0xE0, 0xEF)) {
utf8_bytes_needed = 2;
utf8_lower_boundary = 0x800;
utf8_code_point = _byte - 0xE0;
} else if (this.inRange(_byte, 0xF0, 0xF4)) {
utf8_bytes_needed = 3;
utf8_lower_boundary = 0x10000;
utf8_code_point = _byte - 0xF0;
} else {
this.decoderError(fatal);
}
utf8_code_point = utf8_code_point * Math.pow(64, utf8_bytes_needed);
code_point = null;
}
} else if (!this.inRange(_byte, 0x80, 0xBF)) {
utf8_code_point = 0;
utf8_bytes_needed = 0;
utf8_bytes_seen = 0;
utf8_lower_boundary = 0;
pos--;
code_point = this.decoderError(fatal, _byte);
} else {
utf8_bytes_seen += 1;
utf8_code_point = utf8_code_point + (_byte - 0x80) * Math.pow(64, utf8_bytes_needed - utf8_bytes_seen);
if (utf8_bytes_seen !== utf8_bytes_needed) {
code_point = null;
} else {
let cp = utf8_code_point;
let lower_boundary = utf8_lower_boundary;
utf8_code_point = 0;
utf8_bytes_needed = 0;
utf8_bytes_seen = 0;
utf8_lower_boundary = 0;
if (this.inRange(cp, lower_boundary, 0x10FFFF) && !this.inRange(cp, 0xD800, 0xDFFF)) {
code_point = cp;
} else {
code_point = this.decoderError(fatal, _byte);
}
}
}
}
//Decode string
if (code_point !== null && code_point !== this.EOF_code_point) {
if (code_point <= 0xFFFF) {
if (code_point > 0)result += String.fromCharCode(code_point);
} else {
code_point -= 0x10000;
result += String.fromCharCode(0xD800 + ((code_point >> 10) & 0x3ff));
result += String.fromCharCode(0xDC00 + (code_point & 0x3ff));
}
}
}
return result;
}
`
I am using this Typescript snippet:
function UInt8ArrayToString(uInt8Array: Uint8Array): string
{
var s: string = "[";
for(var i: number = 0; i < uInt8Array.byteLength; i++)
{
if( i > 0 )
s += ", ";
s += uInt8Array[i];
}
s += "]";
return s;
}
Remove the type annotations if you need the JavaScript version. Hope this helps!
-
3The OP asked to not add one char at a time. Also, he does not want to display it as a string-representation of list but rather just as a string. Also, this doesn't convert the chars to string but displays its number. – Albert Mar 13 '14 at 8:09
u8array.toString()
when reading files from BrowserFS that expose Uint8Array object when you callfs.readFile
. – jcubic May 17 '18 at 4:08toString
onUint8Array
returns comma-separated numbers such as"91,50,48,49,57,45"
(Chrome 79) – kolen Dec 17 '19 at 17:01