13

I've used the Facebook feature to download all my data. The resulting zip file contains meta information in JSON files. The problem is that unicode characters in strings in these JSON files are escaped in a weird way.

Here's an example of such a string:

"nejni\u00c5\u00be\u00c5\u00a1\u00c3\u00ad bod: 0 mnm Ben\u00c3\u00a1tky\n"

When I try parse the string for example with javascript's JSON.parse() and print it out I get:

"nejnižší bod: 0 mnm Benátky\n"

While it should be

"nejnižší bod: 0 mnm Benátky\n"

I can see that \u00c5\u00be should somehow correspond to ž but I can't figure out the general pattern.

I've been able to figure out these characters so far:

'\u00c2\u00b0' : '°',
'\u00c3\u0081' : 'Á',
'\u00c3\u00a1' : 'á',
'\u00c3\u0089' : 'É',
'\u00c3\u00a9' : 'é',
'\u00c3\u00ad' : 'í',
'\u00c3\u00ba' : 'ú',
'\u00c3\u00bd' : 'ý',
'\u00c4\u008c' : 'Č',
'\u00c4\u008d' : 'č',
'\u00c4\u008f' : 'ď',
'\u00c4\u009b' : 'ě',
'\u00c5\u0098' : 'Ř',
'\u00c5\u0099' : 'ř',
'\u00c5\u00a0' : 'Š',
'\u00c5\u00a1' : 'š',
'\u00c5\u00af' : 'ů',
'\u00c5\u00be' : 'ž',

So what is this weird encoding? Is there any known tool that can correctly decode it?

2
  • 2
    The bytes 0xC5 0xBE are the UTF-8 encoding for U+017E LATIN SMALL LETTER Z WITH CARON. If those individual bytes are treated as unicode codepoints, they are U+00C5 LATIN CAPITAL LETTER A WITH RING and U+00BE VULGAR FRACTION THREE FOURTHS. So something read each byte and wrote it out as a Unicode codepoint instead of handling multibyte UTF-8 sequences appropriately. Probably something assumed it was converting Latin-1 text to Unicode.
    – Shawn
    Commented Oct 11, 2018 at 5:56
  • @Shawn ok. But how do I force for example the javascripts's JSON.parse() to handle these multi byte utf8 sequences correctly?
    – Jen
    Commented Oct 11, 2018 at 9:23

9 Answers 9

4

The encoding is valid UTF-8. The problem is, JavaScript doesn't use UTF-8, it uses UTF-16. So you have to convert from the valid UTF-8, to JavaScript UTF-16:

function decode(s) {
   let d = new TextDecoder;
   let a = s.split('').map(r => r.charCodeAt());
   return d.decode(new Uint8Array(a));
}

let s = "nejni\u00c5\u00be\u00c5\u00a1\u00c3\u00ad bod: 0 mnm Ben\u00c3\u00a1tky\n";
s = decode(s);
console.log(s);

https://developer.mozilla.org/docs/Web/API/TextDecoder

1
  • maybe solution works, but I don't see how this is valid UTF-8 encoding Commented Apr 25, 2023 at 16:58
3

Thanks to Jen's excellent question and Shawn's comment.

Basically facebook seems to take each individual byte of the unicode string representation, then exporting to JSON as if these bytes are individual Unicode code points.

What we need to do is take last two characters of each sextet (e.g. c3 from \u00c3), concatenate them together and read as a Unicode string.

This is how I do it in Ruby (see gist):

require 'json'
require 'uri'

bytes_re = /((?:\\\\)+|[^\\])(?:\\u[0-9a-f]{4})+/

txt = File.read('export.json').gsub(bytes_re) do |bad_unicode|
  $1 + eval(%Q{"#{bad_unicode[$1.size..-1].gsub('\u00', '\x')}"}).to_json[1...-1]
end

good_data = JSON.load(txt)

With bytes_re we catch all sequences of bad Unicode characters.

Then for each sequence replace '\u00' with '\x' (e.g. \xc3), put quotes around it " and use Ruby's built-in string parsing so that the \xc3\xbe... strings are converted to actual bytes, that will later remain as Unicode characters in the JSON or properly quoted by the #to_json method.

The [1...-1] is to remove quotes inserted by #to_json

I wanted to explain the code because question is not ruby specific and reader may use another language.

I guess somebody can do it with a sufficiently ugly sed command..

1
  • thanks a lot. Your script works perfectly for me. It's a pity that after so many years, Facebook still exports such weird unicode strings to bother people who just want their posts/notes in Facebook to be transferred to their own blogs.
    – AlvaPan
    Commented Apr 25, 2023 at 9:22
3

You can use a regular expression to find groups of almost unicode characters, decode them into Latin-1 and then encode back into UTF-8

The following code should work in python3.x:

import re

re.sub(r'[\xc2-\xf4][\x80-\xbf]+',lambda m: m.group(0).encode('latin1').decode('utf8'), s)
1
  • This seems to be working on parsed strings. Could you explain why you use [\xc2-\xf4][\x80-\xbf]+ regular expression? Commented Dec 21, 2020 at 21:01
2

The JSON file itself is UTF-8, but the strings are UTF-16 characters converted to byte sequences then converted to UTF-8 using escape sequences.

This command fixes a file like this in Emacs:

(defun k/format-facebook-backup ()
  "Normalize a Facebook backup JSON file."
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (let ((inhibit-read-only t)
          (size (point-max))
          bounds str)
      (while (search-forward "\"\\u" nil t)
        (message "%.f%%" (* 100 (/ (point) size 1.0)))
        (setq bounds (bounds-of-thing-at-point 'string))
        (when bounds
          (setq str (--> (json-parse-string (buffer-substring (car bounds)
                                                              (cdr bounds)))
                         (string-to-list it)
                         (apply #'unibyte-string it)
                         (decode-coding-string it 'utf-8)))
          (setf (buffer-substring (car bounds) (cdr bounds))
                (json-serialize str))))))
  (save-buffer))
2

Just in case, if someone is looking for PHP solution ;)

$result = preg_replace_callback(
    '/\\\u00([[:xdigit:]]{2})/',
    function ($matches) {
        return chr(hexdec($matches[1]));
    },
    $str
);

var_dump($result);

The pattern reads: "find something that starts with \u00 and continues with to hexadecimal digits". Then convert those digits into the corresponding byte.

0
1

Just adding the general rule how to get from something like '\u00c5\u0098' to 'Ř'. Putting together the last two letters from the \u parts gets you c5 and 98 which are the two bytes of the utf-8 representation. UTF-8 encodes the code point in two bytes like this: 110xxxxx 10xxxxxx, where x are the actual bits of the character code. You can take the two bytes, use & to get the x parts, put them one after the next and read that as a number and you get the 0x158, which is the code for 'Ř'.

My javascript implementation:

    function fixEncoding(s) {
        var reg = /\\u00([a-f0-9]{2})\\u00([a-f0-9]{2})/gi;
        return s.replace(reg, function(a, m1, m2){
            b1 = parseInt(m1,16);
            b2 = parseInt(m2,16);
            var maskedb1 = b1 & 0x1F;
            var maskedb2 = b2 & 0x3F;
            var result = (maskedb1 << 6) | maskedb2;
            return String.fromCharCode(result);
        })
    }

1

For python, there is an encoding called raw-unicode-escape which will parse these \u00xx characters, so you don't need to use regular expressions.

Further, we can re-encode as pure 8-bit binary using the latin-1 encoding. This encoding is the very obvious and naïve encoding which treats the strings as a series of 8-bit bytes. Then, when we decode as UTF-8, everything is fine again.

import json

with open("broken-fb-encoding.json", "r", encoding="raw-unicode-escape") as f:
    broken_text = f.read()
text = broken_text.encode("latin-1").decode("utf8")
obj = json.loads(text)
1
  • It turns out raw-unicode-escape is a little better because it doesn't try to escape a lone `` character, so I've edited the answer to reflect this. Commented Nov 16, 2023 at 8:13
1

If someone is looking for GO version of the code, here it is:

func decode(s string) string {
    // Create a slice to hold the individual runes
    var runeSlice []rune
    // Convert the string to a slice of runes
    for _, r := range s {
        runeSlice = append(runeSlice, r)
    }

    // Create a byte slice from the rune slice
    byteSlice := make([]byte, len(runeSlice))
    for i, r := range runeSlice {
        byteSlice[i] = byte(r)
    }

    // Convert the byte slice to a UTF-8 string
    utf8String := string(byteSlice)

    // Validate that the string is valid UTF-8
    if !utf8.ValidString(utf8String) {
        // Handle invalid UTF-8
        fmt.Println("Invalid UTF-8 string")
        return ""
    }

    return utf8String
}
0

If your file contains unicodes (\u) you have to read it binary and convert them

const text = "\u00c5\u009f\u00c4\u00b1r\u00c4\u00b1 \u00c3\u0096nemli"
Buffer.from(text, "binary").toString("utf8") // 'Çok önemli'

Not the answer you're looking for? Browse other questions tagged or ask your own question.