We recently migrated our application to JDK 8 from JDK 7. After the change, we ran into a problem with the following snippet of code.
String output = new String(byteArray, "UTF-8");
The byte array may contain invalid UTF-8 byte sequences. The same byte array upon UTF-8 decoding, results in two difference strings on Java 7 and Java 8.
According to the answer to this SO post, Java 8 "fixes" an error in Java 7 and replaces invalid UTF-8 byte sequences with a replacement string, which is in accordance with the UTF-8 specification.
But we would like to stick with Java 7's version of the decoded string.
We have tried to use CharsetDecoder with CodingErrorAction as REPLACE, REPORT and IGNORE on Java 8. Still, we were not able to generate the same string as Java 7.
Can we do this with a technique of reasonable complexity?
byteArray
(minimal excerpt from it), so we can reproduce your problem.CodingErrorAction
won’t help you. Think ofUTF-8
andmodified UTF-8
as being two entirely different encodings. In that case you wouldn’t expect an error recovery option switching to another encoding, would you? So what you need then, is an alternativeCharset
implementation, but that wouldn’t be simpler than the five lines of the linked answer.UTF-8
? What is the purpose of this?0x80-0xff
will form an invalid character and produce the replacement character, even under Java 7. The difference lies only in the few cases where the bytes happened to form a surrogate character (by pure accident), but the solution in the linked question only works for valid sequences (valid regarding modified UTF-8) as it flags errors via Exception rather than producing replacement characters. Oh, and\0
is handled differently.CharsetDecoder
and theDataInputStream
handle errors. I’m not even sure whether it’s possible to getDataInputStream
into producing the same behavior. If you are that deep into the matter, your are more than halfway to implementing your own decoder.