8

We recently migrated our application to JDK 8 from JDK 7. After the change, we ran into a problem with the following snippet of code.

String output = new String(byteArray, "UTF-8");

The byte array may contain invalid UTF-8 byte sequences. The same byte array upon UTF-8 decoding, results in two difference strings on Java 7 and Java 8.

According to the answer to this SO post, Java 8 "fixes" an error in Java 7 and replaces invalid UTF-8 byte sequences with a replacement string, which is in accordance with the UTF-8 specification.

But we would like to stick with Java 7's version of the decoded string.

We have tried to use CharsetDecoder with CodingErrorAction as REPLACE, REPORT and IGNORE on Java 8. Still, we were not able to generate the same string as Java 7.

Can we do this with a technique of reasonable complexity?

10
  • 2
    Please post exact input byteArray (minimal excerpt from it), so we can reproduce your problem. Jun 1, 2015 at 14:30
  • 1
    If your issue is indeed that there are wrongly encoded surrogate pairs, CodingErrorAction won’t help you. Think of UTF-8 and modified UTF-8 as being two entirely different encodings. In that case you wouldn’t expect an error recovery option switching to another encoding, would you? So what you need then, is an alternative Charset implementation, but that wouldn’t be simpler than the five lines of the linked answer.
    – Holger
    Jun 1, 2015 at 14:49
  • 2
    So you tell it to misinterpret arbitrary data as UTF-8? What is the purpose of this?
    – Holger
    Jun 1, 2015 at 16:19
  • 2
    The problem is that if you put in arbitrary content, almost any byte in the range 0x80-0xff will form an invalid character and produce the replacement character, even under Java 7. The difference lies only in the few cases where the bytes happened to form a surrogate character (by pure accident), but the solution in the linked question only works for valid sequences (valid regarding modified UTF-8) as it flags errors via Exception rather than producing replacement characters. Oh, and \0 is handled differently.
    – Holger
    Jun 1, 2015 at 17:35
  • 1
    That would require knowledge about the (modified) UTF-8 format as well as how both, the Java 7 CharsetDecoder and the DataInputStream handle errors. I’m not even sure whether it’s possible to get DataInputStream into producing the same behavior. If you are that deep into the matter, your are more than halfway to implementing your own decoder.
    – Holger
    Jun 1, 2015 at 18:34

1 Answer 1

8

From the pointers provided by @Holger, It was clear that we had to write a custom CharsetDecoder.

I copied over OpenJDK's version of sun.nio.cs.UTF_8 class, renamed it to CustomUTF_8 and used it to construct a string like so

String output = new String(bytes, new CustomUTF_8());

I plan to run extensive tests cross verifying the outputs generated on Java 7 and Java 8. This is an interim solution while I am trying to fix the actual problem of passing output from hmac directly to String without Base64 encoding it first to.

 String output = new String(Base64.Encoder.encode(bytes), Charset.forname("UTF-8"));
2
  • If testing goes well, it would be a good idea to release it as a library and put into Maven Central, so other people with the same problem can use it. Jun 3, 2015 at 8:18
  • @TagirValeev I think its a bad Idea to make it easy for people to do this. The CharsetDecoder class translates a sequence of bytes in a specific charset to a sequence of sixteen-bit Unicode characters. Patch work at this level is dangerous as I cannot be very sure that a sequence of sixteen-bit Unicode characters is interpreted in across two versions of the JVM.
    – Jiraiya
    Jun 4, 2015 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.