28

Suppose you have a string like "€foo\xA0", encoded UTF-8, Is there a way to remove invalid byte sequences from this string? ( so you get "€foo" )

In ruby-1.8 you could use Iconv.iconv('UTF-8//IGNORE', 'UTF-8', "€foo\xA0") but that is now deprecated. "€foo\xA0".encode('UTF-8') doesn't do anything, since it is already UTF-8. I tried:

"€foo\xA0".force_encoding('BINARY').encode('UTF-8', :undef => :replace, :replace => '')

which yields

"foo"

But that also loses the valid multibyte character €

1

4 Answers 4

35
"€foo\xA0".chars.select(&:valid_encoding?).join
2
  • 1
    It doesn't remove the \xF1 in this string "eEspa\xF1a;FB"
    – Dorian
    Sep 24, 2014 at 15:12
  • 2
    @Dorian, on 1.9.3 IRB console, "eEspa\xF1a;FB".chars.select{|i| i.valid_encoding?}.join returns "eEspaa;FB" ...do you not get that behavior or have I misunderstood?
    – acobster
    Mar 20, 2015 at 17:40
35
"€foo\xA0".encode('UTF-16le', invalid: :replace, replace: '').encode('UTF-8')
10
  • 2
    I was under the impression it has a larger character set than UTF-8, meaning you don't loose any valid data. Unfortunately the following doesn't work: "€foo\xA0".encode('UTF-8', :invalid => :replace, :replace => '') because the string is already UTF-8, so it will not be encoded again. Apr 29, 2012 at 18:09
  • 2
    UTF-8 and UTF-16 can both represent all Unicode characters. The only difference is the way the characters are encoded.
    – Zr40
    Nov 10, 2012 at 11:10
  • 1
    All UTF encodings are equally capable of encoding all possible Unicode characters; there's no difference in that regard between UTF-8, UTF-16 and UTF-32. The only practical difference is the output size.
    – Zr40
    Jun 2, 2013 at 7:09
  • 1
    Throws an error with this string: "eEspa\xF1a;FB"
    – Dorian
    Sep 24, 2014 at 15:12
  • 1
    @VanderHoorn: it was ruby < 2.1 because it works with ruby 2.1+
    – Dorian
    Mar 10, 2015 at 13:05
4

Ruby 2.0 and 1.9.3

"€foo\xA0".encode(Encoding::UTF_8, Encoding::UTF_8, :invalid => :replace)

Ruby 2.1+

"€foo\xA0".scrub

These replace the \xA0 with a � symbol by default, you can specify a different replacement parameter.

0
-2
    data = '' if not (data.force_encoding("UTF-8").valid_encoding?)
2
  • This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post - you can always comment on your own posts, and once you have sufficient reputation you will be able to comment on any post.
    – Severin
    Oct 11, 2014 at 12:08
  • @Severin how come not? It looks like an (incorrect) answer to the question. It removes all invalid byte sequence from a string. It just removes all valid ones as well. Oct 11, 2014 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.