3

I was doing some file merging, and two files wouldn't - despite having a key column that matches (I actually generated one key column by copy-pasting from the other). It's the damndest thing, and I worry that I'm either going crazy or missing something fundamental. As an example (and I cannot figure out how to make it reproducible, as when I copy and paste these strings into new objects, they compare just fine), here's my current console:

> q
[1] "1931 80th Anniversary"
> z
[1] "1931 80th Anniversary"
> q == z
[1] FALSE

I str-ed both, just in case I missed something, and...

> str(q)
 chr "1931 80th Anniversary"
> str(z)
 chr "1931 80th Anniversary"

What could be going on here?

4
  • 1
    OK - Encoding() shows that one is UTF-8 and the other is "unknown" - So. Weird. iconv() seems to fix it if I iconv one of them. This is truly bizarre.
    – jebyrnes
    Commented Nov 3, 2020 at 3:47
  • 1
    I can't replicate with the values you provided here so it's not easy to say what's going on. Maybe compare the charToRaw() values for your two strings.
    – MrFlick
    Commented Nov 3, 2020 at 4:29
  • Add dput(q) and dput(z) to your post.
    – Ronak Shah
    Commented Nov 3, 2020 at 5:35
  • 1
    That is most likely not going to help but there might be a "hidden" character in one of your strings. I once experienced that and almost gone crazy as well. It was a soft-hyphen: See: stackoverflow.com/questions/57077145/….
    – Roccer
    Commented Nov 3, 2020 at 7:32

2 Answers 2

11

This was a great puzzler. To answer - to diagnose the problem, charToRaw() was the answer.

> charToRaw(q)
 [1] 31 39 33 31 c2 a0 38 30 74 68 c2 a0 41 6e 6e 69 76 65
[19] 72 73 61 72 79
> charToRaw(z)
 [1] 31 39 33 31 20 38 30 74 68 20 41 6e 6e 69 76 65 72 73
[19] 61 72 79

Oh! Different! It seems to lie in the encoding, which, given that these were both plain ole' CSVs I loaded from, I never would have guessed, but

> Encoding(q)
[1] "UTF-8"
> Encoding(z)
[1] "unknown"

In the end, I used iconv() on q to make it work

> iconv(q, from = 'UTF-8', to = 'ASCII//TRANSLIT') == z
[1] TRUE

This has been a weird journey, and I hope this helps someone else who is as baffled as I was - and they learn a few new functions along the way.

0
3

It looks like you have non-breaking spaces in your string, which isn't really an encoding issue. This happens to me all the time because alt + space inserts a non-breaking space on a Mac, and I use alt on my German keyboard for all sorts of special characters, too. My pinkies are my slowest fingers and they don't always release alt fast enough when I transition from some special character to a space. I discovered this problem writing bash scripts, where <command> | <command> is common and | is alt + 7.

I think stringr::str_replace_all(q, "\\s", " ") should fix your current issue. Alternatively, you can try targeting specific non-printables, e.g. in your situation stringr::str_replace_all(q, "\uA0", " "). To expose the offending characters you can use stringi::stri_escape_unicode(q), which would return "1931\\u00a080th\\u00a0Anniversary". You can then just copy and paste to get the same results as above: stringr::str_replace_all(q, "\u00a0", " ")

1
  • 2
    Yup. That is it. I like your solution! Oddly, it's because I copy-pasted directly from View() in Rstudio - which makes all spaced non-breaking! HA!
    – jebyrnes
    Commented Nov 4, 2020 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.