1

I have a simple and annoying problem, and I apologize for not posting an example. The files are big and I haven't been able to recreate the exact issue using smaller files:

These are tab-delimited files (some entries contain " ; or a single space character). On UNIX, when I access a unique word via: nl file | sed -n '/word/p' I see that my word is on exactly the same line in all my files.

Now I copy the files to my mac. I run the same command on the same exact files, but the line numbers are all different! The total number of lines via wc -l is still identical to the numbers I get in unix, but when I do nl file | tail -n1 I see a different number. Yet, when I enter the number returned from my unix nl, and access the same line via sed '12345p' file I get the correct entry!?

My question: I must have something in some of my lines that is interpreted as linebreaks on my mac but not in unix, and only by nl not sed. Can anyone help me figure out what it is? I already know it's not on every line. I found this issue persists when I load the data into R, and I'm stumped. Thank you!

19
  • How did you copy the files? Did the copy translate line endings or are they identical byte for byte on Unix and Mac?
    – acfrancis
    Nov 26, 2013 at 1:26
  • I copied them using scp. I've never noticed a difference before between a unix file and a mac file.
    – reviewer3
    Nov 26, 2013 at 1:30
  • Yes, they are identical, byte for byte:
    – reviewer3
    Nov 26, 2013 at 1:31
  • 17e4759590d804ecb5c44b17982939ae (unix md5sum)
    – reviewer3
    Nov 26, 2013 at 1:31
  • 1
    Very odd indeed. Redirect the output of nl on each system to a separate text file and compare them with comm -3 nl_unix.txt nl_mac.txt | head -1 to see where the errors start. (Easier than my binary search suggestion above)
    – acfrancis
    Nov 26, 2013 at 2:16

1 Answer 1

1

"Phantom newlines" can be hidden in text in the form of a multi-byte UTF-8 character called an "overlong sequence".

UTF-8 normally represents ASCII characters as themselves: UTF-8 bytes in the range 0 to 127 are just those character values. However, overlong sequences can be used to (incorrectly) encode ASCII characters using multiple UTF-8 bytes (which are in the range 0x80-0xFF). A properly written UTF-8 decoder must detect overlong sequences and somehow flag them as invalid bytes. A naively written UTF-8 decoder will simply extract the implied character.

Thus, it's possible that your data is being treated as UTF-8, and contains some bytes which look like an overlong sequence for a newline, and that this is fooling some of the software you are working with. A two-byte overlong sequence for a newline would look like C0 8A, and a three-byte overlong sequence would be E0 80 8A.

It is hard to come up with an alternative hypothesis not involving character encodings.

1
  • Thank you, Kaz - I have to look into that. I ended up rewriting my scripts and didn't run into the same issue though. I'll see whether I can dig out the old files.
    – reviewer3
    Jul 31, 2014 at 2:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.