23

I have a document A in encoding A displayed in tool A and a document B in encoding B displayed in tool B. If I cut and paste (part of) B into A what might be the resultant character encoding? I realise this depends on tool A and tool B and the information held in the paste buffer (which presumably can contain an encoding?) and the operating system.

What should high-quality tools do? and in practice how many of the common tools (e.g. Word, TextPad, various IDEs, etc.) do a good job?

3
  • Is this on Windows, Unix, ToasterOS? Dec 18, 2009 at 18:06
  • 1
    Word is a Windows app, and googling for "TextPad" turns up "TextPad - the text editor for Windows", so I'm guessing he means Windows.
    – Ken
    Dec 18, 2009 at 18:08
  • 1
    We have both Unix platforms and Windows so any fundamental difference in the two would be useful. My own machine is Windows so I'l be happy with a Windows answer. Dec 18, 2009 at 18:11

4 Answers 4

22

First of all, a text editor's internal representation of text has no bearing on how the text is encoded (serialized) when you save the file. So a document is not "in" an encoding; it's a sequence of abstract characters. When the document is saved to a file (or transmitted over the network) then it gets encoded.

It's up to each application to decide what it puts on the clipboard. Typically, a windows app that knows what it's doing will put a number of different representations on the clipboard. When you paste in the other app, the app will look for the representation that best suits its need.

In your case, a text editor (that knows what it's doing) will put a Unicode representation of a selected string onto the clipboard (where Unicode, in Windows, is typically moved around as UTF-16, but that's not important). When you paste in the other app, it will insert that sequence of Unicode characters into the document at the selection point.

There's an app floating around called "ClipSpy" that will help you see what I'm talking about, interactively.

5
  • 1
    +1 So a good clipboard will try to normalize one version of the characters to UTF-16/Unicode. Dec 18, 2009 at 18:23
  • 3
    I'm not sure what you mean by that. The clipboard doesn't do anything, other than hold onto some bytes that the application put there. The clipboard acts like a key-value store, where the key is something like a mimetype, and the value is a blob (that you then interpret according to the type). Dec 18, 2009 at 18:25
  • @peter.murray.rust that's somewhat Windows specific. But I'm no expert. Dec 18, 2009 at 20:51
  • 11
    Actually, on Windows CF_TEXT, CF_OEMTEXT, and CF_UNICODETEXT are always present. Add one to the clipboard, and the clipboard with convert and add the others. So the clipboard does something.
    – Mihai Nita
    Dec 19, 2009 at 9:02
  • 1
    on OSX, pbcopy and pbpaste use locale environment variables to determine the encoding, for example LANG=en_US.UTF-8 will cause pbcopy and pbpaste to use UTF-8. If not present, defaults to C.
    – cdosborn
    Apr 16, 2015 at 22:05
2

I observed the following behavior when I looked into Unicode normalization: When copying a canonically decomposed string (NFD) in Firefox in macOS 10.15.7, the string is normalized to NFC when pasting it in Chrome. What's weird is that the pasting affects the content of the clipboard: When pasting the string in Firefox again, it's then also canonically composed there. If I don't paste it anywhere else before pasting it in Firefox again, the NFD form survives. Interestingly, the problem doesn't occur in the other direction: When copying a canonically decomposed string in Chrome, it's pasted in NFD form anywhere I can tell. My conclusion is that Firefox stores text to the clipboard differently from other applications. One way to play around with this yourself is to copy 'mañana' === 'mañana' to your JavaScript console. The statement returns false if the NFD form of the string on the right survived the copy & paste.

5
  • Hi @Kaspar Etter, Is there a way to prevent this? I'm experiencing some similar things as you described and it was driving me crazy. Copying from Safari and pasting back to Safari on Mac twists the string as well. Typing to Safari and hitting the "Submit" button also twists the answer in Safari on Mac. This copy and paste issue is not a problem to Safari on iOS though.
    – Martin
    Mar 5, 2021 at 14:41
  • Just to clarify, in Safari on Mac, on "Inspect Element", copy the string, 'mañana' === 'mañana', in @Kaspar Etter's reply, and paste it to Safari's console. It will be shown as "true".
    – Martin
    Mar 5, 2021 at 14:53
  • Not that I know of. And it's not even clear to me which behavior is more desirable. What bothers me is the lack of documentation by the manufacturers and the lack of consistency across vendors. As far as Safari is concerned, normalization only occurs when pasting the string. Copying from Safari and pasting it into Chrome's console leads to no normalization on my Mac. Mar 5, 2021 at 15:05
  • And for what it's worth, opening <html><body><script>document.write('mañana' === 'mañana')</script></body></html> in Safari outputs false. Mar 5, 2021 at 15:06
  • Hi @Kaspar, that's right. It only happens to pasting. But the normalization also happens at least on Canvas by instructure.com. In Classic Quizzes, when students type Hebrew words in quizzes and hit "submit", the input was normalized, but the answer key was not. In the New Quizzes, both the input and the answer key are normalized. It's a mystery to me. Firefox is interesting. Even when I use AppleScript, JXA, or Python or read directly from the string copied in Firefox, it is still normalized. But typing and submit or copying and pasting in Firefox itself, it will not normalize...
    – Martin
    Mar 5, 2021 at 15:22
1

This is a very good question. When you copy/paste, exactly what is copied/pasted - CHARACTERS or BYTES?. And if BYTES, what encoding are they in?

From the answers, it sounds like the answer is "it depends". Different programs will put different things in the clipboard, sometimes placing multiple representations.

Then the pasting program needs to pick the best one and "do the right thing" with it.

0

Following my conversion with @Kaspar Etter, I did some testing. Here is what I found:

Copy from and Paste to:

Firefox:
Firefox to Firefox: NO normalization
Other apps to Firefox: NO normalization
Firefox to other apps: normalization

Even if we use AppleScript, JXA, or Python to directly read the SystemClipboard that contains the text copied from Firefox, the text is still normalized. Since copying and pasting from Firefox to Firefox does not involve normalization, Firefox probably does not normalize the text during the copy process. I have no idea when the normalization happens.

Safari (MacOS, not iOS):
Safari to Safari: normalization
Other apps to Safari: normalization
Safari to other apps: NO normalization

For Safari (MacOS), the normalization also happens at least on Canvas by instructure.com. In the fill-in-blank questions of Classic Quizzes, when students type Hebrew words in quizzes and hit "submit", the input was normalized, but the answer key was not. In that of the New Quizzes, however, both the input and the answer key are normalized. It's a mystery to me.

Chrome:
Chrome to Chrome: NO normalization
Other apps to Chrome: NO normalization (Firefox overrides)
Chrome to other apps: NO normalization (Safari overrides)

Conclusion: Firefox and Safari behave in the opposite way. Chrome behaves normally and consistently (except when it is overridden by Firefox and Safari).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.