2

It's not a real problem in practice, since I can just write BOM = "\uFEFF"; but it bugs me that I have to hard-code a magic constant for such a basic thing. [Edit: And it's error prone! I had accidentally written the BOM as \uFFFE in this question, and nobody noticed. It even led to an incorrect proposed solution.] Surely python defines it in a handy form somewhere?

Searching turned up a series of constants in the codecs module: codecs.BOM, codecs.BOM_UTF8, and so on. But these are bytes objects, not strings. Where is the real BOM?

This is for python 3, but I would be interested in the Python 2 situation for completeness.

9
  • 1
    Only way I know would be codecs.BOM.decode("utf-16-be"), as far as I know using the constants from codecs will always give you bytes Oct 3, 2015 at 12:45
  • Thanks, if the BOM really isn't defined anywhere else then this is the best answer I've seen yet.
    – alexis
    Oct 4, 2015 at 12:58
  • Except your answer is wrong! :-) It converts the bytes into \uFFFE, which I had incorrectly written, but the BOM is \uFEFF. This variant is more verifiably correct: codecs.BOM_UTF8.decode("utf-8"). If this isn't proof that it would be useful to have the BOM available as a unicode string, I don't know what is...
    – alexis
    Oct 4, 2015 at 13:10
  • 1
    I wonder would codecs.BOM_BE.decode("utf-16-be") or codecs.BOM_LE.decode("utf-16-le") work regardless? Oct 4, 2015 at 13:38
  • 1
    It should, shouldn't it! I'd still go with decoding BOM_UTF8, it's easier to verify by inspection...
    – alexis
    Oct 4, 2015 at 13:55

2 Answers 2

1

There isn't one. The bytes constants in codecs are what you should be using.

This is because you should never see a BOM in decoded text (i.e., you shouldn't encounter a string that actually encodes the code point U+FEFF). Rather, the BOM exists as a byte pattern at the start of a stream, and when you decode some bytes with a BOM, the U+FEFF isn't included in the output string. Similarly, the encoding process should handle adding any necessary BOM to the output bytes---it shouldn't be in the input string.

The only time a BOM matters is when either converting into or converting from bytes.

5
  • 1
  • @PadraicCunningham What's your point? Nothing in my answer conflicts with those answers (some of which are quite poor). It's also consistent with the actual behaviour of the utf-8-sig, utf-16, and utf-32 codecs (which automatically insert and strip BOMs).
    – 一二三
    Oct 3, 2015 at 14:39
  • 1
    Thanks for the effort but I do need to write a BOM, as in the question Padraic linked to. To write a byte string I'd have to open the file in binary mode, then reopen it in text mode to for the real contents. The BOM in string form is the only way to go.
    – alexis
    Oct 4, 2015 at 12:53
  • 1
    PS. While you're right about its uses, the BOM is a real unicode codepoint, and it is indeed representable as a (unicode) string. I.e., it's not an out-of-band concept like parity or "end of file". Nothing prevents it from being included among python's constants.
    – alexis
    Oct 4, 2015 at 14:02
  • It's representable, but its use as a proper Unicode character is discouraged: unicode.org/faq/utf_bom.html#bom6
    – 一二三
    Oct 4, 2015 at 14:15
1

I suppose you could use:

unicodedata.lookup('ZERO WIDTH NO-BREAK SPACE')

but it's not as clean as what you already have

1
  • Haha, good point but as "magic constants" go this is arguably worse than just writing \uFEFF! :-) I know this is the codepoint that became the BOM, but that's a historical accident.
    – alexis
    Oct 4, 2015 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.