35

How can I add BOM (unicode signature) while saving file in python:

file_old = open('old.txt', mode='r', encoding='utf-8')
file_new = open('new.txt', mode='w', encoding='utf-16-le')
file_new.write(file_old.read())

I need to convert file to utf-16-le + BOM. Now script is working great, except that there is no BOM.

6
  • for line in file_old: file_new.write(line) is vastly more memory efficient. And why can't you use any of the numerous software that already does this?
    – user395760
    Mar 5, 2011 at 8:41
  • Text editors have to open file to "save as", and it is pretty big. Other software is shareware, or hard to find. Besides, I'm just learning python. Save by line maybe more efficient, but is more complex.
    – Qiao
    Mar 5, 2011 at 8:47
  • If the file is pretty big, that may be all the more reason to convert it line by line -- despite its "complexity".
    – martineau
    Mar 5, 2011 at 17:30
  • And it also depends on how often script is executed. In my case file is 100 mb, converted in <10 seconds once a month.
    – Qiao
    Mar 5, 2011 at 17:51
  • 1
    @JohnMachin actually had the correct answer here. Dec 7, 2012 at 22:10

9 Answers 9

46

Write it directly at the beginning of the file:

file_new.write('\ufeff')
3
  • 2
    Thank, it is often very hard to figure out such simple thing for first time.
    – Qiao
    Mar 5, 2011 at 9:29
  • 2
    @Qiao: It's not that simple. See my answer. Apr 20, 2011 at 12:35
  • 1
    @Ocaso: On Python 2.x the lack of u before '\ufeff' might cause a Unicode encoding error, specifically if you are explicitly setting the encoding via codecs.open().
    – ccpizza
    Mar 13, 2016 at 14:29
40

It's better to use constants from 'codecs' module.

import codecs
f.write(codecs.BOM_UTF16_LE)
6
  • 8
    This is actually the wrong answer for two reasons. First, @JohnMachin has the write answer. Don't use 'utf-16le', just use 'utf-18'. Secondly (and more importantly) given that the OP's code sets an encoding this won't result in the correct behavior at all. Especially on Python3. You are giving bytes to a thing that wants a str. Dec 7, 2012 at 21:59
  • 1
    Err.. 'right', not 'write'. That's what happens when you play with code in one window and write comments in another. Dec 7, 2012 at 22:10
  • 5
    @Omnifarious: And utf-16, not utf-18. :p
    – jamesdlin
    Apr 2, 2013 at 18:37
  • @jamesdlin: Oh, oops! I need to do better about typos like that. Apr 2, 2013 at 20:13
  • 6
    In Python 3, using (default) text-mode open, it errors because you toss it bytes, not string, as Omnifarious already hinted. Casting the bytes to a string, as in f.write(str(codecs.BOM_UTF8)), gets you b'\xef\xbb\xbf' at the start of your file.
    – RolfBly
    Aug 6, 2014 at 10:08
34

Just choose the encoding with BOM:

with codecs.open('outputfile.csv', 'w', 'utf-8-sig') as f:
   f.write('a,é')

(In python 3 you can drop the codecs.)

2
  • 1
    or replace codecs module by io, which works on both python 2 and 3 Jan 24, 2019 at 17:24
  • 2
    This 2017 answer should be voted above the 2011 answers, Feb 27, 2022 at 18:53
22

Why do you think you need to specifically make it UTF16LE? Just use 'utf16' as the encoding, Python will write it in your endianness with the appropriate BOM, and all the consumer needs to be told is that the file is UTF-16 ... that's the whole point of having a BOM.

If the consumer is insisting that the file must be encoded in UTF16LE, then you don't need a BOM.

If the file is written the way that you specify, and the consumer opens it with UTF16LE encoding, they will get a \ufeff at the start of the file, which is a nuisance, and needs to be ignored.

1
  • "Why do you think you need to specifically make it UTF16LE?" winapi can be very particular in what it wants, it only accepts UCS-2 LE with BOM
    – jrh
    Apr 5, 2021 at 20:38
6

I had a similar situation where a 3rd party app did not accept the file I generated unless it had a BOM.

For some reason in Python 2.7 the following does not work for me

write('\ufeff')

I had to substitute it with

write('\xff\xfe')

and that is the same as

write(codecs.BOM_UTF16_LE)

my final output file was written with the following code

import codecs
mytext = "Help me"

with open("c:\\temp\\myFile.txt", 'w') as f:
    f.write(codecs.BOM_UTF16_LE)
    f.write(mytext.encode('utf-16-le'))

This answer may be useless for the original asker but it may help someone like me who stumbles upon this issue

5

For UTF-8 with BOM you can use:

def addUTF8Bom(filename):
  f = codecs.open(filename, 'r', 'utf-8')
  content = f.read()
  f.close()
  f2 = codecs.open(filename, 'w', 'utf-8')
  f2.write(u'\ufeff')
  f2.write(content)
  f2.close()
1
  • 1
    Solid! Was writing to .csv file in Python which Excel never really opened correctly due to missing BOM, now it's all good. Thanks! Jan 6, 2016 at 19:00
0

vitperov's answer for python3:

def add_utf8_bom(filename):
    with codecs.open(filename, 'r', 'utf-8') as f:
        content = f.read()
    with codecs.open(filename, 'w', 'utf-8') as f2:
        f2.write('\ufeff')
        f2.write(content)
return
0

TRY IT:

def add_bom(file, bom: bytes):
    with open(file, 'r+b') as f:
        org_contents = f.read()
        f.seek(0)
        f.write(bom + org_contents)

USAGE:

import codecs

...

file = 'test.txt'
with open(file, 'w', encoding='utf-8') as f:  # without BOM
    f.write('A')

add_bom(file, codecs.BOM_UTF16_LE)

# TEST
with open(file, 'rb') as f:
    print(f.read())  # b'\xff\xfeA'
0

My method of adding BOM is by writing ansi characters '" at the beginning of the file, then open file in UTF-8 and write desired data:

# Create file with ANSI encoding
file= open("file.txt", "a", encoding="ansi", errors='ignore')
# Add BOM at the beginning of the file BOM 0xEFBBBF
file.write("")
# Close file
file.close()
# Open file in UTF-8 and write data
file= open("file.txt", "a", encoding="utf-8", errors='ignore')
file.write("Write your data here, Enjoy!!")

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.