I have a huge text file to process, and my code is the following:
freq_counter = collections.Counter()
unigram_counter = collections.Counter()
with open(filename, 'r', encoding='utf8') as f:
for i, batch in enumerate(read_batch(f, batch_size=1000)):
logger.info("Batch {}".format(str(i)))
frequency_with_batch(batch, freq_counter, unigram_counter, enable)
if data_batch == i+1:
break
def read_batch(file_handle, batch_size=1000):
batch = []
for line in file_handle:
if not line:
continue
batch.append(line)
if len(batch) == batch_size:
yield batch
batch.clear()
if batch:
yield batch
The encoding='utf8' has been working well until it processed the middle of the big text file. On a certain line, it reported the following error:
File "/data5/congmin/tool/utils/my_utils.py", line 484, in read_batch
for line in file_handle:
File "/usr/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 6084-6085: invalid continuation byte
Does this mean some text are utf8 encoded and some are not utf8 in the same file? I did some internet search and some suggested using encoding='latin-1' or encoding="ISO-8859-1". I generally have been using 'utf8' for reading and writing to file. For a big file like this, or a lot of text files, how should I use the encoding parameter if utf8 works most of the time?
EDIT: I changed the encoding=utf8 to ISO-8859-1, the error message was gone. However, the characters output to text file are not readable as below:
ï¼ 2225
ä¸ 1412
533
å 467
å¤ 418
å 417
ADDITION:
I installed 'file' command on my ubuntu and found the file encoding:
file all.txt
all.txt: UTF-8 Unicode text, with very long lines
so it is actually utf-8 file. If it is utf8, why does it produce the error?