5

I'm generating a large dataframe (1.5 GB when saved in CSV format) and need to store it an worksheet of an Excel file along with a second (much smaller) dataframe which is saved in a separate worksheet.

print('Reading temporaty files for variable {}:'.format(Var))
print(' Reading stations')
s=pd.read_csv(StatFile,sep=':',dtype={'ID': 'str'},encoding='utf-8')
print(' Reading data')
d=pd.read_csv(DataFile,sep=':',dtype='str',encoding='utf-8').transpose()
d.columns = d.iloc[0]
d=d[1:].astype('float')
d.reindex_axis(sorted(d.columns), axis=1)
print('Writing out Excel file for variable {}'.format(Var))
writer = pd.ExcelWriter(Path + Var + '.xlsx', engine='xlsxwriter')
d.to_excel(writer, sheet_name='Data')
OutStatCol=['ID','Name','Longitude','Latitude','GRS','OriginalVariable','VariableUnits','URL','JsonNode']
s.to_excel(writer, columns=OutStatCol, index=False, sheet_name='Stations')
writer.save()

My code works fine for smaller dataframes, but with the large ones I get the following error:

Traceback (most recent call last):
  File "./Test2.py", line 29, in <module>
    writer.save()
  File "/home/user/miniconda2/lib/python2.7/site-packages/pandas/io/excel.py", line 1413, in save
    return self.book.close()
  File "/home/user/miniconda2/lib/python2.7/site-packages/xlsxwriter/workbook.py", line 297, in close
    self._store_workbook()
  File "/home/user/miniconda2/lib/python2.7/site-packages/xlsxwriter/workbook.py", line 624, in _store_workbook
    xlsx_file.write(os_filename, xml_filename)
  File "/home/user/miniconda2/lib/python2.7/zipfile.py", line 1148, in write
    self._writecheck(zinfo)
  File "/home/user/miniconda2/lib/python2.7/zipfile.py", line 1114, in _writecheck
    " would require ZIP64 extensions")
zipfile.LargeZipFile: Filesize would require ZIP64 extensions

Is there any way I can specify something like allowZip64=True in the ExcelWriter declaration or in the to_excel() method?

Thanks!

5
  • Have you tried splitting it up in half?
    – jftuga
    Oct 21, 2016 at 18:47
  • I need the entire DataFrame stored in one worksheet and, if I understand correctly what you are suggesting, splitting it in half does not seem to achieve my goal... Oct 21, 2016 at 19:03
  • almost there... it's a keyword you need in pd.ExcelWriter(...)
    – Aaron
    Oct 21, 2016 at 19:15
  • I tried both allowZip64=True as an option and .use_zip64() as an attribute... both returned errors Oct 21, 2016 at 19:25
  • @user6357781 found it yo.. tell me if this works
    – Aaron
    Oct 21, 2016 at 19:36

2 Answers 2

12

This took some source code digging, but...

print('Reading temporaty files for variable {}:'.format(Var))
print(' Reading stations')
s=pd.read_csv(StatFile,sep=':',dtype={'ID': 'str'},encoding='utf-8')
print(' Reading data')
d=pd.read_csv(DataFile,sep=':',dtype='str',encoding='utf-8').transpose()
d.columns = d.iloc[0]
d=d[1:].astype('float')
d.reindex_axis(sorted(d.columns), axis=1)
print('Writing out Excel file for variable {}'.format(Var))
writer = pd.ExcelWriter(Path + Var + '.xlsx', engine='xlsxwriter')

#THIS
writer.book.use_zip64()

d.to_excel(writer, sheet_name='Data')
OutStatCol=['ID','Name','Longitude','Latitude','GRS','OriginalVariable','VariableUnits','URL','JsonNode']
s.to_excel(writer, columns=OutStatCol, index=False, sheet_name='Stations')
writer.save()

should work

figuring out that the writer didn't inherit from workbook took me longer than it should have. writer.book is directly a workbook instance... d'oh

4
  • 2
    ctl - f is your best friend when looking through source... also notepad++ :)
    – Aaron
    Oct 21, 2016 at 19:31
  • 1
    Thanks a lot, it works. Unfortunately after a few hours of processing the final excel file ended up being corrupt... I'll run it again an check it I can g Oct 22, 2016 at 1:48
  • I have the same issue, but this solution does not seem to work on Databricks. Any idea?
    – rpd
    Jul 17, 2020 at 12:51
  • Same here, add xl_writer.book.use_zip64() but in the end the excel is corrupted. Jan 21, 2021 at 16:09
0

I just added engine='xlsxwriter' to the function .to_excel() and it's fixed the problem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.