38

I have multiple zip files containing different types of txt files. Like below:

zip1 
  - file1.txt
  - file2.txt
  - file3.txt

How can I use pandas to read in each of those files without extracting them?

I know if they were 1 file per zip I could use the compression method with read_csv like below:

df = pd.read_csv(textfile.zip, compression='zip') 

Any help on how to do this would be great.

1

4 Answers 4

60

You can pass ZipFile.open() to pandas.read_csv() to construct a pandas.DataFrame from a csv-file packed into a multi-file zip.

Code:

pd.read_csv(zip_file.open('file3.txt'))

Example to read all .csv into a dict:

from zipfile import ZipFile

zip_file = ZipFile('textfile.zip')
dfs = {text_file.filename: pd.read_csv(zip_file.open(text_file.filename))
       for text_file in zip_file.infolist()
       if text_file.filename.endswith('.csv')}
3
  • 3
    What if I have the url of the zipfile? It only works if I have it in my local machine!
    – doplano
    Sep 12, 2022 at 12:07
  • @Farid Alijani True, Please help in case of ZipFile url Nov 8, 2022 at 11:19
  • 2
    @FaridAlijani, @nitinkhatri789, it is generally a bad idea to download an entire dataset each time you run your code. Most of the time it is better to download manually the zip file, and access it locally. If you really need to, you can still use urllib.request.url_retrieve.
    – xavlours
    Dec 14, 2022 at 9:18
8

The most simplest way to handle this (if you have multiple parts of one big csv file compressed to a one zip file).

import pandas as pd
from zipfile import ZipFile

df = pd.concat(
    [pd.read_csv(ZipFile('some.zip').open(i)) for i in ZipFile('some.zip').namelist()],
    ignore_index=True
)
2

I had a similar problem with XML files awhile ago. The zipfile module can get you there.

from zipfile import ZipFile

z = ZipFile(yourfile)

text_files = z.infolist()

for text_file in text_files:
    z.read(text_file.filename)

If you want to concatenate them into a pandas object then it might get a bit more complex, but that should get you started. Note that the read method returns bytes, so you may have to handle that as well.

1

For those who have empty txt files in the zipfile:

from zipfile import ZipFile
z = ZipFile('textfile.zip')
df = pd.concat(
    [pd.read_csv(z.open(i.filename)) for i in z.infolist() if i.compress_size > 0],
    ignore_index=True)

Otherwise, the "pandas.errors.EmptyDataError: No columns to parse from file" would show up.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.