157

If I have a URL that, when submitted in a web browser, pops up a dialog box to save a zip file, how would I go about catching and downloading this zip file in Python?

2
  • 1
    I tried section Downloading a binary file and writing it to disk of this page which worked as a chram. Oct 3, 2018 at 13:23
  • For anyone else looking for a solution with only the standard library stuff, then check out this answer - stackoverflow.com/a/31706921/8604951 and save yourself a couple of minutes of reading and scrolling through the rest of the answers which uses requests (which btw is an amazing library and I would use it if it was possible to do son.
    – Jarmos
    Aug 20, 2023 at 8:32

9 Answers 9

312

As far as I can tell, the proper way to do this in Python 2 is:

import requests, zipfile, StringIO
r = requests.get(zip_file_url, stream=True)
z = zipfile.ZipFile(StringIO.StringIO(r.content))
z.extractall()

of course you'd want to check that the GET was successful with r.ok.

For python 3+, sub the StringIO module with the io module and use BytesIO instead of StringIO: Here are release notes that mention this change.

import requests, zipfile, io
r = requests.get(zip_file_url)
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall("/path/to/destination_directory")
13
  • Thanks for this answer. I used it to solve my issue getting a zip file with requests. May 2, 2016 at 20:22
  • 26
    If you'd like to save the downloaded file in a different location, replace z.extractall() with z.extractall("/path/to/destination_directory")
    – user799188
    Oct 14, 2016 at 8:14
  • 2
    If you just want to save the file from the url you can do: urllib.request.urlretrieve(url, filename).
    – yoavram
    Jan 2, 2018 at 7:24
  • 8
    To help others connect the dots it took me 60minutes too long to, you can then use pd.read_table(z.open('filename')) with the above. Useful if you have a zip url link that contains multiple files and you're only interested in loading one.
    – Frikster
    Apr 20, 2018 at 6:02
  • 1
    When I do z = zipfile.ZipFile(io.BytesIO(r.content)), I get zipfile.BadZipFile: File is not a zip file. Oct 12, 2023 at 14:16
81

Most people recommend using requests if it is available, and the requests documentation recommends this for downloading and saving raw data from a url:

import requests 

def download_url(url, save_path, chunk_size=128):
    r = requests.get(url, stream=True)
    with open(save_path, 'wb') as fd:
        for chunk in r.iter_content(chunk_size=chunk_size):
            fd.write(chunk)

Since the answer asks about downloading and saving the zip file, I haven't gone into details regarding reading the zip file. See one of the many answers below for possibilities.

If for some reason you don't have access to requests, you can use urllib.request instead. It may not be quite as robust as the above.

import urllib.request

def download_url(url, save_path):
    with urllib.request.urlopen(url) as dl_file:
        with open(save_path, 'wb') as out_file:
            out_file.write(dl_file.read())

Finally, if you are using Python 2 still, you can use urllib2.urlopen.

from contextlib import closing

def download_url(url, save_path):
    with closing(urllib2.urlopen(url)) as dl_file:
        with open(save_path, 'wb') as out_file:
            out_file.write(dl_file.read())
0
20

With the help of this blog post, I've got it working with just requests. The point of the weird stream thing is so we don't need to call content on large requests, which would require it to all be processed at once, clogging the memory. The stream avoids this by iterating through the data one chunk at a time.

url = 'https://www2.census.gov/geo/tiger/GENZ2017/shp/cb_2017_02_tract_500k.zip'

response = requests.get(url, stream=True)
with open('alaska.zip', "wb") as f:
    for chunk in response.iter_content(chunk_size=512):
        if chunk:  # filter out keep-alive new chunks
            f.write(chunk)
3
  • 3
    Answers should not rely on links for the bulk of their content. Links can go dead, or the content on the other side can be changed to no longer answer the question. Please edit your answer to include a summary or explanation of the information you link points to.
    – mypetlion
    Jul 11, 2018 at 19:57
  • What is chunk_size here? And can this parameter affect the speed of downloading? Feb 10, 2021 at 18:12
  • 1
    @ayushthakur Here are some links that may help: requests.Response.iter_content and wikipedia:Chunk Transfer Encoding. Someone else could probably give a better answer, but I wouldn't expect chunk_size to make of a difference for download speed if it's set large enough (reducing #pings/content ratio). 512 bytes seems super small in retrospect. Feb 10, 2021 at 19:46
15

Here's what I got to work in Python 3:

import zipfile, urllib.request, shutil

url = 'http://www....myzipfile.zip'
file_name = 'myzip.zip'

with urllib.request.urlopen(url) as response, open(file_name, 'wb') as out_file:
    shutil.copyfileobj(response, out_file)
    with zipfile.ZipFile(file_name) as zf:
        zf.extractall()
3
  • Hello. How can avoid this error: urllib.error.HTTPError: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.? Jul 24, 2019 at 7:36
  • @VictorHerasmePerez, an HTTP 302 response status code means that the page has been moved. I think the issue your facing is addressed here: stackoverflow.com/questions/32569934/…
    – Webucator
    Jul 24, 2019 at 11:34
  • @Webucator What if the zipped folder contains several files, then all those files will get extracted and stored in the system.I want to extract and get just one file from the zipped folder. Any way to achieve this? Apr 28, 2021 at 7:41
9

Super lightweight solution to save a .zip file to a location on disk (using Python 3.9):

import requests

url = r'https://linktofile'
output = r'C:\pathtofolder\downloaded_file.zip'

r = requests.get(url)
with open(output, 'wb') as f:
    f.write(r.content)
4
7

I came here searching how to save a .bzip2 file. Let me paste the code for others who might come looking for this.

url = "http://api.mywebsite.com"
filename = "swateek.tar.gz"

response = requests.get(url, headers=headers, auth=('myusername', 'mypassword'), timeout=50)
if response.status_code == 200:
with open(filename, 'wb') as f:
   f.write(response.content)

I just wanted to save the file as is.

6

Either use urllib2.urlopen, or you could try using the excellent Requests module and avoid urllib2 headaches:

import requests
results = requests.get('url')
#pass results.content onto secondary processing...
2
  • 1
    But how do you parse results.content int a zip?
    – 0atman
    Mar 9, 2012 at 12:02
  • 1
    Use the zipfile module: zip = zipfile.ZipFile(results.content). Then just parse through the files using ZipFile.namelist(), ZipFile.open(), or ZipFile.extractall()
    – aravenel
    Mar 10, 2012 at 16:30
4

Thanks to @yoavram for the above solution, my url path linked to a zipped folder, and encounter an error of BADZipfile (file is not a zip file), and it was strange if I tried several times it retrieve the url and unzipped it all of sudden so I amend the solution a little bit. using the is_zipfile method as per here

r = requests.get(url, stream =True)
check = zipfile.is_zipfile(io.BytesIO(r.content))
while not check:
    r = requests.get(url, stream =True)
    check = zipfile.is_zipfile(io.BytesIO(r.content))
else:
    z = zipfile.ZipFile(io.BytesIO(r.content))
    z.extractall()
3

Use requests, zipfile and io python packages.

Specially BytesIO function is used to keep the unzipped file in memory rather than saving it into the drive.

import requests
from zipfile import ZipFile
from io import BytesIO

r = requests.get(zip_file_url)
z = ZipFile(BytesIO(r.content))    
file = z.extract(a_file_to_extract, path_to_save)
with open(file) as f:
    print(f.read())
1
  • Thank you! Can't believe I had to scroll all the way to the last answer to find one that used requests and didn't write to a file.
    – ipetrik
    Dec 24, 2022 at 1:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.