203

I've been going through the Q&A on this site, for an answer to my question. However, I'm a beginner and I find it difficult to understand some of the solutions. I need a very basic solution.

Could someone please explain a simple solution to 'Downloading a file through http' and 'Saving it to disk, in Windows', to me?

I'm not sure how to use shutil and os modules, either.

The file I want to download is under 500 MB and is an .gz archive file.If someone can explain how to extract the archive and utilise the files in it also, that would be great!

Here's a partial solution, that I wrote from various answers combined:

import requests
import os
import shutil

global dump

def download_file():
    global dump
    url = "http://randomsite.com/file.gz"
    file = requests.get(url, stream=True)
    dump = file.raw

def save_file():
    global dump
    location = os.path.abspath("D:\folder\file.gz")
    with open("file.gz", 'wb') as location:
        shutil.copyfileobj(dump, location)
    del dump

Could someone point out errors (beginner level) and explain any easier methods to do this?

1
  • note if you are downloading from pycharm note that who knows where the "current folder is" Aug 10, 2021 at 17:45

9 Answers 9

228

A clean way to download a file is:

import urllib

testfile = urllib.URLopener()
testfile.retrieve("http://randomsite.com/file.gz", "file.gz")

This downloads a file from a website and names it file.gz. This is one of my favorite solutions, from Downloading a picture via urllib and python.

This example uses the urllib library, and it will directly retrieve the file form a source.

11
  • 3
    Ok, thanks! But is there a way to get it working through requests?
    – arvindch
    Oct 26, 2013 at 15:52
  • 5
    Any possibility to save in /myfolder/file.gz ? Mar 16, 2014 at 17:57
  • 18
    No better possibility than trying it yourself, maybe? :) I could successfully do testfile.retrieve("http://example.com/example.rpm", "/tmp/test.rpm").
    – Dharmit
    Sep 26, 2014 at 5:47
  • 32
    This is deprecated since Python 3.3, and the urllib.request.urlretrieve solution (see answer below) is the 'modern' way
    – MichielB
    Feb 15, 2017 at 9:14
  • 1
    What is the best way to add a username and password to this code? tks
    – Estefy
    Sep 17, 2017 at 22:06
202

For Python3+ URLopener is deprecated. And when used you will get error as below:

url_opener = urllib.URLopener() AttributeError: module 'urllib' has no attribute 'URLopener'

So, try:

import urllib.request 
urllib.request.urlretrieve(url, filename)
5
  • 11
    Weird... Why nobody votes for this answer when Python 2 became deprecated and only this solution should work properly...
    – wowkin2
    Feb 6, 2020 at 21:33
  • 3
    Agreed! I was pulling my hair over the earlier solutions. Wish I could upvote 200 times!
    – Yechiel K
    Feb 21, 2020 at 1:19
  • 1
    how do indicate which folder/path to save the contents of the url? Aug 10, 2021 at 17:35
  • 1
    note if you are downloading from pycharm note that who knows where the "current folder is" Aug 10, 2021 at 17:46
  • You deserve more upvotes. I don't understand why solutions for Python 2 are still accepted. Nov 12, 2022 at 17:09
120

As mentioned here:

import urllib
urllib.urlretrieve ("http://randomsite.com/file.gz", "file.gz")

EDIT: If you still want to use requests, take a look at this question or this one.

9
  • 2
    urllib will work, however, many people seem to recommend the use of requests over urllib. Why's that?
    – arvindch
    Oct 26, 2013 at 15:51
  • 2
    requests is extremely helpful compared to urllib when working with a REST API. Unless, you are looking to do a lot more, this should be good.
    – dparpyani
    Oct 26, 2013 at 16:41
  • Ok, now I've read the links you've provided for requests usage. I'm confused about how to declare the file path, for saving the download. How do I use os and shutil for this?
    – arvindch
    Oct 26, 2013 at 17:30
  • 83
    For Python3: import urllib.request urllib.request.urlretrieve(url, filename)
    – Flash
    May 30, 2014 at 14:04
  • 1
    I am not able to extract the http status code with this if the download fails Sep 29, 2014 at 23:31
45

Four methods using wget, urllib and request.

#!/usr/bin/python
import requests
from StringIO import StringIO
from PIL import Image
import profile as profile
import urllib
import wget


url = 'https://tinypng.com/images/social/website.jpg'

def testRequest():
    image_name = 'test1.jpg'
    r = requests.get(url, stream=True)
    with open(image_name, 'wb') as f:
        for chunk in r.iter_content():
            f.write(chunk)

def testRequest2():
    image_name = 'test2.jpg'
    r = requests.get(url)
    i = Image.open(StringIO(r.content))
    i.save(image_name)

def testUrllib():
    image_name = 'test3.jpg'
    testfile = urllib.URLopener()
    testfile.retrieve(url, image_name)

def testwget():
    image_name = 'test4.jpg'
    wget.download(url, image_name)

if __name__ == '__main__':
    profile.run('testRequest()')
    profile.run('testRequest2()')
    profile.run('testUrllib()')
    profile.run('testwget()')

testRequest - 4469882 function calls (4469842 primitive calls) in 20.236 seconds

testRequest2 - 8580 function calls (8574 primitive calls) in 0.072 seconds

testUrllib - 3810 function calls (3775 primitive calls) in 0.036 seconds

testwget - 3489 function calls in 0.020 seconds

1
  • 1
    How did you get the number of function calls?
    – Abdelhak
    Sep 2, 2018 at 19:43
38

I use wget.

Simple and good library if you want to example?

import wget

file_url = 'http://johndoe.com/download.zip'

file_name = wget.download(file_url)

wget module support python 2 and python 3 versions

6

Exotic Windows Solution

import subprocess

subprocess.run("powershell Invoke-WebRequest {} -OutFile {}".format(your_url, filename), shell=True)
3

For text files, you can use:

import requests

url = 'https://WEBSITE.com'
req = requests.get(url)
path = "C:\\YOUR\\FILE.html"

with open(path, 'wb') as f:
    f.write(req.content)
2
  • Don't you have to req.iter_content()? Or use the req.raw file object? See this Sep 21, 2020 at 8:19
  • No, it just works, haven't you tried? @MichaelSchnerring
    – DaWe
    Sep 21, 2020 at 8:46
2

I started down this path because ESXi's wget is not compiled with SSL and I wanted to download an OVA from a vendor's website directly onto the ESXi host which is on the other side of the world.

I had to disable the firewall(lazy)/enable https out by editing the rules(proper)

created the python script:

import ssl
import shutil
import tempfile
import urllib.request
context = ssl._create_unverified_context()

dlurl='https://somesite/path/whatever'
with urllib.request.urlopen(durl, context=context) as response:
    with open("file.ova", 'wb') as tmp_file:
        shutil.copyfileobj(response, tmp_file)

ESXi libraries are kind of paired down but the open source weasel installer seemed to use urllib for https... so it inspired me to go down this path

-6

Another clean way to save the file is this:

import csv
import urllib

urllib.retrieve("your url goes here" , "output.csv")
2
  • This should probably be urllib.urlretrieve or urllib.URLopener().retrieve, unclear which you meant here.
    – mateor
    Mar 24, 2016 at 21:25
  • 12
    Why do you import csv if you're just naming a file?
    – Azeezah M
    Jun 29, 2016 at 10:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.