Python provides several ways to download files from the internet. This can be done over HTTP using the urllib package or the requests library. This tutorial will discuss how to use these libraries to download files from URLs using Python. The requests library is one of the most popular libraries in. The urllib2 module can be used to download data from the web (network resource access). This data can be a file, a website or whatever you want Python to download. The module supports HTTP, HTTPS, FTP and several other protocols.
Active2 years, 11 months ago
![Python Urllib2 Download File Python Urllib2 Download File](/uploads/1/2/6/0/126082423/784701649.jpg)
Two part question. I am trying to download multiple archived Cory Doctorow podcasts from the internet archive. The old one's that do not come into my iTunes feed. I have written the script but the downloaded files are not properly formatted.
Q1 - What do I change to download the zip mp3 files?Q2 - What is a better way to pass the variables into URL?
This script was adapted from here
JustjoeJustjoe
1811 gold badge3 silver badges10 bronze badges
2 Answers
Here's how I'd deal with the url building and downloading. I'm making sure to name the file as the basename of the url (the last bit after the trailing slash) and I'm also using the
dcolishdcolish
with
clause for opening the file to write to. This uses a ContextManager which is nice because it will close that file when the block exits. In addition, I use a template to build the string for the url. urlopen
doesn't need a request object, just a string.
15.1k1 gold badge20 silver badges25 bronze badges
An older solution on SO along the lines of what you want:
Community♦
pyfuncpyfunc
55.1k13 gold badges131 silver badges130 bronze badges
Not the answer you're looking for? Browse other questions tagged pythonurllib2 or ask your own question.
Active1 year, 8 months ago
Python Urllib2 Download File Mac
I'm downloading an entire directory from a web server. It works OK, but I can't figure how to get the file size before download to compare if it was updated on the server or not. Can this be done as if I was downloading the file from a FTP server?
@Jon: thank for your quick answer. It works, but the filesize on the web server is slightly less than the filesize of the downloaded file.
Examples:
It has anything to do with the CR/LF conversion?
Prashant Kumar
Open Download File
12.5k13 gold badges41 silver badges60 bronze badges
PabloGPabloG
20.1k10 gold badges40 silver badges54 bronze badges
7 Answers
I have reproduced what you are seeing:
Outputs this:
What am I doing wrong here? Is os.stat().st_size not returning the correct size?
Edit:OK, I figured out what the problem was:
this outputs:
Make sure you are opening both files for binary read/write.
Jonathan WorksJonathan Works
1,6251 gold badge15 silver badges13 bronze badges
Using the returned-urllib-object method
info()
, you can get various information on the retrived document. Example of grabbing the current Google logo:
It's a dict, so to get the size of the file, you do
urllibobject.info()['Content-Length']
And to get the size of the local file (for comparison), you can use the os.stat() command:
dbrdbr
125k58 gold badges258 silver badges320 bronze badges
The size of the file is sent as the Content-Length header. Here is how to get it with urllib:
Jonathan WorksJonathan Works
Python Download File Using Urllib2
![File File](/uploads/1/2/6/0/126082423/379325312.jpg)
1,6251 gold badge15 silver badges13 bronze badges
Also if the server you are connecting to supports it, look at Etags and the If-Modified-Since and If-None-Match headers.
Using these will take advantage of the webserver's caching rules and will return a 304 Not Modified status code if the content hasn't changed.
Jonathan WorksJonathan Works
1,6251 gold badge15 silver badges13 bronze badges
A requests-based solution using HEAD instead of GET (also prints HTTP headers):
Usage
ccpizzaccpizza
14k8 gold badges92 silver badges96 bronze badges
MadhusudhanMadhusudhan
For a python3 (tested on 3.5) approach I'd recommend:
yukashima huksayyukashima huksay
1,4092 gold badges15 silver badges38 bronze badges