In this tutorial, you will learn how to download files from the web using different Python modules. Additionally, you will download regular files, web pages, Amazon S3, and other resources.
Finally, you’ll learn how to overcome various challenges you may encounter, such as downloading redirected files, downloading large files, completing a multi-threaded download, and other strategies.
Use requests
You can use the requests module to download files from a URL.
Consider the following code:
You simply get the URL using the get method of the requests module and store the result into a variable called “myfile”. Then, write the contents of this variable to the file.
Use wget
You can also download files from a URL using Python’s wget module. **You can use pip to install the wget module with the following command:
Consider the following code, which we will use to download a logo image for Python.
In this code, the URL and path (where the image will be stored) are passed to the download method of the wget module.
Download redirected files
In this section, you will learn how to use requests to download a file from a URL that will be redirected to another URL with a .pdf file. The URL looks like this:
To download this pdf file, use the following code:
In this code, the first step we specify is the URL. Then, **we use the get method of the request module to get the URL. In the get method, we set allow_redirects to True, which will allow redirections in the URL,** and the redirected content will be assigned to the variable myfile.
Finally, we open a file to write the fetched content.
Download large files in chunks
Consider the following code:
First, we’ll use the get method of the requests module as before, but this time, we’ll set the stream attribute to True.
Next, we create a file named PythonBook.pdf in the current working directory and open it for writing.
We then specify the chunk size to download each time. **We’ve set it to 1024 bytes, then iterated through each chunk,** and wrote the chunks in the file until the end of the chunk.
Isn’t it beautiful? Don’t worry, we will show a progress bar of the download process later.
Download multiple files (parallel/batch download)
To download multiple files simultaneously, import the following module:
We imported the os and time modules to check how long it takes to download a file. The ThreadPool module allows you to run multiple threads or processes using a pool.
Let’s create a simple function that chunks the response to a file:
This URL is a two-dimensional array that specifies the path and URL of the page you want to download.
Just like we did in the previous section, we pass this URL to requests.get. **Finally, we open the file (path specified in the URL) and write the page content.
Now, we can call this function for each URL individually, or we can call this function for all URLs at the same time. **Let’s call this function separately for each URL in a for loop, **Pay attention to the timer:
Now, replace the for loop with the following lines of code:
Run the script.
Download using progress bar
The progress bar is a UI component of the clint module. Enter the following command to install the clint module:
Consider the following code:
In this code, we first imported the requests module, and then, we imported the progress component from clint.textui. The only difference is that ** is in the for loop. **When writing content to a file, we use the bar method of the progress bar module.
Use urllib to download web pages
In this section, we will use urllib to download a web page.
The urllib library is Python’s standard library, so you don’t need to install it.
The following lines of code can easily download a web page:
Specify herewhat you want to save the file for and the URL where you want to store it.
In this code, we have used the urlretrieve method and passed the URL of the file, as well as the path to save the file. The file extension will be .html.
Downloading through proxy
If you need to use a proxy to download your files, you can use the urllib module’s ProxyHandler. Look at the following code:
In this code, we create the proxy object, ** and open the proxy by calling urllib’s build_opener method, ** and pass in the proxy object. Then we create a request to get the page.
In addition, you can also use the requests module as described in the official documentation:
You just need to import the requests module and create your proxy object. **Then, you can get the file.
Use urllib3
urllib3 is an improved version of the urllib module. You can download and install it using pip:
We will use urllib3 to get a web page and store it in a text file.
Import the following modules:
When processing files, we use the shutil module.
Now, we initialize the URL string variable like this:
Then, we used urllib3’s PoolManager, which keeps track of the necessary connection pooling.
Create a file:
Finally, we send a GET request to get the URL and open a file, then write the response to the file:
Download files from S3 using Boto3
To download files from Amazon S3, you can use the Python boto3 module.
Before starting, you need to install the awscli module using pip:
For AWS configuration, run the following command:
Now enter your details by pressing:
To download files from Amazon S3, you need to import boto3 and botocore. Boto3 is an Amazon SDK that allows Python to access Amazon web services (such as S3). Botocore provides a command line service for interacting with Amazon web services.
Botocore comes with awscli. To install boto3, run the following command:
Now, import these two modules:
When downloading files from Amazon, we need three parameters:
-
Bucket name
-
The name of the file you need to download
-
The name of the file after downloading
Initialize variables:
Now, **we initialize a variable to use the session’s resources. **To do this, we will call boto3’s resource() method and pass in the service, which is s3:
Finally, use the download_file method to download the file and pass in the variables:
Use asyncio
The asyncio module** is mainly used to handle system events. **It works around an event loop that waits for an event to occur and then reacts to that event. The reaction can be to call another function. This process is called event processing. The asyncio module uses coroutines for event handling.
To use asyncio event handling and coroutine functionality, we will import the asyncio module:
Now, define the asyncio co-method like this:
The keyword async indicates that this is a native asyncio coroutine. Inside the coroutine, we have an await keyword, which returns a specific value. We can also use return keyword.
Now, let’s use co-creation to download a file from a website:
In this code,we create an asynchronous coroutine that downloads our file and returns a message.
We then call main_func using another asynchronous coroutine, which waits for URLs and groups all URLs into a queue. **asyncio’s wait function will wait for the coroutine to complete.
Now, in order to start the coroutine, we have to put the coroutine into an event loop using asyncio’s get_event_loop() method, and finally, we execute that event loop using asyncio’s run_until_complete() method.
Downloading files using Python is fun. Hope this tutorial is useful to you!
Finally:
Python learning materials
If you want to learn Python to help you automate your office, or are preparing to learn Python or are currently learning it, you should be able to use the following and get it if you need it.
① Python learning roadmap for all directions, knowing what to learn in each direction ② More than 100 Python course videos, covering essential basics, crawlers and data analysis ③ More than 100 Python practical cases, learning is no longer just theory ④ Huawei’s exclusive Python comic tutorial, you can also learn it on your mobile phone ⑤Real Python interview questions from Internet companies over the years, very convenient for review
There are ways to get it at the end of the article
1. Learning routes in all directions of Python
The Python all-direction route is to organize the commonly used technical points of Python to form a summary of knowledge points in various fields. Its usefulness is that you can find corresponding learning resources according to the above knowledge points to ensure that you learn more comprehensively.
2. Python course video
When we watch videos and learn, we can’t just move our eyes and brain but not our hands. The more scientific learning method is to use them after understanding. At this time, hands-on projects are very suitable.
3. Python practical cases
Optical theory is useless. You must learn to follow along and practice it in order to apply what you have learned to practice. At this time, you can learn from some practical cases.
Four Python Comics Tutorial
Use easy-to-understand comics to teach you to learn Python, making it easier for you to remember and not boring.
5. Internet company interview questions
We must learn Python to find a high-paying job. The following interview questions are the latest interview materials from first-tier Internet companies such as Alibaba, Tencent, Byte, etc., and Alibaba bosses have given authoritative answers. After finishing this set I believe everyone can find a satisfactory job based on the interview information.
This complete version of the complete set of Python learning materials has been uploaded to CSDN. If friends need it, you can also scan the official QR code of csdn below or click on the WeChat card at the bottom of the homepage and article to get the method. [Guaranteed 100% free]
