Python: Requests Library [Beginner]

8 min readApr 16, 2023

Requests is a python library used to make HTTP requests. It is a simple and elegant library that allows you to send HTTP requests to websites and APIs, as well as to handle the responses you receive from those requests.

Through this article, you will see some of the useful features of the python requests library. Typically, you will learn how to:

Make requests using the different HTTP methods.
Send data using a GET or POST request.
Use HTTP headers to customize your requests.
Download files.

NOTE📝: To have a good understanding of the concepts that we are going to see, you must have at least a basic knowledge of how HTTP works. If that’s not the case, i recommend you to read this article.

Installation

Knowing that, requests is not a python built-in library, we need first to install it.

This can easily be done, using the pip install requests command:

As you can notice, i have already installed the requests package on my system.

After successfully installing it, you can check the library’s information using the pip show requests command:

In this example, you can see that i’m using the version 2.28.2 of the python requests library. Keep in mind that, this can vary.

Great! Now let’s delve into this library features.

Requests: HTTP Methods

HTTP supports several methods used for different purposes.

In this section, we will mainly focus on the GET and POST requests. If you want to learn more about the other methods, you can check this article.

The GET method is used to request data from the server while the POST method is used to send data to a server to create/update a resource.

GET Method

To send a request using the GET method, you can proceed as follows:

Sending a Get request using the requests library

NOTE📝: In the figure above, the upper part (before the red line) represents the Python code snippet while the bottom part represents the results got after the execution of the code.

Let’s now break down our code snippet:

import requests: import the requests library.
url : variable that stores the URL we want to access.
requests.get(url): method used to send a GET request to the specified URL as an argument.
response : store the response message returned by the server.
response.request.method: return the http method used by the client.
response.status_code: display the status code returned by the server.
response.headers.get('content-type'): return the content type returned by the server (json in our example).
response.json(): display the content of the server’s response in json format.

POST Method

To send a request using the POST method, you can proceed as follows:

Sending a POST request using the requests library

Let’s break down this code snippet:

requests.post(url) : send a POST request to the specified URL as an argument.

Great! Let’s now explore how to send data to a server using a GET or POST method.

Requests: Sending Data

Sending data to a server using a GET or a POST request can be useful, but typically for different purposes.

In fact, a GET request is typically used to access a specific resource or retrieve information from the server, while a POST request is used to create or update a resource on the server.

Let’s delve into this by starting with the GET method.

Sending data: Get method

When you send a GET request, the data (e.g. query parameters) is usually included in the URL. This is how GET method works when sending data.

Probably, you are wondering what i mean by query parameters.

Well, to better understand that, let’s take a quick look to a URL structure.

A URL (Uniform Resource Locator) is an address used to locate resources on a server, and it typically consists of various components, such as the protocol, domain, port number, path, query, and fragment.

Here, we will focus on the query parameter. For more information, you can check out this article.

The query parameter is an optional component of a URL that allows parameters to be passed to the server. It is preceded by a query separator (?) and typically consists of key-value pairs separated by ampersands (&).

For instance, when you search for a specific topic using medium’s search box, your browser makes a GET request to medium’s web server by passing you search term (here python) to the query parameter q (?q=python) in the URL as you can see on the image below:

Sending data using a browser with the GET method

To get more information about what is really going on, you can use the Developer Tools’s Network Inspection tab:

Developer Tools — Network Inspection Tab

Well, now that we have a better understanding of how a GET method sends data, let’s close the browser and use Python requests library instead.

For that, we are going to execute the following instructions:

Sending data using requests’s GET method

Let’s break down the code snippet above:

params = {“q": “python"} : dictionary representing the data we want to send to the server. Here q represents the query parameter’s key while python represents the query parameter’s value. When using several query string parameters, you only need to separate them by a comma (for instance params = {“q": “python", "q2": "c"} )
params=params : params is a requests.get()‘s optional parameter that takes as argument, the data we want to sent to the server.
response.url : return the requested URL. Here we can notice that, the query parameter was added to the origin URL.
response.text : display the server’s response message.

Below, you will find some advantages and disadvantages of using a GET method:

When using a Get request, you can easily add your favorite URLs to your bookmarks. However an issue that you may face when using this method is the length restriction. Indeed, a URL has a limited length of 2048 characters which means that you won’t be able to go beyond that.
Furthermore, as you may notice, the data are part of the URL when using a GET method. While sending data like simple search terms is not risky in that case; sending sensitive information such as username and password, can however be hazardous, simply because whether or not you are using the HTTPS protocol, the URL is not encrypted and can be intercepted and read by an attacker performing a Man In The Middle Attack for instance. For that reason, it is preferable to use the POST method which sends the data in the request body instead of the URL directly.

That’s what we are going to cover in the following section.

Sending Data: POST Method

The POST method is a good choice when it comes to send sensitive data or data of any length.

To send a request using the POST method, you can proceed as follows:

Sending data using requests’s POST method

payload : dictionary representing the data that is sent to the server.
requests.post(url, data=payload) : send a request to the specified url with the given payload. The data in the requests.post() method is an optional parameter. If you omit it, the request will be sent without any request body data.
As you can see in its json response, the server successfully received our data ('form': {'password': 'iloveyou', 'username': 'foobar'} ).

NOTES📝:

As you may notice, the data are not added to the URL this time. This totally makes sense, knowing that the POST method sends data using the request body rather than the URL like the GET method does.
Moreover you can send as much as data you want, see that there is no data length restriction. However, when using POST method, we cannot add a specific URL to our bookmarks see that the data are not part of it.

Excellent! Let’s learn how to manipulate HTTP headers in order to customize our requests.

Requests: Headers

HTTP headers are used to add metadata to a request or response.

We have two types of HTTP headers: HTTP request headers and HTTP response headers.

To learn more about these headers, you can check out this article.

In this section, we will focus on HTTP request headers see that we have no control over the server headers unless we are the server administrator which is not the case.

Well, let’s get started.

Sending a GET request with a specific header

Let’s break down the code snippet above:

headers : dictionary whose keys and values are respectively the HTTP headers and HTTP headers’ values.
headers=headers : headers is a requests.get() ‘s optional parameter that takes a dictionary of headers as an argument.
response.json() : return the client’s HTTP headers that the server received.

NOTE📝: It is absolutely possible to define your own header (in this example, i used a header called ‘Personal-Header’ which is not a standardized header). However, it’s entirely up to the server to treat your header or not.

Super!! It’s time to cover the last section which covers file downloading.

Requests: File Downloading

The requests library is a powerful and versatile library. In addition to to send and receive data, it also allows you to download files in a simple way.

To do that, let’s take a look at the code snippet below:

Downloading an image using requests’s GET method

Accept : HTTP header used to specify the content MIME that the client is willing to accept (here, the client tells the server that it will only accept png format).
open("file", "wb") : python built-in function used to open a file. "wb” is an argument (file mode) used to open a file in “binary write” mode.
os.path.exists() : method used to check if a given path exists.
os.path.abspath() : return the absolute path of a file.
f.name : represent the name of the file we opened.

NOTE📝: The same process can be applied to download any file format(zip, mp3, mp4, pdf, and so on). Nevertheless, you need to first get the URL of the file to download.

By the way, here is the downloaded image 😄 :