🥳Requests

Requests is a popular third-party library for making HTTP requests in Python. It provides a simple and intuitive interface for sending HTTP requests and handling responses, making it easy to interact with APIs and web services.

With Requests, you can send GET, POST, PUT, DELETE, and other types of HTTP requests, set headers and cookies, handle authentication, and parse JSON responses. It also supports features such as SSL verification, timeouts, and connection pooling, making it a powerful tool for building robust and scalable applications.

Requests is widely used in the Python community and is supported by a large and active community of developers. It is compatible with Python 2.7 and 3.x, and can be installed using pip or another package manager.

Whether you're working with REST APIs, web scraping, or building web applications, Requests is a valuable tool for simplifying and streamlining your HTTP requests. In the following sections, we'll explore the key features and usage patterns of the Requests library.

Making requests

An overview of making requests with the Requests library:

a. Sending GET requests

To send a GET request with Requests, you can use the requests.get() method and pass in the URL of the resource you want to retrieve. For example:

import requests

response = requests.get('https://api.example.com/data')
print(response.text)

This sends a GET request to the https://api.example.com/data URL and prints the response content.

b. Sending POST requests

To send a POST request with Requests, you can use the requests.post() method and pass in the URL of the resource you want to post to, along with any data you want to send in the request body. For example:

import requests

data = {'key1': 'value1', 'key2': 'value2'}
response = requests.post('https://api.example.com/submit', data=data)
print(response.text)

This sends a POST request to the https://api.example.com/submit URL with the data dictionary in the request body, and prints the response content.

c. Sending other types of requests

Requests supports other HTTP methods such as PUT, DELETE, PATCH, and HEAD, which can be accessed using the corresponding methods (requests.put(), requests.delete(), etc.). For example:

import requests

response = requests.put('https://api.example.com/update', data={'key': 'new_value'})
print(response.text)

This sends a PUT request to the https://api.example.com/update URL with the data dictionary in the request body, and prints the response content.

d. Adding headers and authentication

To add headers or authentication to a request, you can pass in additional arguments to the requests.get() or requests.post() methods. For example, to add a custom header:

import requests

headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get('https://api.example.com/data', headers=headers)
print(response.text)

This sends a GET request to the https://api.example.com/data URL with the User-Agent header set to Mozilla/5.0.

To add basic authentication, you can pass in the auth argument with a tuple of the form (username, password):

import requests

auth = ('username', 'password')
response = requests.get('https://api.example.com/data', auth=auth)
print(response.text)

This sends a GET request to the https://api.example.com/data URL with basic authentication using the username and password values.

e. Handling errors and exceptions

Requests can raise exceptions for various types of errors, such as timeouts, connection errors, and invalid URLs. To handle these errors, you can wrap your request code in a try/except block and catch the appropriate exceptions. For example:

import requests

try:
    response = requests.get('https://api.example.com/data')
    response.raise_for_status() # Raise an exception if the response status code is not 200
    print(response.text)
except requests.exceptions.HTTPError as error:
    print(f'Request failed with HTTP error: {error}')
except requests.exceptions.RequestException as error:
    print(f'Request failed: {error}')

This sends a GET request to the https://api.example.com/data URL, checks that the response status code is 200, and prints the response content. If an HTTP error or any other type of error occurs, the appropriate exception is caught and a message is printed.

Request parameters and data

a. Passing query parameters

To pass query parameters in a request URL, you can use the params parameter in the requests.get() or requests.post() methods. You can pass a dictionary of key-value pairs representing the query parameters. For example:

import requests

params = {'key1': 'value1', 'key2': 'value2'}
response = requests.get('https://api.example.com/data', params=params)
print(response.text)

This sends a GET request to the https://api.example.com/data URL with the query parameters key1=value1 and key2=value2.

b. Submitting form data

To submit form data in a request, you can use the data parameter in the requests.post() method. You can pass a dictionary of key-value pairs representing the form fields. For example:

import requests

data = {'username': 'john', 'password': 'secret'}
response = requests.post('https://api.example.com/login', data=data)
print(response.text)

This sends a POST request to the https://api.example.com/login URL with the form fields username and password containing the respective values.

c. Uploading files

To upload files in a request, you can use the files parameter in the requests.post() method. You can pass a dictionary of key-value pairs where the key is the name of the file field, and the value is a file-like object or a tuple specifying the filename, content, and content type. For example:

import requests

files = {'file': open('document.pdf', 'rb')}
response = requests.post('https://api.example.com/upload', files=files)
print(response.text)

This sends a POST request to the https://api.example.com/upload URL with the file field containing the document.pdf file.

d. Setting request timeouts

To set a timeout for a request, you can use the timeout parameter in the requests.get() or requests.post() methods. You can specify the timeout value in seconds as a float or an integer. For example:

import requests

response = requests.get('https://api.example.com/data', timeout=5)
print(response.text)

This sends a GET request to the https://api.example.com/data URL with a timeout of 5 seconds. If the request takes longer than 5 seconds to complete, a Timeout exception will be raised.

Setting a timeout is useful to ensure that your program doesn't hang indefinitely if a request takes too long to respond.

These are some of the common ways to work with request parameters and data in the Requests library. Depending on your specific needs, you may need to explore other features and options provided by the library.

Response content

a. Accessing response headers and status codes

To access the headers and status code of a response, you can use the headers and status_code attributes of the response object, respectively. For example:

import requests

response = requests.get('https://api.example.com/data')
print(response.status_code) # Prints the HTTP status code (e.g. 200, 404)
print(response.headers) # Prints the response headers (as a dictionary)

This sends a GET request to the https://api.example.com/data URL and prints the status code and headers of the response.

b. Reading response content

To read the content of a response, you can use the text or content attributes of the response object. The text attribute returns the response content as a string, while the content attribute returns the response content as bytes. For example:

import requests

response = requests.get('https://api.example.com/data')
print(response.text) # Prints the response content as a string
print(response.content) # Prints the response content as bytes

This sends a GET request to the https://api.example.com/data URL and prints the content of the response.

c. Parsing JSON responses

If the response content is in JSON format, you can use the json() method of the response object to parse the JSON data into a Python object (typically a dictionary or a list). For example:

import requests

response = requests.get('https://api.example.com/data.json')
data = response.json() # Parses the response content as JSON and returns a Python object
print(data)

This sends a GET request to the https://api.example.com/data.json URL, parses the response content as JSON, and prints the resulting Python object.

d. Handling redirects

By default, Requests automatically follows redirects up to a maximum of 30 redirects. If you need to disable or customize the handling of redirects, you can use the allow_redirects parameter in the requests.get() or requests.post() methods. For example:

import requests

response = requests.get('https://api.example.com/data', allow_redirects=False)
print(response.status_code) # Prints the HTTP status code of the response (e.g. 301, 302)
print(response.headers['Location']) # Prints the URL of the redirect (if any)

This sends a GET request to the https://api.example.com/data URL without following redirects. The status_code attribute of the response indicates whether the response was a redirect, and the Location header (if present) contains the URL of the redirect.

These are some of the common ways to work with response content in the Requests library. Depending on your specific needs, you may need to explore other features and options provided by the library.

Session management

Creating sessions:

To create a session in Requests, you can use the requests.Session() class. A session allows you to persist certain parameters, such as cookies and headers, across multiple requests. For example:

pythonCopy codeimport requests

session = requests.Session()
response = session.get('https://api.example.com/login')
# Perform additional requests using the session

This creates a session object using requests.Session(), and you can then use this session to send multiple requests while maintaining the same session context.

Persisting cookies between requests:

By default, a session in Requests automatically persists cookies across requests. This means that if a response sets a cookie, the session will automatically include that cookie in subsequent requests. For example:

pythonCopy codeimport requests

session = requests.Session()
response = session.get('https://api.example.com/login')
# Cookie is automatically stored in the session

# Perform subsequent requests
response = session.get('https://api.example.com/data')

In this example, the session automatically persists any cookies set by the server in the first request and includes them in the subsequent request.

Setting session headers and authentication:

You can set headers and authentication parameters for a session using the session object's attributes. For example, to set a custom header for all requests made with the session:

import requests

session = requests.Session()
session.headers['User-Agent'] = 'Mozilla/5.0'
response = session.get('https://api.example.com/data')

This sets the 'User-Agent' header to 'Mozilla/5.0' for all requests made with the session.

Similarly, you can set authentication parameters for the session using the auth attribute. For example, to use basic authentication:

import requests

session = requests.Session()
session.auth = ('username', 'password')
response = session.get('https://api.example.com/data')

This sets the username and password for basic authentication for all requests made with the session.

By using sessions, you can persist cookies, headers, and authentication parameters across multiple requests, which is useful when interacting with APIs or web services that require session-based interactions.

Remember to properly handle exceptions and errors when working with sessions and making requests using the session object.

These are some of the common techniques for session management in the Requests library. Depending on your specific needs, you may need to explore additional features and options provided by the library.

Advanced usage

a. Customizing SSL certificates:

If you need to customize the SSL certificate verification for HTTPS requests, you can pass in the verify parameter to the requests.get() or requests.post() methods. You can pass in a path to a CA certificate file, or set it to False to disable certificate verification.

By default, Requests verifies SSL certificates for HTTPS requests, which means that it ensures that the server's certificate is valid and trusted. However, in some cases, you may need to customize the certificate verification process, such as when working with self-signed certificates or custom CA certificates.

To customize the certificate verification, you can pass the verify parameter to the request method. The verify parameter can be set to True, False, or a path to a CA certificate file.

If verify is set to True (the default value), Requests will verify the SSL certificate using the system's trusted CA certificates.

If verify is set to False, Requests will skip the SSL certificate verification and allow insecure HTTPS connections. This is not recommended, as it can expose your application to security risks.

If verify is set to a path to a CA certificate file (in PEM format), Requests will verify the SSL certificate using the specified CA certificate. For example:

import requests

response = requests.get('https://api.example.com/data', verify='/path/to/cert.pem')
print(response.text)

This sends a GET request to the https://api.example.com/data URL and uses the /path/to/cert.pem file as the CA certificate.

You can also use a directory containing multiple CA certificates instead of a single file. In this case, the directory should contain one or more PEM-encoded files with the .pem extension.

Customizing SSL certificates can be useful in situations where you need to work with self-signed certificates or custom CA certificates. However, you should be careful when disabling SSL certificate verification or using custom CA certificates, as this can expose your application to security risks.

b. Streaming large responses:

If you're working with large responses (e.g. streaming media or log files), you can use the stream parameter to avoid loading the entire response content into memory at once. You can then iterate over the response content in chunks using the iter_content() method.

By default, when you make a request using Requests, the entire response content is downloaded into memory before being returned to your program. For large responses, this can cause performance issues or even cause your program to run out of memory.

To avoid loading the entire response content into memory at once, you can set the stream parameter to True when making the request. This tells Requests to download the response content in chunks instead of all at once. You can then use the iter_content() method to iterate over the content in chunks. For example:

import requests

response = requests.get('https://api.example.com/large_data', stream=True)
for chunk in response.iter_content(chunk_size=1024):
    # Process the chunk of data

This sends a GET request to the https://api.example.com/large_data URL and iterates over the response content in chunks of 1024 bytes.

The iter_content() method returns a generator that yields chunks of the response content as bytes. You can then process each chunk of data as needed, without loading the entire response content into memory at once.

Using the stream parameter and the iter_content() method is particularly useful when working with large responses, such as streaming media or log files. It can help reduce memory usage and improve the performance of your program.

c. Sending requests in parallel:

If you need to send multiple requests in parallel, you can use the concurrent.futures module in Python to create a pool of worker threads or processes. You can then use the Session() class to create a session object for each worker and perform the requests concurrently.

Here's an example of sending requests in parallel using the concurrent.futures module and the Requests library:

import requests
import concurrent.futures

# Define a function to perform a request
def send_request(url):
    session = requests.Session()
    response = session.get(url)
    # Process the response as needed
    return response.text

# List of URLs to send requests to
urls = [
    'https://api.example.com/data1',
    'https://api.example.com/data2',
    'https://api.example.com/data3'
]

# Create a ThreadPoolExecutor to manage the worker threads
with concurrent.futures.ThreadPoolExecutor() as executor:
    # Submit the requests concurrently
    futures = [executor.submit(send_request, url) for url in urls]

    # Process the responses as they become available
    for future in concurrent.futures.as_completed(futures):
        try:
            response_text = future.result()
            # Process the response text
            print(response_text)
        except Exception as e:
            # Handle exceptions raised during the request
            print(f"An error occurred: {e}")

In this example, the send_request() function is defined to perform a request using a session object created within the function. The function takes a URL as a parameter and returns the response text.

The list urls contains the URLs to which the requests will be sent.

A ThreadPoolExecutor is created to manage the worker threads. Each URL in the urls list is submitted to the executor using the executor.submit() method, which schedules the execution of the send_request() function for each URL. This allows the requests to be sent concurrently.

The concurrent.futures.as_completed() function is used to iterate over the completed futures as they become available. It returns an iterator that yields futures in the order in which they complete. The result of each future is obtained using the future.result() method.

You can process the responses as they become available and perform any necessary operations on the response data.

Sending requests in parallel can help improve the efficiency and performance of your program, especially when dealing with a large number of requests or when the requests are time-consuming.

d. Using proxies:

If you need to send requests through a proxy server, you can use the proxies parameter in the requests.get() or requests.post() methods. You can pass in a dictionary with the proxy type (e.g. http, https) as the key and the proxy URL as the value.

Here's an example of using a proxy server with the Requests library:

import requests

# Define the proxy server URL
proxy = {
    'http': 'http://proxy.example.com:8080',
    'https': 'https://proxy.example.com:8080'
}

# Send a request through the proxy server
response = requests.get('https://api.example.com/data', proxies=proxy)
print(response.text)

In this example, the proxy dictionary defines the URL of the proxy server to be used. The http and https keys specify the proxy types for HTTP and HTTPS requests, respectively.

To send a request through the proxy server, the proxies parameter is set to the proxy dictionary. This tells Requests to use the specified proxy server for the request.

e. Working with authentication schemes other than basic and digest:

Requests supports several authentication schemes other than basic and digest, such as OAuth and Kerberos. To use these schemes, you can install the appropriate authentication library (e.g. requests-oauthlib for OAuth) and use the corresponding authentication class in Requests.

Here's an example of using OAuth authentication with the Requests library and the requests-oauthlib library:

import requests
from requests_oauthlib import OAuth1

# Define the OAuth credentials
client_key = 'your_client_key'
client_secret = 'your_client_secret'
access_token = 'your_access_token'
access_token_secret = 'your_access_token_secret'

# Create an OAuth1 authentication object
oauth = OAuth1(client_key, client_secret, access_token, access_token_secret)

# Send a request with OAuth authentication
response = requests.get('https://api.example.com/data', auth=oauth)
print(response.text)

In this example, the requests_oauthlib library is used to handle the OAuth authentication. The OAuth1 class is used to create an OAuth1 authentication object, which takes the OAuth credentials as parameters.

The auth parameter is set to the oauth object when making the request. This tells Requests to use OAuth authentication for the request.

To use other authentication schemes, you would need to install the appropriate authentication library and use the corresponding authentication class in Requests.

When working with authentication schemes other than basic and digest, make sure to carefully read the documentation for the authentication library you're using and follow any security best practices recommended by the library.