Practitioner Level

ServerCheck: Making Async HTTP Requests with Python

Pricing
1 hour
  • 4 Learning Objectives

About this Hands-on Lab

We frequently have to check whether one of our servers has access to other servers on our internal network. To make this a little easier for ourselves, we’ve decided to use Python to write a CLI that can take either a JSON file with servers and ports to check or a list of host/port combinations to make requests to. In this hands-on lab, we will take a list of server/port combinations and make HTTP requests concurrently so that we can get the status of our servers as quickly as possible. We’ll also finalize the `servercheck` CLI.

Learning Objectives

Successfully complete this lab by achieving the following learning objectives:

Create the http Module within the servercheck Package and Install requests

To begin, we need to change to the servercheck directory and create and activate our virtualenv.

$ cd servercheck
$ pipenv install --python python3.7
...
$ pipenv shell
(servercheck) $

Next, we need to create the module to work in. We’ll call it http.py. To be a module within a package, we’ll need to place this file within the servercheck directory that contains an __init__.py.

(servercheck) $ touch servercheck/http.py

Now that we have a module, we need to install the requests package as a dependency of our tool:

(servercheck) $ pipenv install requests

We’ll also want to add this to the REQUIRED list in our setup.py:

setup.py (partial)

# What packages are required for this module to be executed?
REQUIRED = ["click", "requests"]
Make Concurrent Requests and Return the Results

From within our new http module, we will create one function to act as the public interface that we’ll eventually call from our CLI. Let’s call this function ping_servers. It will receive the argument servers:

servercheck/http.py

def ping_servers(servers):
    results = {'success': [], 'failure': []}
    asyncio.run(make_requests(servers, results))
    return results

The main thing we need to do here is create our results dictionary and trigger our unwritten make_requests coroutine that will handle creating and running the HTTP request tasks.

Next, we’ll create the make_requests, ping, and get functions:

servercheck/http.py

import asyncio
import requests
import os

def get(server):
    debug = os.getenv("DEBUG")
    try:
        if debug:
            print(f"Making request to {server}")
        response = requests.get(f"http://{server}")
        if debug:
            print(f"Received response from {server}")
        return {"status_code": response.status_code, "server": server}
    except:
        if debug:
            print(f"Failed to connect to {server}")
        return {"status_code": -1, "server": server}

async def ping(server, results):
    loop = asyncio.get_event_loop()
    future_result = loop.run_in_executor(None, get, server)
    result = await future_result
    if result["status_code"] in range(200, 299):
        results["success"].append(server)
    else:
        results["failure"].append(server)

async def make_requests(servers, results):
    tasks = []

    for server in servers:
        task = asyncio.create_task(ping(server, results))
        tasks.append(task)

    await asyncio.gather(*tasks)

def ping_servers(servers):
    results = {"success": [], "failure": []}
    asyncio.run(make_requests(servers, results))
    return results

Since we want all of the HTTP requests to be made as soon as possible, we’ll schedule each in its own task and utilize asyncio.gather to run all of the tasks and wait until they are all finished.

The requests library isn’t designed to work with asyncio, so we need to manually run our HTTP requests on the event loop to make sure that we can avoid waiting on the network IO.

Test Against Additional Servers Using REPL

Now that we have all of our functions, let’s test them out. Our hands-on lab environment has two other servers that have a few web servers listening on various ports. We can test our code by loading our package into the REPL and interacting with these servers:

(servercheck) $ DEBUG=true PYTHONPATH=. python
>>> from servercheck.http import ping_servers
>>> servers = ('web-node1:80', 'web-node2:80', 'web-node1:3000', 'web-node2:3000', 'web-node1:8080')
>>> ping_servers(servers)
Making request to web-node1:80
Making request to web-node2:80
Making request to web-node1:3000
Making request to web-node2:3000
Making request to web-node1:8080
Failed to connect to web-node1:3000
Received response from web-node1:80
Received response from web-node2:80
Received response from web-node2:3000
Received response from web-node1:8080
{'success': ['web-node1:80', 'web-node2:80', 'web-node2:3000', 'web-node1:8080'], 'failure': ['web-node1:3000']}
>>>

We can see from our debug output that all of the requests are started rapidly and the responses come back in the order that they finish. Without the DEBUG environment variable set, we’d only see the results dictionary.

Utilize servercheck.http.ping_servers in the CLI Function

Now that we have the ping_servers function, we can use it in the cli function to make our tool work the way that we expect it to. The cli function already collects the server information that the user passes in; we’ll pass that information into the ping_servers function.

servercheck/cli.py

import click
import json
import sys
from .http import ping_servers

@click.command()
@click.option("--filename", "-f", default=None)
@click.option("--server", "-s", default=None, multiple=True)
def cli(filename, server):
    if not filename and not server:
        raise click.UsageError("must provide a JSON file or servers")

    # Create a set to prevent duplicate server/port combinations
    servers = set()

    # If --filename or -f option is used then attempt to read
    # the file and add all values to the `servers` set.
    if filename:
        try:
            with open(filename) as f:
                json_servers = json.load(f)
                for s in json_servers:
                    servers.add(s)
        except:
            print("Error: Unable to open or read JSON file")
            sys.exit(1)

    # If --server or -s option are used then add those values
    # to the set.
    if server:
        for s in server:
            servers.add(s)

    # Make requests and collect results
    results = ping_servers(servers)

Now we can print the results to match our design goal:

servercheck/cli.py

import click
import json
import sys
from .http import ping_servers

@click.command()
@click.option("--filename", "-f", default=None)
@click.option("--server", "-s", default=None, multiple=True)
def cli(filename, server):
    if not filename and not server:
        raise click.UsageError("must provide a JSON file or servers")

    # Create a set to prevent duplicate server/port combinations
    servers = set()

    # If --filename or -f option is used then attempt to read
    # the file and add all values to the `servers` set.
    if filename:
        try:
            with open(filename) as f:
                json_servers = json.load(f)
                for s in json_servers:
                    servers.add(s)
        except:
            print("Error: Unable to open or read JSON file")
            sys.exit(1)

    # If --server or -s option are used then add those values
    # to the set.
    if server:
        for s in server:
            servers.add(s)

    # Make requests and collect results
    results = ping_servers(servers)

    print("Successful Connections")
    print("---------------------")
    for server in results['success']:
        print(server)

    print("n Failed Connections")
    print("------------------")
    for server in results['failure']:
        print(server)

Let’s make sure that our package is installed into our virtualenv in an editable way before testing the final product:

(servercheck) $ pip install -e .
...

Next we’ll create a test JSON file called example.json within our project’s directory:

[
    "web-node1:80",
    "web-node1:8000",
    "web-node1:3000",
    "web-node2:80",
    "web-node2:3000"
]

To test our tool, we can use the example.json and also pass some more information -s options to the servercheck executable.

(servercheck) $ servercheck -f example.json -s 'web-node1:80' -s 'web-node1:9000'
Successful Connections
----------------------
web-node1:80
web-node2:80
web-node2:3000
web-node1:8000

Failed Connections
------------------
web-node1:3000
web-node1:9000
Contact SalesSee Pricing

Additional Resources

We frequently have to check whether one of our servers has access to other servers on our internal network. To make this a little easier for ourselves, we've decided to use Python to write a CLI that can either take a JSON file with servers and ports to check or a list of host/port combinations to make requests to. The team has already created the CLI to collect information from the user. Now we're ready to use the server/port combinations to ensure that we receive successful HTTP responses to each request.

In this hands-on lab, we're going to write the code that can take a set of server/port combinations and concurrently make requests for each one. The server/port combinations will not contain the scheme, and we will utilize http. We want to return a dictionary that contains our results divided up into successful and failed responses. When we're done, it will look like this:

{
    'success': [
        'IP1:PORT1',
        'IP1:PORT2'
    ],
    'failure': {
        'IP1:PORT3',
        'IP2:PORT1'
    }
}

We'll be working in a project at ~/servercheck. Its virtualenv can be activated using pipenv shell from within the project.

If you'd like to follow along on your own development machine, you can clone this github project. To make test HTTP connections, you'll still be able to utilize the web nodes using their public IP addresses.

To feel comfortable completing this lab, you'll need to know how to do the following:

What are Hands-on Labs

Hands-on Labs are real environments created by industry experts to help you learn. These environments help you gain knowledge and experience, practice without compromising your system, test without risk, destroy without fear, and let you learn from your mistakes. Hands-on Labs: practice your skills before delivering in the real world.

Sign In
Welcome Back!

Psst…this one if you’ve been moved to ACG!

Don't have an account?

Get Started
Who’s going to be learning?
MyselfMy Organization