HTTP & Python

Back in the day, if you wanted to play around with HTTP requests or share some files over a web server, you would usually have to dive into setting up something tuff like Apache or NGINX servers 😅. That could be a real pain quest, especially if you were also trying to get a dynamic web app off the ground using something like Django, Flask, or FastAPI...

But here's a little secret sauce ingredient here : Python's got this neat built-in HTTP server that can save you from all that trouble.

Picture this: you're trying to share a bunch of files with your classmates or anyone on the same WiFi. Or maybe you've downloaded some static stuff from the web for a project and need to work on it offline, or you're itching to tinker with HTTP requests directly from your terminal. Perhaps there's even a Python script you want to run without being right next to your computer.

Python's http.server module has got your back, letting you do all of this with just a simple command. It's like having a little web server Swiss Army knife right in your Python toolkit! Let's explore how to play with it 🚀

Run HTTP web server in one pyhton line

Just open a terminal and see if python3 is installed (use the command python3 --version) then you can just write this :

python3 -m http.server

You should see this output here indicating that your server is running well on your local machine :

Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

So, when you kick off this Python HTTP server command, it sets up shop on port 8000 and throws open its doors to all the network connections your computer can handle. Think of it like setting up a lemonade stand and using a loudspeaker to tell everyone you're open for business. The tech behind the scenes decides whether to use the old-school IPv4 (that's like your classic 0.0.0.0 address) or the newer IPv6 (the one with all the colons: ::). Either way, anyone hooked up to the same network can just type in your computer's IP address and bam, they're checking out whatever you're serving up on your HTTP server 😎

But wait, how do you even know your computer's IP address? Well, it's not like it's written on the back of your laptop.

Tip

Depending on whether you're using a Mac, a Windows PC, or a Linux machine, you'll have to do a bit of detective work with some commands (like ifconfig for Mac, ipconfig for Windows, or ip address/hostname -I for Linux) to find it. Or, if you're curious about your public IP address (that's the one the outside world sees, usually your router's address) you can see it on every internet site.

Just remember, this public IP isn't the same as your local one, and unless you've fiddled with your router's settings to forward ports, nobody from the internet can sneak into your HTTP server through that public IP.

Oh, and a heads up: if port 8000 is already busy hosting some other digital party, Python's going to throw the classical Address already in use error when you try to start your server. So, keep that in mind and read about Linux commands for ports management !

If you want to specify the port to expore you can write the port folowing the previous command like this : python3 -m http.server 8080

Note

Web browsers automatically use port 80 to communicate with web servers over the HTTP protocol, even when you don’t explicitly specify the port number in the address bar. For example, these two addresses are equivalent:

• http://localhost:80
• http://localhost

For security reasons, you may restrict access to your HTTP server by running it on an address belonging to the virtual loopback interface or localhost, which no one except you will be able to speak to. You can bind a specific network interface or IP address by using the -b option like this :

 python3 -m http.server -b 127.0.0.2 8080

By default, Python serves the files located in your current working directory where you executed the command to start the server. So, when you visit the home address / of your server in a web browser, then you’ll see all the files and folders in the corresponding directory. If you want to expose your server with an other directory use the -d parameter like :

python3 -m http.server -d ~/Pictures/Cats

Execute remote scripts with the Common Gateway Interface (CGI)

Alright, let's talk about running scripts from a distance using this old-school method called the Common Gateway Interface, or CGI for short. It's kind of like finding an old flip phone when everyone's using smartphones nowadays we call it be a hippster 😂. CGI used to be the go-to way to make websites interactive, letting you do cool stuff like filling out forms or logging in before all these newfangled frameworks came along. It's a bit of a dinosaur in the web world now but knowing about where things started it always cool 😎.

So, back in the day, if you were coding in Perl or Python and wanted to make your website do something fancy on the spot, CGI was your buddy. It was simple: your script just needed to know how to talk and listen through standard input and output, and understand environment variables. Web servers had a special spot called cgi-bin/ for these CGI scripts. When someone clicked a link to a CGI script, the server ran it, passed it some info like the user's request, and whatever the script spit out, the server sent back as a webpage.

The cool part was that CGI didn't care what programming language you were speaking. You could mix and match however you wanted, as long as your scripts could handle the basics. Adding or changing features was as easy as dropping a new script into the cgi-bin/ folder—no server reboot required.

But CGI had its downsides also... It was like starting your car's engine every time you wanted to drive another 10 feet—slow and a gas guzzler, because it had to start up a whole new process for each request. Plus, it could be a security nightmare if you weren't careful about cleaning up the data coming into your scripts.

Python CGI scripts

Despite its retirement, you might bump into CGI in the wild, like at a yard sale. Python's got your back here too, with its http.server module that can run CGI scripts without fuss. Just make a cgi-bin/ folder, drop in a script like hello.py that says "Hello, World!" in the most web-server-friendly way, and you're good to go. Just remember, if you're using a Mac or Linux, to start your script with #!/usr/bin/env python3 so the server knows it's a Python script and not some random text file! Let's take a look at this example below :

#!/usr/bin/env python3

print(
    """\
Content-Type: text/html

<!DOCTYPE html>
<html lang="en">
<body>
<h1>Hello, World!</h1>
</body>
</html>"""
)

Now make this file executable with the command chmod +x and start the server with the --cgi flag like this :

python3 -m http.server --cgi

Serve content with http library

The http.server module in Python is like having a Swiss Army knife for setting up web servers right from your terminal. But if you're feeling adventurous and want to tweak things to your liking, you can dive into the code and mess around with its inner gears. Essentially, this module has a couple of server classes that are all about making your server life easier. You've got:

• http.server.HTTPServer: Your basic web server that handles requests in a straightforward, one-at-a-time manner.
• http.server.ThreadingHTTPServer: The cool cousin of HTTPServer that can juggle multiple requests at once by using threads. This means it can deal with a bunch of people trying to access your server simultaneously without breaking a sweat.

When you're setting up one of these servers, you need to give it a request handler, which is like telling your server how to treat different kinds of requests it might get. Python gives you three options to start from:

• http.server.BaseHTTPRequestHandler: This is the blank canvas where you paint your server's behavior for different web requests, like GET or POST.
• http.server.SimpleHTTPRequestHandler: Use this if you want to serve up files directly from your project directory, like images, CSS, or JavaScript.
• http.server.CGIHTTPRequestHandler: Pick this for running CGI scripts, making your server more dynamic and interactive.

You're not stuck with just these, though. You can extend BaseHTTPRequestHandler to make it do pretty much whatever you want. This could be serving specific files, handling user logins with cookies, or creating custom responses based on the user's request. It's like building your own mini web framework without all the extra fluff.

For instance, you might want to keep track of users with session cookies, manage user logins, redirect users to different pages, or deal with form data from a POST request. The sky's the limit with what you can do by extending these classes. You're basically crafting your own web server behavior, using Python's http.server module as the foundation. It's a great way to learn the nuts and bolts of how web servers work and how you can manipulate them to do your things ⛏.

Let's write a simple Python example for a better illustration of the http.server module and demonstrate more clearly how to handle GET and POST requests. This includes handling of query parameters, form data, and cookies in a more structured and potentially practical way. Additionally, we'll implement example do_GET and do_POST methods to provide a clearer idea of how to use the parsed data.

from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import parse_qs, urlparse
from http.cookies import SimpleCookie
import cgi

class WebServer(BaseHTTPRequestHandler):
    @property
    def parsed_url(self):
        return urlparse(self.path)

    @property
    def query_data(self):
        return parse_qs(self.parsed_url.query)

    def read_post_data(self):
        ctype, pdict = cgi.parse_header(self.headers.get('Content-Type'))
        content_length = self.headers.get('Content-Length')
        if ctype == 'multipart/form-data':
            if content_length:
                pdict['CONTENT-LENGTH'] = int(content_length)
            # ensure the 'boundary' byte-string is correctly formatted for Python 3
            pdict['boundary'] = bytes(pdict['boundary'], "utf-8")
            post_vars = cgi.parse_multipart(self.rfile, pdict)
        elif ctype == 'application/x-www-form-urlencoded':
            length = int(self.headers.get('Content-Length', 0))
            post_vars = parse_qs(self.rfile.read(length).decode('utf-8'), keep_blank_values=True)
        else:
            post_vars = {}
        return post_vars

    def cookies(self):
        if "Cookie" in self.headers:
            return SimpleCookie(self.headers["Cookie"])
        else:
            return SimpleCookie()

    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        message = "Handling GET request bro !"
        self.wfile.write(message.encode("utf-8"))
        # Example of using query_data
        # query_string = self.query_data.get('key', [''])[0]

    def do_POST(self):
        post_data = self.read_post_data()
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        message = "Handling POST request bro !"
        self.wfile.write(message.encode("utf-8"))
        # Example of using post_data
        # post_var = post_data.get('key', [''])[0]


# you can change the server port here in this method
def run(server_class=HTTPServer, handler_class=WebServer, port=8000):
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    print(f"Starting httpd on port {port}...")
    httpd.serve_forever()

if __name__ == "__main__":
    run()

You can now call your http server with GET and POST with your favourite request client (curl, wget, postman...) and you should see this output in your teminal :

127.0.0.1 - - [28/Feb/2024 19:09:18] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [28/Feb/2024 19:09:18] "POST / HTTP/1.1" 200 -

cgi VS http.server modules

The cgi module is designed to facilitate the writing of CGI scripts. It provides functions and classes that assist in parsing and handling data received via HTTP requests, especially form data submitted through POST requests. Historically, CGI (Common Gateway Interface) was a standard way for web servers to interact with external programs, often scripts, that process user data and generate dynamic content. The cgi module is well-suited for handling multipart/form-data (used for file uploads) and application/x-www-form-urlencoded (the default content type for HTML forms) content types.

On the other hand, http.server is a module for building HTTP servers. It includes basic request handlers, such as BaseHTTPRequestHandler, which can be extended to handle different HTTP methods, including GET and POST. While it provides the infrastructure for receiving HTTP requests and sending responses, it doesn't offer specialized tools for parsing complex POST data directly.

The choice between using the cgi module and the http module (or more specifically, components like http.server) in Python for handling POST requests depends on the specific needs of your application and the nature of the data being processed.

Note

Modern Python web development often relies on more comprehensive web frameworks (like Flask, Django, or FastAPI) that provide their own mechanisms for handling POST requests and form data. These frameworks offer more features, better security, and scalability compared to building directly on top of cgi or http.server.

For simple use cases or learning purposes, directly using http.server and cgi might be sufficient. However, for production-grade applications, considering a web framework might be more appropriate.

In summary, the choice to use the cgi module for handling POST requests in a Python HTTP server context is primarily driven by the need to easily parse form data and file uploads, leveraging the module's specialized capabilities for these tasks. However, the approach to handling POST requests should be aligned with the overall architecture of your application and the requirements of your project.

Handle Cookies

You can create a new request (e.g., a POST request to http://localhost:8000) with Postman or an equivalent. Before sending the request, you'll need to include the cookie in the request headers manually. To do this, go to the Headers tab in your request pane. Add a new header: For the key, enter Cookie, and for the value, enter the cookie string exactly as it was set in the response (e.g., session=test_session23456). If you're testing with the same domain and haven't cleared cookies in Postman, it should automatically include the cookie in the request. Then send the POST request you should have the following result :

Play with cookies

To play with cookies using Postman or any similar tool, we will first need to modify our handler to explicitly set a cookie in response to a request (for example, in the do_GET method) and then attempt to read that cookie in subsequent requests (you could test this in either the do_GET or do_POST method).

Let's adjust your code to set a cookie when handling a GET request and read this cookie back when handling a POST request. This way, you can easily test cookie handling by first making a GET request to set the cookie and then a POST request to see how the server responds to the cookie it previously set.

def do_GET(self):
    # Set a cookie
    cookie = SimpleCookie()
    cookie["session"] = "test_session"
    cookie["session"]["path"] = "/"
    cookie["session"]["httponly"] = True
    self.send_response(200)
    self.send_header("Content-type", "text/html")
    # Include the cookie in the response headers
    for morsel in cookie.values():
        self.send_header("Set-Cookie", morsel.OutputString())
    self.end_headers()
    message = "Cookie set!"
    self.wfile.write(message.encode("utf-8"))

and for the POST request here :

def do_POST(self):
    # Attempt to read the cookie
    cookies = self.cookies()
    session_cookie = cookies["session"].value if "session" in cookies else "No cookie"

    # Prepare the response
    self.send_response(200)
    self.send_header("Content-type", "text/html")
    self.end_headers()
    message = f"Handling POST request! Found cookie: {session_cookie}"
    self.wfile.write(message.encode("utf-8"))

Now you can open Postman and create a new GET request to your server (e.g., http://localhost:8000). Send the request. You should receive a response indicating that a cookie has been set 😎

Using ThreadingHTTPServer

Earlier we talked about ThreadingHTTPServer from Python's http.server module (the cool cousin lol). Let's write a mini example in order to demonstrates how to set up a simple web server that can handle multiple requests concurrently by utilizing threads. Each incoming request is handled in a separate thread, allowing the server to process multiple requests simultaneously without blocking.

The example includes a custom request handler class that responds to GET requests by sending back a simple HTML page that displays a greeting message along with the path requested.

from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler
import threading

class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.end_headers()
        message = f"Hello, World! You requested {self.path}\n"
        self.wfile.write(message.encode('utf-8'))

def run(server_class=ThreadingHTTPServer, handler_class=SimpleHTTPRequestHandler, port=8000):
    server_address = ('', port)
    httpd = server_class(server_address, handler_class)
    print(f"Starting httpd on port {port}...")
    print(f"Running in thread: {threading.current_thread().name}")
    httpd.serve_forever()

if __name__ == '__main__':
    run()

Now you know how to build basics HTTP servers with python, test it with postman and play with cookies, hope you learn a thing or two, congrats 🥳

HTTPS Encryption

When you try to view an HTML file by just opening it from your computer with a web browser, it might not always display correctly. This happens because browsers are designed to request and fetch files from a server, and they struggle to do this properly without one. Essentially, your browser wants to chat with a server to get the files it needs to show you the page.

Up until now, any chat you've had with your HTTP server has been out in the open, without encryption. That's totally okay if you're just messing around or getting a feel for how HTTP conversations go, seeing the back-and-forth of requests and responses in their birthday suits (aka plain text). However, for the sake of security, web browsers put their foot down and limit access to certain features, like the Geolocation API, if your page isn't coming from a safe, secure connection.

If you're curious to dig deeper into how this all works, you can try your hand at making HTTP requests the old-school way. Grab a command-line tool like netcat or a Telnet client (PuTTY if you're on Windows), and connect to an active HTTP server. From there, you can manually type out and send HTTP requests, getting a front-row seat to the action.

Unfortunately, the http.server module built into Python doesn’t support the HTTPS protocol out of the box. To run your web server securely, you’ll have to extend it by wrapping the underlying TCP socket with a TLS/SSL layer like this :

# my super secure file : server.py

from http.server import HTTPServer, SimpleHTTPRequestHandler
from ssl import PROTOCOL_TLS_SERVER, SSLContext

ssl_context = SSLContext(PROTOCOL_TLS_SERVER)
ssl_context.load_cert_chain("/path/to/cert.pem", "/path/to/private.key")
# change the host address if you want
server = HTTPServer(("0.0.0.0", 443), SimpleHTTPRequestHandler)
server.socket = ssl_context.wrap_socket(server.socket, server_side=True)
server.serve_forever()

Self-Signed Certificate Using Python

If you’re on macOS or Linux, then you can create and sign a new certificate using a command-line client for the OpenSSL library like this :

$ openssl req -x509 \
              -nodes \
              -days 365 \
              -newkey rsa:2048\
              -out cert.pem \
              -keyout private.key

Afterward, you’ll have two PEM-encoded files required by the Python script mentioned earlier. The first file is a certificate (-x509) using a 2,048-bit RSA encryption algorithm (-newkey rsa:2048), which will remain valid for one year (-days 365). The other file is the corresponding private key stored in an unencrypted form (-nodes) that you should keep secret.

While a self-signed certificate will encrypt your data, web browsers won’t trust it due to the unknown certificate authority that signed it—namely, you 😅

Each browser comes with a list of well-known and trusted certificate authorities. You can also install additional certificates on your operating system to make the browser accept such a connection as secure. More details with PyOpenSSL

That's it for this article, hope your enjoy now are a HTTP python 🥷🏼

More ressources about HTTPS

So you want to become a real HTTPS shinobi 🥷🏼

• What Is HTTPS & How Does It Work? [Explained]
• The 6-Step "Happy Path" to HTTPS
• HTTPS in the real world
• HTTPS on Stack Overflow: The End of a Long Road
• Letsencrypt under the hood : how it works
• Performing & Preventing SSL Stripping: A Plain-English Primer
• Consider Security and Performance Limitations
• An Overview of TLS 1.3 – Faster and More Secure

Wrap it up

Let's summarisa what we have learn so far

• How HTTP servers operate in a synchron way and with threading.
• Understanding the request-response cycle, status codes, and headers.
• Some security mechanisms like HTTPS and cookies.

Now I think you have a solid understanding of HTTP servers, hope you enjoyed it 🥳