HTTP servers and an overview of the CGI interface

After finishing the Rust book and its final project (a very simple HTTP server), I decided to expand it to deal with more scenarios, and also add in some dynamic behavior with CGI for fun. This was a good opportunity to learn more about HTTP and CGI and inspired me to write about it.

An introduction to the HTTP protocol

#

The HTTP protocol is used to access resources and transmit data over the web. When you access a web page in a browser, it sends an HTTP request to a computer running the server. Consider the following request:

GET / HTTP/1.1
Host: example.org

The first line contains an HTTP verb, in this case GET. There are different HTTP verbs (or methods) used to perform different actions with a given resource. The GET method conventionally indicates that we want to read the resource. The following \ represents a path to the resource we want to access (in this case, the root path). And finally, HTTP/1.1 represents which version of HTTP we are using. Following this first line, a list of headers is supplied. These headers give the server some metadata about our request. In our case, we just supplied the Host header, which tells the server program what is the domain to which we are sending our request.

If you have the netcat command available, you can actually interact with an HTTP server and directly send a request. Run netcat example.org 80, which opens up a TCP connection to example.org at port 80 (the default port for HTTP). Then you can copy and paste the request above and press enter twice. The server will respond with something like this:

HTTP/1.1 200 OK
Accept-Ranges: bytes
*a bunch of other HTTP headers*
...
Content-Length: 1256

*the actual HTML for the page*

At this point you can close the connection to the server by killing netcat with CTRL+C.

The response contains an initial line that is somewhat similar to the request’s, indicating the HTTP version being used, and then an HTTP status code, in this case 200 OK indicating that the request could be fulfilled successfully. Following that we have some headers for the response, a blank line, and then the actual data of interest (the body).

Speaking of it, the HTTP request can also have a body, for instance:

POST /anything HTTP/1.1
Host: httpbin.org
Content-Length: 10

1234567890

The above request accesses the /anything path on httpbin.org, a utility for inspecting and testing HTTP requests. This specific path responds with data about the request sent to it. If you run netcat httpbin.org 80 and send this request you will get a response similar to this:

HTTP/1.1 200 OK
Date: Thu, 28 Dec 2023 05:28:02 GMT
Content-Type: application/json
Content-Length: 332
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {}, 
  "data": "1234567890", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Content-Length": "10", 
    "Host": "httpbin.org", 
    "X-Amzn-Trace-Id": "..."
  }, 
  "json": 1234567890, 
  "method": "POST", 
  "origin": "XXX.XXX.XXX.XXX", 
  "url": "http://httpbin.org/anything"
}

Here we get a JSON formatted response containing the request body, as seen in the data section of the response. The headers section displays the request headers we sent (the X-Amzn-Trace-Id header wasn’t sent by us, but was added on their end). The request body is used for sending additional information to the server such as form data.

An HTTP server’s job is to receive and parse requests and return the correct HTTP responses. A simple server is able to serve static HTML pages and files, but many modern applications require things such as database access, file storage, connection to external services, etc. To deal with these scenarios we need servers capable of running more intricate code for each request and generating the content to be returned to the user.

Dynamic behaviour with CGI

#

One way to add the dynamic behaviors described above is through CGI (or Common Gateway Interface). It is an interface definition for HTTP servers and it specifies how to communicate with external programs when a request is received. Then, as long as this external program conforms to the CGI specification, it can read the incoming data and generate an adequate response.

From now on we will refer to an external program which can communicate to a server via CGI as a CGI script (although compiled programs also work perfectly fine for this). HTTP servers capable of communicating with such programs will be referred to as CGI servers.

Interface description

#

When a CGI server receives an incoming request, it needs to check if it should return a static page or call a CGI script, which is usually inferred by the request path. CGI scripts are commonly accessed via paths beginning with /cgi-bin/. It then executes the corresponding script from the configured folder in a new process. General data about the request is sent via CGI Metavariables, which are similar in spirit to HTTP headers and passed as environment variables to the process, and the request body is sent via standard input. After the script is finished executing, it returns the response metadata as well as the body formatted such that it can be parsed by the server and returned as a proper HTTP response.

Precise details regarding the metadata sent to CGI scripts and back to the server can be found in the official RFC (3875), but an overview of such details is given below.

CGI Metavariables

#

HTTP headers not covered by the fields above can be passed to the CGI script via extra metavariables prefixed with HTTP_.

CGI Response

#

The output from the CGI script is returned via the standard output. It contains a header section and a body section, separated by a blank line. The header section contains one CGI header per line (not to be confused with HTTP headers) and the body is returned to the client as is. The CGI response headers define what kind of response is to be generated.

CGI response header fields

#

As with the CGI metavariables, the script is allowed to return extra headers relating to the HTTP protocol or to the specific CGI implementation.

CGI response types

#

Some CGI script examples

#

To understand the CGI interface more concretely, let’s discuss some examples. The examples below can be found in the repository for my CGI server implementation in Rust:

1
2
3
4
5
6
7

#!/bin/bash

read INDATA
printf "Content-Type: text/html\n\n"
printf "Body: ${INDATA}<br />"
export VAR=$(env)
printf "${VAR//$'\n'/<br />}"

The example above is a bash script that returns an HTML page containing the environment variables it received as well as the incoming request body.

In this case, we are accessing the environment variables and standard input directly, but some languages have tools to abstract this interfacing for us. Take for instance our next example, now in Python:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/bin/python

# Import modules for CGI handling
import cgi

# Create instance of FieldStorage
form = cgi.FieldStorage()

# Get data from fields
first_name = form.getvalue('first_name')
last_name = form.getvalue('last_name')

print("Content-type:text/html")
print()
print("<html>")
print('<head>')
print("<title>Greeter</title>")
print('</head>')
print('<body>')
print("<h2>Hello %s %s</h2>" % (first_name, last_name))
print('</body>')
print('</html>')

In this example, we use the cgi module which parses data sent via a request so that we can acess it directly. This page might be accessed after submitting the following form:

1
2
3
4
5
6

<form action = "/cgi-bin/simple_form.py" method = "post">
    First Name: <input type = "text" name = "first_name"><br />
    Last Name: <input type = "text" name = "last_name" />

    <input type = "submit" value = "Submit" />
</form>

As long as the Python script resides in the /cgi-bin/simple_form.py path, after submitting this form we will be greeted with the supplied first and last names correctly set.

CGI drawbacks and modern alternatives

#

Nowadays CGI isn’t used as much, and more modern mechanisms have been developed. One of the main drawbacks of CGI is that it creates a separate process every time it receives a request for a CGI script, which creates a lot of overhead and makes the server less performant. We have new alternatives: the Apache HTTP server, for instance, has modules supporting communication with specific languages (mod_perl, mod_php, mod_python, etc). There are also language-specific mechanisms such as Python’s WSGI and specific programs for receiving these requests, such as Gunicorn. These can be called by more general HTTP servers acting as a reverse proxy, such as the Apache server or Nginx.

Conclusion

#

The CGI specification was written in 1993 as a way to allow developers to create scripts that integrated with HTTP servers for dynamic interactions. Even though it isn’t as popular anymore, understanding the inner workings, advantages and drawbacks of old interfaces and protocols can give us an understanding of why modern solutions are implemented as they are. It can also give us an appreciation for the engineering work that goes into defining and perfecting these interfaces and protocols that will be used by people all over the world. Implementing a simple version of this certainly was a fun experience, and if you are curious to see such an implementation you are welcome to see my repository and play with it! :)

Thank you so much for reading my first post!

References

#

• RFC 3875 (the original proposal for the CGI interface)
• Chapter 3 of “CGI Programming 101”, by Jacqueline D. Hamilton
• Chapter 3.2 of O’Reilly’s “CGI Programming on the World Wide Web”, by Shishir Gundavaram

Credits

#

• Thumbnail image by Miguel Á. Padriñán on Pexels