Reading 25: Networking

A network interface is identified by an IP address. IP version 4 addresses are 32-bit numbers written in four 8-bit parts. For example (as of this writing):

• 18.9.22.69 is the IP address of a MIT web server.

• 173.194.193.99 is the address of a Google web server.

• 104.47.42.36 is the address of a Microsoft Outlook email handler.

• 127.0.0.1 is the loopback or localhost address: it always refers to the local machine. Technically, any address whose first octet is 127 is a loopback address, but 127.0.0.1 is standard.

You can ask Google for your current IP address. In general, as you carry around your laptop, every time you connect your machine to the network it can be assigned a new IP address.

Hostnames are names that can be translated into IP addresses. A single hostname can map to different IP addresses at different times; and multiple hostnames can map to the same IP address. For example:

• web.mit.edu is the name for MIT’s web server. You can translate this name to an IP address yourself using dig, host, or nslookup on the command line, e.g.:

  $ dig +short web.mit.edu
  18.9.22.69
  

• google.com is the name for Google. Try using one of the commands above to find google.com’s IP address. What do you see?

• mit-edu.mail.protection.outlook.com is the name for MIT’s incoming email handler, a spam filtering system hosted by Microsoft.

• localhost is a name for 127.0.0.1. When you want to talk to a server running on your own machine, talk to localhost.

Translation from hostnames to IP addresses is the job of the Domain Name System (DNS). It’s super cool, but not part of our discussion today.

A single machine might have multiple server applications that clients wish to connect to, so we need a way to direct traffic on the same network interface to different processes.

Network interfaces have multiple ports identified by a 16-bit number. Port 0 is reserved, so port numbers effectively run from 1 to 65535 (which is 2¹⁶ - 1, the maximum 16-bit number).

A server process binds to a particular port — it is now listening on that port. A port can have only one listener at a time, so if some other server process tries to listen to the same port, it will fail.

Clients have to know which port number the server is listening on. There are some well-known ports that are reserved for system-level processes and provide standard ports for certain services. For example:

• Port 22 is the standard SSH port. When you connect to athena.dialup.mit.edu using SSH, the software automatically uses port 22.
• Port 25 is the standard email server port.
• Port 80 is the standard web server port. When you connect to the URL http://web.mit.edu in your web browser, it connects to 18.9.22.69 on port 80.

When the port is not a standard port, it is specified as part of the address. For example, the URL http://128.2.39.10:9000 refers to port 9000 on the machine at 128.2.39.10.

__A__://__B__:__C__

* see What if Dr. Seuss Did Technical Writing?, although the issue described in the first stanza is no longer relevant with the obsolescence of floppy disk drives

A web server typically divides up the website it serves into sections, called routes. For example, the server for web.mit.edu might have routes for:

• /education (accessible by the URL http://web.mit.edu/education)
• /research
• /campus-life

and so forth.

In a web API, the route often defines the function name of the request. For example, the US National Weather Service API defines routes for:

• https://api.weather.gov/points/... to get information related to a latitude/longitude point
• https://api.weather.gov/gridpoints/... to get the forecast for a particular area
• https://api.weather.gov/alerts/active... to get weather-warning messages for a particular area

To think about calling a web API like a function, we need to understand how to pass parameters to it, and how to get results back.

There are three ways to provide parameters.

Path components. The path of the URL consists of /-separated components. In addition to specifying the function name, the remaining components may be additional parameters, for example:

• https://api.weather.gov/points/42.3541,-71.1104 provides a latitude,longitude pair as a path component
• https://api.weather.gov/gridpoints/BOX/69,75/forecast has three path component parameters: the forecast office BOX, the grid location for that office 69,75, and the type of information desired, forecast.

Query parameters. After the path of a URL comes an optional query, which starts with ? and consists of name=value pairs separated by &. You may have already seen this in your web browsing, because it is commonly used by web forms. For example:

• https://www.google.com/search?q=MIT has a query parameter q with value MIT.
• https://api.weather.gov/alerts/active?area=MA&severity=Minor has two query parameters: area with value MA, and severity with value Minor.

Body parameters. These parameters do not appear in the URL. Some HTTP requests can have a body, which is an arbitrary sequence of additional bytes. The format of the body can vary:

• plain text
• a blob of binary data, perhaps loaded from a file
• form data, which is a set of name-value pairs similar to query parameters
• JSON

Body parameters are less common than path or query parameters. HTTP requests come in two common flavors, only one of which is allowed to have a body:

• GET requests are the normal requests that curl makes, and that your web browser makes when you type a URL into the address bar or click on an ordinary hyperlink. GET requests have a URL but no body, so they can only have path and query parameters. GET is designed for observer operations that do not mutate any state on the server. A web client can be confident that repeated issues of GET will not mutate or create anything new, so if a browser needs to reissue a GET request in order to reload a page, it can do that safely.

• POST requests have both a URL and a body. You can’t issue a POST request from your web browser’s address bar, but curl can do it, and web forms often use POST as well. POST is intended for operations that change or create data on the server: mutators, producers, creators. Web clients are much more careful about reissuing POST requests. During your web browsing, you may at some point have seen a confirmation dialog box asking if you really want to resend a form – that’s because the browser is not sure whether it’s safe to make the POST twice, and potentially buy a second set of expensive airline tickets, for example.

HTTP has a few other kinds of requests as well, including PUT and DELETE, but these are rarely used in web APIs. The kind of HTTP request is called its method, rather confusingly, because it has little to do with the idea of methods of an object. The HTTP request method is more like our classification of operations on an ADT, since GET is intended for observers, and POST for mutators, producers, and creators.

An HTTP request from a client is normally followed by an HTTP response from the server, which includes two ways to provide results:

Status code. Every HTTP response has a three-digit status code, indicating whether it succeeded or not. Here are some common status codes:

• 200 OK for a request that completed successfully
• 404 Not Found for a web API or web page that does not exist
• 400 Bad Request typically used when the parameters to the request were bad for some reason
• 500 Internal Server Error when the request fails for some other reason

Reply body. Like a request, a reply can also have a body of data attached to it, of arbitrary type. Typical body types are:

• HTML (normally used in web servers designed for human browsing, rather than web APIs)
• plain text
• JSON

Text is good for simple results that are readily parsable (such as a comma-separated pair of numbers, perhaps). For complex results, however, JSON has become the preferred result type. We saw above what JSON results look like when we queried the National Weather Service for Cambridge weather.

Specifying how parameters are passed and results are returned is not enough: it fills a similar role to method signatures when defining an ADT. We still need the rest of the spec:

• What are the preconditions of a request? For example, if a particular parameter of a request is a string of digits, is any number valid? Or must it be the ID number of a record known to the server?

• What are the postconditions? What action will the server take based on a request? What server-side data will be mutated? What reply will the server send back to the client?

The control flow of a web server is an input loop waiting for incoming connections from web browsers. When a new connection arrives, the server reads and parses the request. The request is then routed to the handler registered for the route matching the request. Typically only a prefix of the request has to match the route, so the request http://web.mit.edu/community/topic/arts.html will be routed to the handler for /community unless there is a more specific (longer prefix) route registered.

The handler is then responsible for constructing the response to the request.

The most popular router for Node is Express. You can create a new Express Application object like this:

import express from 'express';

const app = express();

Note that express is a factory function that returns Application objects. Sometimes complex servers have more than one Application, but most need only one.

You can make the application start listening for HTTP connections:

const PORT = 8000;  // port on which the server will listen for incoming connections
app.listen(PORT);

and then add routes to the application using get():

app.get('/echo', (request: Request, response: Response) => {
    ...
});

Here, get refers to the GET method of the HTTP protocol, as described above. When you type a URL into a web browser’s address bar, you are telling the browser to issue a GET request.

The first argument is the route prefix, and the second argument is a callback function. The app object will call the callback function anytime an incoming request starts with the /echo prefix. The arguments to the callback are a Request object that provides observer methods giving information about the request, and a Response object with mutator methods for generating a response that goes back to the web browser.

The body of the callback function (shown as ... above) should examine the request and generate a response. As an example of how it might examine the request, if the incoming request was http://localhost:8000/echo?greeting=hello, then:

const greeting = request.query['greeting'];

returns "hello", which came from the query part of the URL (the part starting with a question mark, ?greeting=hello).

As an example of generating a response, the handler should first provide a status code (typically 200 OK when the request is successful), and the type of response it is returning (HTML, plain text, JSON, or something else):

response.status(HttpStatus.OK); // using http-status-codes library
response.type('text');

And then write the response using the send method:

response.send(greeting + ' to you too!');

The response is now complete, and the web browser should display the text passed to send.

These mutator operations (status, type, and send) are declared to return the Response object itself rather than just void, so it is common to see them chained together like this:

response.status(HttpStatus.OK).type('text').send(greeting + ' to you too!');

or, more readably:

response
  .status(HttpStatus.OK)
  .type('text')
  .send(greeting + ' to you too!');

This kind of function-calling syntax – where the return value of a method is immediately used to call another method – is called method call chaining.

Here is the full code for the /echo route:

app.get('/echo', (request: Request, response: Response) => {
  const greeting = request.query['greeting'];
  response
    .status(HttpStatus.OK)
    .type('text')
    .send(greeting + ' to you too!');
});

Once again, this lambda function is an example of a listener callback: a piece of code that we as clients are handing to the Application object, for it to call whenever an event occurs, in this case the arrival of a request matching the route.

Since the Specifications reading, we’ve used the term client to refer to code that uses a class. In this reading, we started using client to mean one party in a network client/server communication. With Express, we need to keep track of both specific meanings of client:

• A client of the web server is a web browser, sending it requests over the network. For example, a client of our Echo web server sends an HTTP request to GET /echo?greeting=hello

• A client of the Express Application object is code we’ve written against the API that it provides. For example, a client of Application calls get() and passes in a callback function that is ready to handle requests for the /echo route.

Although the full topic of error handling in Express is outside the scope of this course, fortunately the simplest kind of error handling is easy: throwing an exception in the callback function will send an error back to the browser showing the stack trace. For example:

app.get('/bad', (request: Request, response: Response) => {
  throw new Error('always fails');
});

displays something like this when a web browser visits http://localhost:8000/bad:

Error: always fails
    at ./server.ts:30:11
    at Layer.handle [as handle_request] (./node_modules/express/lib/router/layer.js:95:5)
    at next (./node_modules/express/lib/router/route.js:137:13)
    at Route.dispatch (./node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (./node_modules/express/lib/router/layer.js:95:5)
    at ./node_modules/express/lib/router/index.js:281:22
    at Function.process_params (./node_modules/express/lib/router/index.js:335:12)
    at next (./node_modules/express/lib/router/index.js:275:10)
    at expressInit (./node_modules/express/lib/middleware/init.js:40:5)
    at Layer.handle [as handle_request] (./node_modules/express/lib/router/layer.js:95:5)

Displaying a stack trace to the user wouldn’t be appropriate for a production web server, but it’s a good way to get started.

A web server often has to do some work in response to a web request: reading files from the filesystem, making queries to a database, writing data somewhere. If that work involves calling asynchronous functions, then we may need to use await in the callback function, which means in turn that the callback function must itself be defined with the async keyword:

app.get('/lookup', async (request: Request, response: Response) => {
  ...
  const data = await database.lookup(query);
  ...
});

In the current version of Express (Express 4), this code mostly works, except that the automatic exception-handling shown in the previous section doesn’t work anymore, because Express 4 wasn’t written to catch errors from a Promise returned by the callback function. It only catches exceptions thrown by the initial call of the callback function.

Fortunately, there is a simple module that patches this, by wrapping your async function with a function that handles the promise correctly:

import asyncHandler from 'express-async-handler';

app.get('/lookup', asyncHandler(async (request: Request, response: Response) => {
  ...
  const data = await database.lookup(query);
  ...
}));

Express 5 (the next version, currently in beta release) fixes this problem.