Design Projects

Project assignment // New information // Pointers // Post-mortem // Design project 2

Design project 1:
A cache for the World Wide Web

This is where you will find the most up to date information on design project 1. The design project handout itself is available in PostScript or HTML form.

  Design project 1: A cache for the World Wide Web

Design project 1 is due in recitation on Thursday, March 21.

New information

Here is a list of new information regarding the design projects. Please send us mail (6.033-staff@mit.edu) if you have suggestions to add to this list.

• The official policy on late papers is as follows: For each recitation that you wait to turn in the paper, you lose one full letter grade on the paper. So if you turn in the paper by the Tuesday after Spring Break, you are graded normally and then one full letter grade is subtracted. If you wait till the Thursday after Spring Break, then two full letter grades are subtracted. Note that you need a B or higher to have the paper qualify for Phase II. 3/19/96, 4pm

• There are now a few design project-related questions and answers on the 6.033 discuss meeting. (Click here to jump directly to the meeting.) Feel free to post anything here, including questions or general comments. 3/19/96, 4pm

• Due to several requests by 6.033 students, the SIPB webmasters have prepared some interesting information about the statistics for the www.mit.edu web server. 3/19/96, 4pm

• If you really want the most up to date stats on MITnet traffic, you can check out the Network Watch discuss meeting. These stats are somewhat useful in that they give a breakdown of traffic per router. 3/19/96, 4pm

• More stats! Matt Braun has generated some MITnet backbone statistics that provide a breakdown by protocol. HTTP is the clear leader. NOTE: There was a mistake in these stats that has now been corrected at the last minute. Do not worry if you cited the old percentages in your paper, as the recitation instructors are aware of this last minute change. updated 3/21/96, 3:00am

Pointers

We have compiled a set of pointers that seem relevant to this project. Please send us mail (6.033-staff@mit.edu) if you have suggestions for additions, clarifications, errata, or whatever.

• A good starting point is a paper by Steve Glassman called "A Caching Relay for the World Wide Web". A hardcopy of this paper was attached to the design project handout [See footnote]. Note that this paper, written in 1994, discusses a cache for use by about 60 people, requesting about 2000 documents a day. How does that compare with MIT in 1996? When you read Glassman's paper, be sure to think about the scaling issues in relation to the cache that you will design.

• Here are several existing projects that involve using HTTP proxy servers as a cache. If you don't know what a proxy server is, you can read more about them in the specification for HTTP. (That link should place you close to the definition of a proxy server; just search for "proxy" in the document to find out more.) While you are following the links below, be sure to identify problems that these projects would have if they were applied directly as a solution to the design project.
  • The W3C web server supports caching in proxy mode.
  • The Harvest Cache and HTTPD-Accelerator is a spin-off of the Harvest project at the University of Colorado, Boulder.
  • Lagoon is a proxy-cache project developed at Eindhoven University in the Netherlands.
  • There is also a cache for all of Britain (!!): the UK National Web Cache at HENSA Unix.
  • NLANR has a project investigating an efficient national architecture for handling highly popular data. It also includes references to other project.

• The World Wide Web Consortium (W3C) is the place to look for web specifications. Most notably, you might be interested in the specs for HTTP or URLs.

• For other Internet specifications, you should refer to Internet Requests For Comments (RFCs). IP is defined in RFC 791, and TCP is defined in RFC 793.

• You can learn more about MITnet by checking out the MIT Network Operations page. Most notably, there is a map of the topology of MITnet, and a graph of MIT backbone traffic for the past 5 years. There is also an ASCII listing of the data from this graph. Note that traffic from external sources (e.g., an MIT user fetching data from a remote web page) has grown exponentially from about 2 gigabytes a day to 20 gigabytes a day in the past five years. That should be enough to convince you that this design project is addressing a real-world problem.

Post-mortem on the design project

Now that design project 1 is finished and graded, David Wetherall and the rest of the staff have put together a page of notes on design project 1, including the criteria used to grade the projects and a bunch of possible solutions.

Top // 6.033 home // Last updated 3/17/96