6.313 Term Paper

Maintaining Memory Coherence in a Wide-Area

Computation Environment

Roger R. Dingledine Professor: Tom Knight

6.313 Term Paper  Roger Dingledine

Contents

1 Introduction and Problem Description 1

2 General Solutions 2 2.1 PVM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Globus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.3 Legion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.4 WebOS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Specific Solutions 10 3.1 Directory-based Cache Ownership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2 Massive Prefetching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.3 Multiple Coherence Granularities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.4 Miss Checks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.5 Relaxed Memory Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4 Conclusion 13

1 Introduction and Problem Description

In today’s world of increasingly complex computational problems and increasingly powerful computers, many people are turning to distributed computing as a means of accomplishing their tasks. Companies with sufficient resources set up extremely fast intranets and place their computers upon these intranets, to ensure very low latency for messages between them. However, the problem becomes much more complex when the computers involved are owned by single users wishing to set up a multicomputer across the Internet; typically these computers are in different states or even different countries, and have an expected round-trip latency of perhaps 100ms to a full second or worse for message passing.

The example application that I will focus on here is a distributed virtual world, where users connect from around the world and interact with the database and each other. The need for nearby servers is clear, and it would be convenient for all the servers to share the same database in order to maintain a coherent world for the users. Specifically, the requirements for this application ideally include:

• Clients and servers use only standard widely available inexpensive hardware.

• Clients and servers can transparently use different operating systems and different types of hardware.

• While some amount of specialized code is needed to handle the interconnections, the majority of the code should not need to know the current topology or configuration of the system.

• Each client can have a relatively fast network connection to his local server, e.g. for high-resolution graphics.

• The servers communicate with each other over normal Internet connections, potentially across very large geographical distances and networks with unreliable service.

• The databases maintain coherence: one person can pick up an object on a server in Turkey, a different person can try to pick up the same object in the virtual world on a server in Florida, and the dispute will get resolved correctly in a reasonably short amount of time.

• Simulated users: each server will control the behavior and actions of an arbitrary number of virtual users, including their interaction with the rest of the users and virtual users worldwide.

• Mobile login: the user in Turkey can fly to Florida and connect to the system without excess overhead in authorization.

• The system is fault tolerant: if a server falls off the network unexpectedly or needs to be upgraded, the rest of the servers recover transparently and continue operation.

• The system is incrementally scalable, meaning new servers can be added and load will shift to them.

• The connections made between clients and servers, and between servers and other servers, must be able to be authenticated and encrypted to ensure that nobody tampers with the communications.

In short, the system needs to behave just like a typical single-server model, using normal hardware, normal Internet connections, and for the most part normal software conventions. The primary concession that we will give is that each individual server has sufficient CPU power to do just about anything it wants locally, and it can have an arbitrarily large bandwidth to any of its local clients. The bottleneck is entirely in the slow unreliable link between the servers, since the servers must remain geographically apart while the clients can conceivably login to or near the console of a given server.

2 General Solutions

There are a number of large projects and applications designed to act as a global computing network, to allow computers around the world to share resources and distribute applications more evenly. Their primary solution to the latency problem is to build a very fast network between the participating servers. In addition, they tend to divide applications up between computing environments, meaning a given application is only running at one location. This means that most of their energy is spent maintaining a good idea of which machines are in use and which are free, and in transferring the materials necessary to start a job on a given site. While these design requirements do not entirely match the above goal that data flow quickly, reliably, and frequently between the servers, they have done some very important research in that direction. Following is a brief summary of several pioneering global computation designs, with an eye towards their work in minimizing communications latency between geographically separate servers, or alternatively using this system itself to solve our problem.

2.1 PVM

In the Parallel Virtual Machine (PVM) model of supercomputing, there are a number of machines all running the PVM daemon. These machines can range from massively parallel supercomputers to scalar workstations. Each user has a set of machines within the PVM (called a ‘pool’) on which his applications may run; he can optionally specify specific machines, architecture types, or requirements within the pool for specific tasks [Sunderam 1990]. For example, problems which require a lot of time and drive space should not be running on pool elements with low-end CPUs and small hard drives.

The unit of parallelism in PVM is the ‘task’ – a single-threaded process which alternates between communication and computation^{1}. When you want to design an application for PVM, you write it in the form of tasks, where each task does some subsection of your desired computation. For instance, you might have a task for input, a task for computation, and a task for display; the computation task might actually be designed to run in parallel (SPMD), with many versions of it each running on different machines in the pool and each working on a different subset of the input data. PVM applications need to be started manually from a single task, and from there additional tasks may be spawned automatically by the initial task (sometimes called the master task) or also started manually.

The tasks in a PVM communicate by explicit messages. Each task is compiled and linked with the PVM library, which provides interface routines for cooperation between the various tasks of an application, and makes use of internal message passing protocols (such as for mulitprocessor machines) where possible. Since all of this is done explicitly by the programmer, there is an enormous amount of control available. The PVM library is only available for C, C++, and Fortan 77, meaning there are a limited number of languages available for applications; but C is the modern language of choice for fast flexible programs, so that isn’t much of a problem.

For the distributed virtual world, this could be implemented as a single application, with tasks running on each of the servers. While this provides a good framework for the system, including very convenient libraries to pass messages between the tasks, it doesn’t actually provide as much as it seems: an alternative implementation using separate processes manually spawned and maintained at each server, communicating via the UNIX IPC message control mechanism, would be just as good (if not better) in terms of performance.

1http://www.netlib.org/pvm3/book/node17.html

2.2 Globus

The Globus model for parallel computing is a much more comprehensive framework, including a set of resource management utilities, support for remote file access and execution, security and fault detection, and a directory-based system-, architecture-, and task-database called the “Metacomputing Directory Structure 2”. Globus is a joint project between Argonne National Laboratory and USC’s Information Sciences Institute: their goal is to create a wide-area parallel computing infrastructure that can be used by many universities at once. The core of their design, at least from the standpoint of efficient interaction between machines, is based around the Nexus communication library and their implemention [Foster 1998] of the Message Passing Interface (MPI) standard. The strength of the message passing paradigm lies in its high-level view of communications; earlier metacomputing prototypes often opened TCP/IP sockets directly, making applications difficult to port from one architecture to another. Nexus abstracts away the underlying network topology and configuration, providing the programmer with a simple and standard interface while still making efficient use of the available networks. Some of this efficiency is gained by making use of the multi-computer aspect of the environment, meaning if many messages are going to a single endpoint they can be merged on the way, reducing network usage; similarly, a message can be sent to multiple destinations via a multicast system. And while there have been several similar designs for an efficient low-level message passing system, such as Active Messages (AM) [Mainwaring 1996] and Fast Messages (FM) [Pakin 1996], the Nexus system boasts lower overhead in terms of message size, as well as a more flexible environment via its one-sided communication paradigm.

By using the Globus Metacomputing Directory Service (MDS), the system is able to efficiently keep track of which servers are up, which networks and available, and how loaded each resource is. The MDS makes use of the Lightweight Directory Access Protocol (LDAP) [Howes 1995], a widely accepted standard for interacting with a directory system, to process queries and new information regarding system resources. This ends up providing a much richer set of information than the previous solutions (which used text files, for example), as well as allowing dynamic update of data including automatic discovery of new resources, and distributed maintenance of information for decreased latencies and increased availability.

2http://www-fp.globus.org/details/

One of the most important features of the Globus wide area computing toolkit is the Remote I/O (RIO) system, which attempts to address some of the high-latency issues for long-distance communication over unreliable links [Foster 1997]. In particular, the RIO library makes use of the Nexus communication library for client/server communication, allowing remote access to files for read and write, as well as access for remote execution. The goal is to reduce time spent in “staging” – transferring files from the source site to the site on which the user wants to run them – both by automating the process for the sake of convenience, and by configuring it so it fetches only the parts of the input files that are actually requested by the executing program, and optionally prefetches parts of the input files which it knows it will require soon. In addition, there are several performance optimizations which specifically address high-latency communications:

• Forwarder nodes: client and server processes communicate directly, which means that in the general case each process will have to know how to use a variety of communications channels, since some of the connections will be with nearby processors over local MPI interfaces, and some of them will be over a TCP/IP-based channel with remote machines. This dramatically increases the overhead for that process in terms of keeping track of the communications channels. If forwarder nodes are introduced that simply keep track of a single communication channel type, then they can not only simplify the function of the other computational nodes by removing the communications burden, but they can also optimize communications by being designed specifically for communication over their channel, and also by throttling traffic when necessary, to avoid network congestion down the line.

• Certain types of requests are sent from multiple clients to a single server; these requests can be collected, possibly by a forwarder node, and bundled to the server in a single request. This allows a drastic decrease in the amount of I/O time that the server needs to spend dealing with these clients.

• Since the round-trip message time is so high, making use of asynchronous I/O operations will allow computation to proceed much more smoothly. Asynchronous operations allow several I/O requests to be outstanding at any given time – thus computation and I/O will be overlapping in the application itself.

• Their last optimization is simply to reduce the number of round-trip messages that are sent over the slow line. This can be accomplished by client-side buffering, either accomplished by each client individually, by several clients in a group (such as for item two above), or by the forwarder nodes themselves. Basically, this means waiting until a certain number of messages have accumulated before sending them all at once; it assumes a fairly constant stream of messages.

The Globus project offers many very important features for solving the distributed virtual world problem. In particular, it provides a framework that automates the process of keeping track of which servers are up and the best way to get to them, as well as much of the dirty work with the low-level protocols, especially the message passing interface and implementation. The ability to reduce the required amount of network traffic over high-latency lines, as well as the ability to prefetch certain materials and use asynchronous I/O to pipeline requests, might contribute significantly to a successful implementation of the distributed virtual world. However, there are still several problems. First of all, several of the performance optimizations, such as the multiple client to single server message collection, or even the message buffering and bundling itself, are only proposed and currently unimplemented. Secondly, all of the experimental data that Foster mentions is done on a single IBM SP multicomputer, to allow for “a controlled environment in which the impacts of different performance issues can be easily measured.” Experimental results for Nexus latency on a fast local ethernet are in the tens of microsecond range, which is effectively zero compared to latency for transcontinental networks. This means that the actual issues at hand have not yet been well-tested, and the implementation of the infrastructure is not yet complete. While Globus might be a useful starting point, it is far from a complete solution.

2.3 Legion

The Legion infrastructure is very similar to the Globus system, in that it attempts to provide a solution not only for multicomputer parallel processing, but also for directory services, seamless communication between tasks, etc [Grimshaw 1996]. Legion is an ongoing project at the University of Virginia to create a prototype for a global infrastructure of interconnected machines. It has some advantages over the Globus project, as well as some weak points.

Legion is based on the object-oriented paradigm of computing: each task in Legion is an object, and it interacts with other objects via methods [Lewis 1996]. Their focus is on performance and simplicity; they’ve designed the Legion system to allow for multiple actual implementations of each segment, as long as each implementation behaves according to the Legion specifications. By taking a top-down approach to the modularization of the system, they hope to make it much more scalable than the sum-of-services approach of the Globus system [Viles 1997]. Their object model makes Legion primarily for remote execution of coursegrain applications that fit on a single address space – it’s not so much a structure for distributing large complex processes, as a structure for distributing lots of small processes more smoothly and conveniently.

On the other hand, they have a very flexible design for a file system called ELFS (ExtensibLe File System^{3}) which has a lot of similarities with Globus’s RIO library. It supports user specification of caching and prefetch strategies, asynchronous operations for overlapping I/O (including prefetching) with computation, multiple outstanding I/O requests, and data format heterogeneity. This last item is most interesting, because Globus seems to have overlooked it – where Globus just assumes that each task is going to deal with encapsulating its data before sending it, ELFS automatically translates data as it is read or written, based on the methods within the process (object).

Rather than a straight message passing interface, Legion makes use of a high-level interface description language (IDL), which provides RPC-like interaction between the processes. For low-level interaction, it uses the Active Message [Eicken 1992] protocol; this basically puts more intelligence in the messages themselves, allowing them to specify which processes they’re destined for and what should be done with them when they arrive. While this clearly offers more flexibility, Globus shied away from this idea because of its high overhead.

3Where do they get these names?

The main feature of Legion is that it’s relatively mature: it has been designed pretty thoroughly, and its core has been implemented and has been running as a testbed at the University of Virginia for over a year. On the other hand, they still store their directory information in a static text file, meaning dynamic addition of machines is not smooth or transparent. In addition, the experiments they’ve reported [Vahdat 1996] were run only with a few dozen Sparcs and Indigos, meaning their arguments about better support for massive scaling are still purely theoretical.

The most important idea that the Legion project brings to the solution of the distributed virtual world problem is their security model [Wulf 1995]. Their goal is to have Legion services running on as many machines on the Internet as possible, and therefore they need an effective security system – effective not only from the point of view of the machines hosting the Legion tasks, but also from the point of view of each Legion task. This means that in theory, the task of maintaining the virtual world can be distributed among hundreds or even thousands of participating cites, decreasing the maximum distance between servers. However, this has several problems. First of all, this doesn’t directly solve the latency problem in maintaining coherency between any two given servers – and it might even increase the problem, by increasing the complexity of communication between all the servers. Secondly and more immediately important, Wulf’s paper demonstrates an alarming lack of concern for the implementation side of his ideas. Starting with his assumption that users will not make use of Legion unless it’s secure, he proceeds to describe an objectoriented model that will protect the Legion objects from malicious objects calling methods and trying to cheat at the authentication stage. However, he neglects to address the idea that his “objects” won’t be operating in theory-space, but rather on an actual machine with an actual kernel that can read and modify memory arbitrarily. Until this issue is addressed thoroughly (perhaps through encryption and signatures), running the virtual server application on untrusted servers is not an option.

2.4 WebOS

Berkeley’s WebOS project approaches the problem from a very different angle. They have designed and written several simple CGI scripts for use by normal web servers, to turn the web communication paradigm into a two-way system. In short, they have taken the global name space provided by the world wide web, and built a cache-coherent authenticated global file system on top of it.

To do this, WebFS ^{4} needs to address many issues. They designed a global file system, a secure cachecoherent file system to allow applications to interact with guarantees that the files they manipulate are up to date and the changes they make will be propagated quickly. While they could have used AFS as such a file system, it has several limitations, such as its invasiveness (kernel modifications are required for each client) and more importantly the fact that very few sites currently run AFS. They provide a loadable device driver called SLIC which patches the syscall table to intercept file system operations which reference URLs [Vahdat 1996], meaning that applications can transparently refer to URLs as files on the file system. Following this, they wrote CGI scripts for ‘open’, ‘read’, ‘close’, ‘stat’ (perform a Unix stat(1) on the file), and ‘access’ (determine if the user has access rights to this file). For data consistency, they implemented the same cache consistency model as AFS (“last writer wins”) – each host maintains a directory of which remote sites are caching copies of each item; when the item is modified, all other copies are sent invalidation notices.

Also, they needed to cover the issues of security and authentication. The authentication is done by public key encryption, where each host knows the public keys of the other hosts and can provide digital signatures to authenticate each request. The security itself is gained by specifying access lists for each file or directory tree, meaning the requesting user needs to authenticate himself as one of the trusted users allowed to access the given file [Belani 1998]. The use of public key cryptography rather than shared key cryptography introduces the issue of securely distributing public keys, but this was deemed more acceptable than the problem introduced by shared key cryptography of having a trusted third party knowing all the shared keys. In order to implement secure remote execution of applications, they created another CGI program called ‘execute’, which is given a local application and a desired destination host. Since the program is run as user ‘nobody’ with very few privileges (they even intercept and disallow potentially damaging system calls), the CGI program has the ability to receive tickets from the user representing transfer of access. For instance, the calling user might give the application the ability to write to a certain directory to which only the user has access, in order to keep the results relatively private.

4A technical draft describing WebFS in more detail is available at http://www.cs.berkeley.edu/~vahdat/webfs/webfs.html

Along with these two major aspects, several other glue elements need to be created. For instance, new protocols need to be developed to take advantage of this system. Specifically, they need to be able to make use of the authentication layer of the WebFS system, as well as allow applications to specify the level of cache coherence required. Transactions need to have well-defined failure modes, meaning a program which is running on a remote machine should not leave its host file system in an inconsistent state if it aborts unexpectedly [Vahdat June 1998].

The WebOS project has implemented some fairly impressive applications to work on top of their metacomputing environment. For instance, their “Rent-A-Server” application allows web servers to react to periods of increased load by shedding load onto other idle servers, which use the WebFS system to mirror the primary server [Vahdat July 1998]. This means that any given server only needs to provide hardware that will deal with the average load for the machine, and it can provide support to nearby servers during periods of light load, and get support from other servers during periods of heavy load. Another application is the Internet Weather Service ^{5}, which allows users to query network load on any given major segment of the Internet. In theory, in exchange for providing the answer, users will return the data they’ve just gotten regarding the most recent statistics for that network link, and so the weather service can keep relatively up to date information about many different links on the network simply be responding to queries.

3 Specific Solutions

While the above solutions present some very compelling ideas for the distributed virtual world problem, none of them provides a sufficient infrastructure that can be used easily. One major feature of them is that they’re designed for arbitrary calculations, and to support more than one application working concurrently; in our case, we only need a single application, but we need it to run very well in parallel. The most important problem is high network latency; for the most part, the most effective and widespread solution to the latency problem is effective caching – trying to make sure an up to date copy of the data that a given server wants to use is already present locally at the time it’s requested. There are a number of features to these general solutions which scale relatively well to conditions of extremely high latency.

5for instance, http://www.internetweather.com/

3.1 Directory-based Cache Ownership

The single most important solution is the idea of directory-based ownership for each object in the virtual world [Lenoski 1990]. Basically, each server has a directory which lists “ownership” of each object – when a given server owns an object, then it has the power to modify it. It keeps track of other servers which are using that object as well, and after modification sends invalidation notices to those other servers. The server which owns a given object is sometimes referred to as its ‘home’.

When a server wants to own an object (typically because it wants to make a modification to it, or expects to soon), it sends a request to the current owner. If it’s lucky, the current owner yields ownership, and eventually gets around to notifying the rest of the servers of the change in ownership (notifications might be queued to be distributed in bulk via multicast, or might be piggy-backed on top of other messages to servers). If the server that is contacted is not the current owner, then it knows who the current owner is likely to be, because it yielded ownership of the object to that server relatively recently. Thus in most cases, ownership requests take one or at most several hops before arriving at the current owner.

3.2 Massive Prefetching

Because ownership of an object grants the ability to freely modify it, the high latency of the network is much less of an issue except during ‘change of ownership’ requests. Further optimizations can be gained by heuristics to try to predict when an object will want to own an object, and obtain ownership before it’s actually necessary [Chen 1992]. For instance, if a client is on a particular server and moves into a room (object) that is not homed at that server, it’s reasonable to expect that he might want to interact with that room in the near future. On the other hand, if there are other clients in that room that are based at other servers, it is much less clear where it should be homed. Probably some set of heuristics that take into account the level of client activity and previous behavior, as well as an integer associated with a request-for-ownership that indicates the urgency of that request (ranging from “I might want to do something with that someday” to “I need to modify that right now”), will end up being most appropriate.

3.3 Multiple Coherence Granularities

Another very important aspect to the object distribution is that the objects will be different sizes; some of an object’s members will be directly or indirectly dependent on other members in the object, and some of them will be independent. This means that supporting multiple coherence granularities in the database [Scales 1996], and in particular exploiting knowledge of the database format itself, is crucial to allow objects to be owned by multiple servers if each is trying to modify a different attribute of the object. The main tradeoff here is that it’s tricky to determine which members of an object are independent, and more resources are used at each server if objects are split up into sections. In addition, the overhead of transferring ownership of all sections of an object (which might turn out to be the most common behavior) might not be worth it. In any case, remaining aware that ownership should be at least at the object level will help to eliminate the ‘false sharing’ problem that many distributed memory systems have, where two servers fight over ownership of a given object even though they intend to modify independent parts of it. A large array is a good example of this scenario [Scales 1996].

3.4 Miss Checks

The mechanism of intercepting a memory request for an item not stored locally and transparently fetching the item is known as a “miss check” [Zwarkadas 1996]. In the case of our application, the miss checking will have to be particularly intelligent, because it will involve efficiently querying the local ownership directories and then obtaining an up-to-date version of the data. In many applications, miss checking is done as a library linked at compile-time, intercepting all memory accesses; this increases the overhead somewhat, but it’s not enough to be comparable to the degree of network latency that the application will be encountering. Because it’s done at compile-time, some amount of parsing and interpretation of the source code can be done to assist in prefetching of required data [Callahan 1991]; however, given the nature of the virtual world where human users will be choosing actions, this becomes much less reasonable for the majority of the data. On the other hand, the parallel nature of the server means that infrequent fractional-second delays are reasonable while data changes ownership. If one of the general solutions from Section 2 is used, this low-level issue will hopefully be handled by that solution; alternatively, a simple library can be added that takes care of doing the remote fetch and update transparently, via Nexus or another implementation of the message passing interface standard.

3.5 Relaxed Memory Models

Another optimization that should be considered is making use of relaxed memory models for the virtual world. That is, in some cases ownership may not be required for making a modification to an object: instead, simply passing a summary of the requested changes to the current owner suffices [Scales 1996]. Indeed, this implies that such writes can be non-blocking assuming a reliable underlying transport and message delivery system.

4 Conclusion

“The greatest challenge, and least agreement among practitioners, is found in software support for parallel systems.” [Fox 1991] Granularity (number of nodes, as well as object size), communication bandwidth and latency, interconnect topology, overlap (or not) of communication and calculation, and node architecture itself all play important roles in the interaction and execution of parallel computing systems.

However, the distributed virtual world problem is one of the most complex, since it requires very strict cache coherence among servers that must remain geographically separate. The majority of the currently proposed solutions assume that the latency for a remote fetch can be limited to around a thousand processor cycles. Over a typical Internet connection, the remote fetch latency is perhaps four orders of magnitude higher. While there are a variety of solutions which scale to this degree of latency, none of them are yet able to fully address the requirements of the distributed virtual world application.

On the other hand, there are a number of important techniques which have been relatively well-researched. Combined, these techniques (along with some experimentation and further research) will soon be sufficient to implement a prototype application.

Sources

Eshwar Belani, Amin Vahdat, Thomas Anderson, and Michael Dahlin. “The CRISIS Wide Area Security Architecture.” In Proceedings of the 1998 USENIX Security Symposium, January 1998.

R. Bianchini, L. I. Kontothanassis, R. Pinto, M. De Maria, M. Abud, and C. L. Amorim. “Hiding Communication Latency and Coherence Overhead in Software DSMs.” In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Oct 1996.

Callahan, D., et al., “Software Prefetching”, In ASPLOS-IV, April 1991.

Tien-Fu Chen and Jean-Loup Baer. “Reducing memory latency via non-blocking and prefetching caches.” In Proceedings of the 5th ASPLOS Conference, pages 51-61, 1992.

S. Dwarkadas, A. Cox, and W Zwaenepoel. “An integrated Compile-Time/Run-Time Software Distributed Shared Memory System.” In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Oct 1996.

T. von Eicken, D. E. Culler, S.C. Goldstein, and K.E. Schauser. “Active Messages: A Mechanism for Integrated Communication and Computation.” In Proceedings of the International Symposium on Computer Architecture, pages 256-266, 1992.

I. Foster, D. Kohr, R. Krishnaiyer, J. Mogill. Proc. “Remote I/O: Fast Access to Distant Storage.” Workshop on I/O in Parallel and Distributed Systems (IOPADS), pg. 14-25, 1997.

I. Foster, J. Geisler, W. Gropp, N. Karonis, E. Lusk, G. Thiruvathukal, S. Tuecke. “Wide-Area Implementation of the Message Passing Interface.” Parallel Computing, 24(11), 1998.

Fox, Geoffrey C. “Achievements and prospects for parallel computing.” Concurrency: Practice and Experience, 3(6):725-739, December 1991.

Grimshaw, A.S., and Wulf, W.A. “Legion – a view from 50,000 feet.” In Proceedings of the Fifth IEEE Symposium on High Performance Distributed Computing, IEEE Computer Society Press, Syracuse, N, August 6-9, 1996.

T. Howes and M. Smith. “The LDAP application program interface.” RFC 1823, 08/09 1995.

D. Lenoski et al. “The Directory-Based Cache Coherence Protocol for the DASH Multiprocesor.” In Proceedings of The 17th Annual International Symposium on Computer Architecture, Seattle, WA, pages 148-159, 1990.

Lewis, M., and Grimshaw, A.S., “The core Legion object model.” In Proceedings of the Fifth IEEE Symposium on High Performance Distributed Computing, IEEE Computer Society Press, Syracuse, N, August 6-9, 1996.

A. Mainwaring. “Active Message applications programming interface and communication subsystem organization.” Technical report, Dept of Computer Science, UC Berkeley, Berkeley, CA 1996.

B. Clifford Neuman and Santosh Rao. “The Prospero Resource Manager: A Scalable Framework for Processor Allocation in Distributed Systems.” In Concurrency: Practice and Experience, 6(4):339-355, June 1994.

S. Pakin, M. Lauria, and A. Chien. “High performance messageing on workstations: Illinois Fast Messages (FM) for Myrinet.” In Proceedings of Supercomputing ’95. IEEE Computer Society Press, 1996.

Daniel J. Scales, Kourosh Gharachorloo, and Chandramohan A. Thekkath. “Shasta: A Low Overhead, Software-Only Approach for Supporting Fine-Grain Shared Memory.” In Proceedings of the 7th International Conference on Architectural Support for Programming Languages and Operating Systems, Oct 1996.

V. S. Sunderam. “PVM: A framework for parallel distributed computing.” Concurrency: Practice and Experience, 2(4):315-339, December 1990.

Amin Vahdat, Michael Dahlin, and Thomas Anderson. “Turning the Web Into a Computer.” University of California, Berkeley, Technical Draft, May 1996.

Amin Vahdat and Thomas Anderson. “Transparent Result Caching.” In Proceedings of the 1998 USENIX Technical Conference, June 1998.

Amin Vahdat, Thomas Anderson, Michael Dahlin, David Culler, Eshwar Belani, Paul Eastham, and Chad Yoshikawa. “WebOS: Operating System Services For Wide Area Applications.” In Seventh Symposium on High Performance Distributed Computing, July 1998.

Viles, C.L., Lewis, M.J, Ferrari, A.J., Nguyen-Tuong, A., and Grimshaw, A.S. “Enabling flexibility in the Legion run-time library.” In Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications (PDPTA ’97), pp. 265-274. Las Vegas, Nevada, June 30-July 2, 1997.

Wulf, W.A., Wang, C., and Kienzle, D. “A new model of security for distributed systems.” University of Virginia Computer Science Technical Report CS-95-34, August 1995.

Chad Yoshikawa, Brent Chun, Paul Eastham, Amin Vahdat, Thomas Anderson, and David Culler. “Using Smart Clients to Build Scalable Services.” In Proceedings of USENIX ’97, January 1997.