A design project is a different beast from a quiz. As in real life, the 6.033 design projects are underspecified, and it is your job to complete the specification in a sensible way given the overall requirements of the project. As with designs in practice, the requirements often need some adjustment as the design is fleshed out.
Questions and answers will be posted here as they are generated. Newer questions and answers will go at the top.
Q: Should the Executive Summary have illustrations?
A: Yes, a figure or two illustrating your system's general architecture would be appropriate in the Executive Summary.
Q: Regarding crashes in the transcoder and the AI software, the DP1 assignment says "a human operator will notice such crashes and restart those modules." To what attention should we give crash recovery?
A:You should not ignore crash recovery, but you do not need to read chapter 7 in order to do the design project. Specifically, you don't need to give us psuedo-code for how your system will recover from crashes, but you should at least address the issue with a few sentence in english.
For example, saying that the human operator will reboot the entire computer and restart everything in the event of a crash is not an acceptable answer. You should address:
You can assume that your operating system has a kill() system call that allows one process to kill another process. Pipes and sockets between killed processes and processes that are still living get closed, so an attempt to read on such a pipe gives an end-of-file while an attempt to write on a pipe or socket generates an exception.
Q: I went to the Writing Lecture on Friday. What they told doesn't exactly match the writing requirements on the DP1 description. Which one is correct?
A: As the writing faculty and technical faculty talked about DP1, we came to realize that the initial guidelines may not best capture the qualities of an outstanding design report. The lecture notes from Friday's lecture are your best organizational guide. We also added a longer description about the Executive Summary to the DP1 FAQ.
Here are some organizational differences between the DP1 assignment and the Friday lecture description:
1.0 Design Overview
The goal of this design is to provide . . . We accomplish this goal by . . . . We achieve fault isolation by . . Our concurrency model . .
5.0 Conclusion
Our Surveillance@Home design uses processes to . . . This design does not cope well with long term overload because . . .[explain why you did not address this issue]
Q: May I use "I" in my report?
A: Sure, there is no problem with using personal pronouns in your report. Personal pronouns are better than passive voice.
Q: What is an Executive Summary?
A: Executive summaries provide a condensed version of a longer report's content. For DP1, summarize the actual design and its properties (and in case of DP1, how it meets the requirements).
Q: What's the format of an Executive Summary?
A: The Executive Summary is a stand alone document that is placed BEFORE the title page. Because it is placed before the title page, your name and title of your project should be placed on the Executive Summary. The Executive Summary should be organized into enumerated chunks, given descriptive headings, and highlighted. Do not staple the Executive Summary to the report.
Q: Who reads your Executive Summary?
A: Executive summaries are written for readers who do not have time to read the entire report. The target audience of the executive summary is your 6.033 TA, who has no time to read the whole document because he/she is off on spring break writing his/her thesis (or relaxing on the beach). The TA, however, must recommend A, B, C, or lower to his/her recitation instructor, based on the executive summary. Your executive summary should crisply explain how your design works, how it fits the requirements, and why it is one to select.
Q: Why did the Executive Summary originate?
A:
The dramatic increase in the sheer amount of information available on most topics has spawned a corresponding growth in the size and number of reports. Unfortunately, there has not been a similar expansion in the amount of time available to executives or policy makers to read these increasingly lengthy reports. The solution to this problem is the executive summary.(Source: http://www.columbia.edu/~ftg1/WRITING%20EXECUT.SUMMARY.html)
Q: How do we find the frame boundaries when we call transcode()?
A: You don't have to. The camera's proprietary data format transcoder data stream is self-synchronizing. That is, the data format is set up so that an arbitrary block of data can be recorded from the camera, sent to the transcode() function, and if it is possible to get a JPEG out of that data stream, the transcode() function will do that. (Or, of course, it will crash trying.)
This actually isn't so far fetched as you might think. One way to build such a data stream is to have a sequence of 8 bytes that marks the beginning of each new stream segment (what MPEG calls a "key frame.") After the key frame is sent, the camera sends a series of smaller chunk, each one indicating a set of differences from the last frame. Then, at some later point in time, another key frame is sent. The camera guarantees that the 8-byte sequence only appears in the data stream at the beginning of key frames. (There are several ways that the camera can guarentee this, including escaping data that would contain the 8-byte sequence.)
Unfortunately, we don't know how often key frames appear, but they seem to come several times each second.
Q: Does the transcoder maintain internal state?
A: No.
Q: Are we limited to 4 gigabytes of main memory?
A: No. You can assume that the central computer's architecture is 64-bit, and you can plug in lots of memory modules. DP1 doesn't specify any explicit limit to the amount of memory you can use; keep in mind that DRAM modules cost about $100/GB today.
Q: Which interface should we use for spawning threads:
fork_thread from DP1, or create_thread from
the class notes? How does fork interact with threads?
A:
The kernel interface provided in DP1 is supposed to be an example, not
a requirement. Feel free to use create_thread in your
pseudocode if you prefer.
fork creates a clone of the current process (with a new
address space) instead of a new thread. However, we didn't specify
exactly what happens if a multi-threaded process calls
fork. It makes more sense for the new address space to
have only one thread than for all the threads in that address space
to be "copied". That is, the following example spawns off a thread in
the same address space as the parent, and then spawns a process with a
single thread (not two):
create_thread( do_some_stuff_in_same_address_space );
parent thread continues to run here
pid <-- fork();
if pid == 0:
only the child process (with a single new thread and new address space) runs this
status <-- do_some_stuff_in_new_address_space();
exit( status );
Q: Can we use the select syscall and non-blocking
read/write to do asynchronous I/O within a
single thread or process?
A:
Sure, if you understand its interface well. As with any kernel
modification, just make sure you specify the intended interface
unambiguously. You might find the select manpage useful,
although you might not; UNIX manpages tend to be references, not
tutorials.
Q: How detailed should the design report or proposal be?
A: You are writing the design specification for an experienced systems programmer who is very familiar with the topics covered in 6.033; e.g., your TA. The spec should be detailed enough that the reader could sit down and write out the exact code to implement the system without making any decisions for him or herself. There should be no ambiguities in how to implement any part of the system that is core to the topic of the design project.
Note that this doesn't mean you are supposed to provide the code for everything (even the pseudocode). However, it does mean that every functional unit should have a straightforward translation to code. If you have a system diagram, it should be clear how every box is translated into a program, subroutine, class, process, or thread; and how every line is implemented as a protocol over a communication channel such as a stack, pipe, network, RPC, or so on. For example, if you have a box (a process) with three lines going in and four lines going out, you might consider whether it's possible to write blocking code to monitor that many channels simultaneously.
Q: Does the AI function require that the block of images it processes be from the same camera? Does it somehow try and correlate frames to identify a possible suspect? Or can I send it it interleaved images from different cameras, and it decided on a per frame basis?
A: Yes, the frames have to be from the same camera. The function correlates between frames.
Q: Is there a limit to the number of processes that we can have on the server?
A: You may have tens of thousands of pocesses. If you need more than that, you should explain why.
Q: Is there a limit to the number of processes that we can have on the server?
A: You may have tens of thousands of pocesses. If you need more than that, you should explain why.
Q: How many different programs can we run on the server?
A: With the limited API that we have given you, you may have many threads, but they all share the same executable. Our kernel has a fork() call but no exec() call. However, different threads can run different functions after fork() is called.
Q: Is there any policy regarding the use of a commercial web server as an alternative to writing our own? Say, Apache, with PHP, that reads from a buffer somewhere the images necessary?
A: You don't need to "write your own webserver," but you do need to specify its behavior at a high level. "Uses Apache" is not a behavior specification. "Operates in the MP [or MT] model" is closer. "Spawns a new thread to handle each request; each thread pulls a response from a shared pool of images" is better.
Q: I had a question about the transcoder. Can it transcode 1000 different video snippets at once no matter what the size, or can it only handle 1Mb per second? If its the latter, then the transcoder can only convert between 1 and 20 camera feeds at a time, and I can design the system that way. If it can do all of the feeds at the same time, this would be optimal, but sounds somewhat unrealistic. I reread the specs on the transcoder, but it was a bit vague, so maybe you could clear it up.
A: We didn't specify the performance of the transcoder, except that under "normal" conditions the CPU can handle all 1000 streams at once in real time. Under load spikes of certain types, the transcoder might temporarily exceed the capacity of the CPU to keep up.
Q: What should we put in the design proposal?
A: The DP1 assignment says:
Design proposal (800 words or less, approx. 2 pages). Due: February 26, 2004. This should be a concise summary of your design choices and of the overall system design. Also, if any of your design decisions are "unusual" (particularly creative, experimental, or risky, or specifically warned against in the assignment), it would be wise to describe them here. You will receive feedback from both your TA and the Writing Program in time to adjust your final report.
The audience for the design proposal is the 6.033 staff. You should assume that we're familiar with the DP1 assignment and with the material taught in 6.033. In particular, you shouldn't repeat anything from the assignment in your proposal; you will need to make every word count towards explaining your design.
You should try to explain your design such that an experienced systems programmer (e.g. your TA) could write the code for the central computer just from reading your description. Obviously, you will need to limit the level of detail to fit within the 800 word limit, but try to be as detailed as possible within that bound.
Here are a few general writing recommendations:
Most proposals also address the following issues:
Here are some ideas for focusing your proposal:
Q: How should we format the design proposal?
A: Format it so that there is room for us to write comments on your paper.
Q: Is there an example design proposal?
A: No, there isn't. This is the first year we have asked for DP1 proposals. Sorry!
Q: Is the proposal graded?
A: Yes. It will be graded like the other short assignments in this class, with a grade of A, B or C. A grade of C means that you are not on the right track. B means that you are on the right track, but something is missing. A means that you seem to be on top of everything.
Q: How does the DP1 proposal grade count towards my final grade?
A: The primary purpose of the proposal grade is to give you feedback, not to nail you on your final grade. The proposal grade will count towards the DP1 grade, depending on how you respond to our feedback. If you get a C on the proposal but address all of the issues we raised in your final report, we won't count the proposal grade against you. Obviously, if you hand in the proposal late (or not at all), that will count against your DP1 grade.
Q: Are we allowed to change our design after the proposal is accepted?
A: Yes, especially if you come up with a better approach!
Q: What do you want us to turn in?
A: You should turn in two copies of your proposal. The proposal should be less than 800 words; we think this should be about two pages, so you should be handing us four pages total. Make sure there is room for us to write comments!
Q: What design goals should we focus on?
A: For the proposal, you should focus on fault isolation. Consider the mechanisms we've learned in class: client/server separation, RPC, address spaces, threads, pipes, and so on. Correctness is important; performance is not so important. A design that ignores all but one camera would not be a good design, because it would not be correct. But you don't want to worry about whether your design can handle ten thousand cameras on a 100Mhz Pentium.
Q: Do we need to worry about the transcoder or the AI module going into an infinite loop?
A: 6.033 won't really get into the techniques for handling unresponsive modules until the second half of the semester, so we won't hold it against you if you decide not to handle this case. In practice, an infinite loop would be less likely than an illegal instruction or wild memory operation, since pointer arithmetic errors are so common. However, if you want to learn more about how to approach this kind of problem, we recommend you read ahead in the class notes about timeouts. Section 4.E.2 is a good starting point; you might also look at pages 2-62 (thread_destroy) and 2-70.
Q: What do we really need to know about HTTP? Do we need to read the entire spec?
A: No. We needed a reference for HTTP 1.1, so we just put in the whole RFC. Don't worry about reading the entire spec; it's impossible. You just need to know the essence of how a GET request is processed. You may find sections 1.4, 4, 5, 6, and 9.3 useful as a starting point.
Q: Do figures count into the word limit?
A: No, but figures should illustrate your text, making it easier for us to understand. You shouldn't try to use figures as a replacement for text, to get around the word limit.
Q: Does the kernel have a preemptive scheduler?
A: Yes.
Q: Can multiple processes write to a single pipe simultaneously?
A: While it is possible for multiple processes to write to a single pipe simultaneously, it usually results in undesired behavior, because the data from each process can be interleaved arbitrarily. This means that if process A writes "aaa" and process B writes "bbb" to a pipe simultaneously, process C can read "ababab" from the read and of the pipe. This makes it hard to send any sort of intelligible message this way, unless there is some external synchronization mechanism for the writers. In practice, it's best to avoid this approach. You can achieve a similar effect by having the "reader" listen on an internal port and accept "network" connections from the "writers".
Q: Does the central computer need to analyze 100% of the incoming data from the cameras?
A: Under normal conditions, the hardware (network and CPU) can keep up with the load from all cameras, so yes, it should. However, if some or all cameras start sending motion-intensive data which bogs down the processing modules, the CPU may not be able to keep up. In this case, the central computer will have to drop some of the incoming data to shed load. How you do this is up to you. However, consider the design scenario: is it better to drop every other second of video from every camera, or better to drop all the video from half the cameras?
//
Last updated $Date: 2004/03/16 00:14:54 $ by $Author: ctl $