By J. H. Saltzer, February 7, 1996

• Nibble around the edges first. Read the abstract, the first paragraph or two of the paper (because the abstract often isn't helpful) the last paragraph or two. Look at the section headings and the figures. Generally, try to figure out what, at a high level, is the primary point of the paper.

• Don't be alarmed by non-standard terminology. It is standard to be non-standard.

• Is the writer convincing? Are the assumptions reasonable, the arguments sound, and the claims supported? Are there non-sequiturs or fallacious lines of reasoning? Be skeptical but look for useful gems; many papers fail one or more of these tests, yet still contain ideas of considerable value.

• For papers that describe system designs, was the design actually implemented, or is this just a proposal for how other people should do it? If it was implemented, was it actually placed in service or was it just a toy system that never got outside the laboratory? This consideration is key to deciding how much significance to apply to the author's claim that the ideas are worthwhile.

• Who wrote it? What qualifications or special professional expertise or experience does the author have? Are those qualifications relevant to the topic of *this* paper?

• Who were the author's collaborators, contacts, sources, and advisors? Did the author work in a vacuum? Or surrounded by competent colleagues? Do the citations suggest that the author is familiar with the field, or does the paper cite only the author's own work? Do the acknowledgements offer any clues?

• Why did the author write this paper? Was it just a ticket to a workshop in Rio? One more paper for a tenure resume? Or was there something really worth reporting?

• Where was the paper published? Some journals and conferences are refereed, some are not. Is the place of publication particularly prestigious? Is the particular department this paper was found in edited to the same standard as the rest of the journal? Some conferences are very selective, others will publish anything. Some authors take internal memos and simply submit them as papers. (Unfortunately, sometimes nothing else ever gets published on the design in question.)

• What is the author's affiliation? Is his institution known for quality work or an interesting collection of resources or a particularly intellectual environment?

• When was the paper published, and when was the work performed? Is this a recent paper (recent papers will have the latest hot ideas, but may be a flash in the pan) or has it been around a long time (old papers, especially in computer science, may be quite obsolete, but they may also have stood the test of time.)