Background

The project was undertaken to improve teaching and learning at MIT. Oversight groups such as the Task Force on the Undergraduate Educational Commons and the Visiting Committee of the Office of the Dean for Undergraduate Education have stressed that assessment should be an Institute policy.

A Working Group on the Collection of Teaching Data was established in 2006 by Provost Rafael Reif and Dean for Undergraduate Education Daniel Hastings. It reported that MIT, like other universities, is under increased external pressure to report data and outcomes; but even without this pressure, having a more complete picture of the state of our education — including understanding how well students are learning, what contributes to that learning, and what may hinder it — can only benefit us. However, the limitations of the existing central paper-based subject evaluation system and the process of collecting teaching data made it difficult, if not impossible, to present timely and accurate feedback on the MIT educational experience.

The subject evaluation system was very slow and labor-intensive, and its database technology was old and in need of an upgrade. Survey forms were inflexible (questions needed to fit the form layout, and no additional open-ended questions could be asked), only three instructors for any single subject could be evaluated, and students who missed the class on the day which the forms were distributed did not have the chance to submit an evaluation.

A critical element of the subject evaluation process is the data on who is teaching what subject and what section within the subject, and who is enrolled in which section. In the old version of Who's Teaching What (WTW) used for collecting this data, it was not always possible to determine who is providing instruction to which students. There was no way to distinguish alternative teaching models (e.g., multiple faculty co-teaching one subject as distinct from different faculty teaching separate sections of the same subject). For those departments that kept their teaching data in an electronic system, there was no interface between their systems and WTW. There were no summary reports on teaching data by department, and no required core data.

The Working Group on the Collection of Teaching Data made recommendations to improve MIT's practices which became the charter for the project team:

• Convert the central subject evaluation system from paper to online, with all departments able (but not required) to use the system and add subject-specific questions
• Redesign subject evaluation and the Who’s Teaching What interface in parallel
• Have a core set of questions for all MIT subject evaluation systems
• Develop a method to collect or link core data from independent sources to facilitate analysis at the school/Institute level and to provide single access for students to view reports

The project team in turn identified issues and in 2007 made initial recommendations for implementation of an improved Who's Teaching What application and an Online Subject Evaluation system.

Policy and business process issues:

• What teaching data should be collected/standardized? When, how, and by whom?
• How long should the survey be? What questions? Who is eligible? Incentives?
• Who can see/query what results? Where do comments go? Rules for editing/suppressing? Where to post results?

Technical issues:

• Teaching data: Web services? Integration with MIT systems? Upload/download?
• What survey technology:open/community source, hosted, commercial, homegrown?
• Publishing results: Content management? Reporting tools? Authorization? Integration?

Initial recommendations:

• Implement online subject evaluation/Who’s Teaching What through a multi-year pilot project beginning in 2007-08 in order to provide the time required to articulate and prioritize the multiple policy and process issues that must be decided before expansion of the system
• Combine MIT-built modules with the appropriate “malleable” evaluation technology from a short list of commercial, open source, and homegrown solutions in use at peer institutions so that a pilot could be implemented quickly; do not build from scratch
• Create and initiate a governance structure and process by early fall 2007 to set policy, determine priorities, and articulate critical success criteria
• Use the current Institute paper survey tools and reports
• Collect teaching data at the section, rather than subject, level
• Do not offer incentives for completing the survey, either positive (like a raffle) or negative (like withholding grades)
• Allow students to see quantitative, but not qualitative, results