Background

The educational imperative

Advancing from teaching to learning in our classrooms is one of the strategic themes of the Office of the Dean for Undergraduate Education. Making this paradigm shift will require accurate and timely feedback on subjects, pedagogic approaches, and teaching staff. The Task Force on the Undergraduate Educational Commons has recommended that assessment be made an Institute policy, including:

1. Improving the breadth of coverage and the usefulness of end-of-term class evaluations.
2. Encouraging a feedback cycle between students and faculty throughout the term.
3. Assessing the curriculum as well as the teaching.

It is against this set of objectives that the OSE/WTW project has been launched.

(top of page)

Current subject evaluation system

Since 1997, Institute-wide subject evaluation has been conducted via two paper forms — one with 24 questions for Science and Engineering subjects and one with 28 questions for HASS subjects. The questions are on a 1-7 Likert scale (measure of agreement or disagreement with a statement). Between 700-750 subjects are evaluated each term — approximately 40% of all that are offered.

The current process works like this:

1. Departments enter teaching data for subjects into Who’s Teaching What (WTW), an online interface housed in MITSIS used primarily for subject evaluation purposes. The department administrator indicates whether or not the subject is to be evaluated.
2. The Office of Faculty Support (OFS) downloads WTW data from the Data Warehouse into a Filemaker database, verifies its accuracy, and prints forms with instructors’ names.
3. The forms are distributed to the departments.
4. Departments deliver forms to instructors.
5. Evaluations are held during the last 10 days of classes. Students are given the paper forms to fill out during class.
6. Forms are returned to OFS, sorted by hand, and sent to Document Services in the Libraries to be scanned.
7. Form images are processed with ReadSoft Eyes and Hands software. Reports are produced as hard copy to departments and updated on the student subject evaluation website.
8. Paper forms are returned to departments for them to type or scan open-ended comments and distribute as they wish.

This system has many limitations:

• It’s currently at capacity — very labor-intensive.
• The subject evaluation database technology is old and needs to be upgraded.
• Forms are inflexible (questions must fit form layout).
• Only three instructors for any single subject can be evaluated.
• Students who miss the class on the day which the forms are distributed do not have the chance to submit an evaluation.
• It takes 6 weeks to turn around reports to departments and post results online.

(top of page)

Current collection of teaching data

A critical element of the subject evaluation process is the data on who is teaching what and who is enrolled in which section of a subject. At MIT, this data is not obtained easily, if at all. The Registrar’s Office doesn’t keep information on sections, and departments have their own methods for capturing this data (sometimes, only the instructor and the students know what’s happening). The Who’s Teaching What interface (WTW) was developed to capture teaching data for the primary purpose of subject evaluation, but only about 40% of subjects use it each term.

WTW also has its share of problems:

• There’s only one way to enter teaching data — department administrators enter subjects one at a time, by hand.
• For those departments which keep their teaching data in an electronic system, there’s no interface between their systems and WTW.
• No summary reports on teaching data by department are available.
• There is no way to distinguish alternative teaching models (e.g., multiple faculty co-teaching one subject).
• There are no required core data.

(top of page)

2006: Formation of the Working Group on the Collection of Teaching Data

In 2006, Provost Rafael Reif and Dean for Undergraduate Education Daniel Hastings established a working group to examine in detail the current practices for collecting information on Institute teaching. MIT, like other universities, is under increased external pressure to report data and outcomes; but even without this pressure, having a more complete picture of the state of our education — including understanding how well students are learning, what contributes to that learning, and what may hinder it — can only benefit us. Oversight groups such as the Task Force on the Undergraduate Educational Commons and the Visiting Committee of the Office of the Dean for Undergraduate Education have stressed that assessment should be an Institute policy.

The working group was asked to examine current practices and make recommendations for the collection of data on a semester-by-semester basis related to who is teaching what, enrollment by sections, and subject evaluation. Their recommendations included:

• Convert the central subject evaluation system from paper to online, with all departments able (but not required) to use the system and add subject-specific questions;
• Redesign subject evaluation and the Who’s Teaching What interface in parallel;
• Have a core set of questions for all MIT subject evaluation systems;
• Develop a method to collect or link core data from independent sources to facilitate analysis at the school/Institute level and to provide single point of access for students to view reports.

(top of page)

2007: Formation of an Online Subject Evaluation/Who's Teaching What project team

The recommendations of the Teaching Data Collection Working Group become the charter for the Technical Analysis Group/Online Subject Evaluation team, which convened in January 2007 with the task of coming up with a technical plan to implement the recommendations. The team arrived at the following findings based on consultations with and input from members of the MIT community, people from other schools, and potential providers, both commercial and open source:

• The subject evaluation process at MIT is complex. It involves collecting teaching data, conducting the survey, and publishing the results — all of which vary by department. We now describe the "nature of the beast" as a three-humped camel:

The Subject Evaluation Beast
	Collecting Teaching Data	Conducting Survey	Publishing Results
Policy & Business Process	What data? When collected? How collected? How standardized? By whom?	How long? What questions? Include listeners? Give incentives?	Comments go where? Who can see what? Who can query what? Rules for editing? Rules for suppressing? Summarization levels? Post results to Stellar?
Technology	Web Service? Integration with MIT systems? - departments - HR - student records - scheduling - Stellar Upload/download?	Open/community source? Hosted service? Commercial package? Homegrown?	Content management? Reporting tools? Faculty activities system? Authorization? Integration with other MIT systems? Report generation?

• The market for software application products in this area is not yet mature, and MIT requirements are nascent and evolving.
• Our peer institutions, while moving online, have yet to implement university-wide systems.
• There is a strong demand in MIT departments for a central online subject evaluation system that integrates with MIT’s data infrastructure, provides for evaluation of teaching assistants (TAs), captures student comments, and reports results quickly.
• Policy and process issues are as great as, if not greater than, technical issues. The implementation of any subject evaluation solution will require these to be addressed; for example, should drops be allowed to evaluate instructors, what kinds of incentives are permissible, and how should qualitative comments be treated.
• The key technical challenges are expected to include evolving a web services model, integration with Stellar and other systems, upload/download of data from departmental systems, and aligning with the recommendations of the Vision project for the student information system.

Based on these findings, the Technical Analysis group recommended that:

• The online subject evaluation system should be implemented through a multi-year project, beginning in 2007-08 with improvements to Who's Teaching What and pilots of the online versions of the current Institute paper survey tool and reports.
• A governance structure and process should be created and initiated by early fall 2007 to set policy, determine priorities, and articulate critical success criteria.
• The tactical approach focuses on combining MIT-built modules with the appropriate “malleable” evaluation technology from a short list of commercial, open source, and homegrown solutions in use at peer institutions. This approach will permit a pilot to be implemented quickly and demonstrate progress toward the larger goals of improving the collection of teaching data and the assessment of teaching and learning. Building the system from scratch is not recommended since that would require significant resources and would delay implementation of a pilot to validate and clarify the requirements that have emerged during Phase 1 of this project.
• Limiting the pilot to online versions of the current paper forms and reports would provide the time required to articulate and prioritize the multiple policy and process issues that must be decided before expansion of the system — it is anticipated that this will require significantly more deliberation and consultation with faculty, students, and administrators than does the choice of technology.

(top of page)

NEXT: THE PLAN FOR IMPROVEMENT >>

 

A Joint Project of the Office of the Dean for Undergraduate Education (DUE) and Information Services & Technology (IS&T)