Online Subject Evaluation/
Who's Teaching What
Academic Year 2007-08: Beta Pilot
- Who’s Teaching What developed
- Online subject evaluation pilot, Spring 2008: 22 subjects in 4 departments; external vendor used for survey and instructor reports
- Advisory committee formed to address policy and business process issues in advance of wider adoption
- Steering committee formed
Because data from Who's Teaching What would be used to set up subject evaluations in the Spring 2008 online pilot, there was time to make only modest changes to the existing WTW application; it would then be open to revision the following year based on user feedback. Changes made in Spring 2008 included tracking teaching data by section, rather than subject, and the ability to create custom department-specific section format types (e.g., Workshop) and teaching roles (e.g., Conductor). Teachers could now be added to WTW via a type-ahead list from which the user could select anyone currently part of the MIT community.
An external vendor for the online evaluations was chosen after extensive review. They conducted the evaluation pilot and produced summary reports for instructors and department administrators. Because of their reporting complexity, no joint or meets-with subjects were included in the pilot. Only students who were both registered for credit and had a valid Kerberos ID could participate.
Student participants were asked to give general comments about the online evaluation process. They largely preferred online to paper, but were concerned that their fellow students might not complete the online surveys. They felt many of the questions were irrelevant or confusing. They suggested some UI improvements, including a Save button so that they could return and make edits.
Academic Year 2008-09: Adoption as a Production Pilot
- Online subject evaluation pilot, Fall 2008: 53 subjects in 9 departments
- Online subject evaluation pilot, Spring 2009: 226 subjects in 16 departments; moved survey and reporting from vendor to in-house
While it had been necessary to outsource the evaluation initially because of time constraints, by Fall of 2008 it became clear that the vendor could not meet MIT's needs around survey configuration or reporting. By Spring 2009, both the evaluations and the summary reports had been moved in-house. Departments could now submit extra questions about specific instructors or sections as well as subjects, and cluster subjects (joint and meets-with) could be configured and reported correctly. Instructors and administrators could monitor response rates in real-time. A search engine was developed so that students and others in the MIT community could find reports from both the paper-based and online evaluations.
The evaluations were extended through reading period, which increased the response rate modestly from the Spring 2008 pilot.
Academic Year 2009-10: Full Rollout
- Online subject evaluation pilot, Fall 2009: 457 subjects in 27 departments
- Online subject evaluation pilot, Spring 2010: 655 subjects in 34 departments
Much of the development effort in AY2009-10 focused on improving Who's Teaching What, online evaluations, and reports so that they were fully scalable and sustainable without developer intervention. Evaluations for half-term subjects were introduced in Fall 2009.
As of Academic Year 2010-11, MIT is no longer offering paper-based subject evaluations, and the old version of Who’s Teaching What is no longer in use. Development of both new and originally-requested features is complete as of Spring 2011.