Project Plan

Overview

An overview of the project plan is provided below:

	Online Subject Evaluation and WTW Improvements	Policy and Process Issues
Phase 1 FY07	Discovery	Identify issues
Phase 2 FY08	WTW improvements WTW/online subject evaluation Beta pilot with four participating departments	Create governance and prioritize requirements
Phase 3 FY09	Production pilot with increased department participation WTW/OSE feature development and refinement	Implement new policies
Phase 4 FY10	Implement integrated production system and get all departments on board Integrate SIS Vision Project	Revise, maintain

 

Schedule

Details of the schedule are as follows:

PHASE 1: DISCOVERY (FY07) COMPLETE

Policy issues were identified and recommendations made.

PHASE 2: BETA PILOT (FY08) COMPLETE

Four departments — Physics, Chemical Engineering, Literature, and Philosophy — participated in the online subject evaluation (OSE) and Who's Teaching What (WTW) beta pilot in spring 2008. All other departments, as well as non-pilot subjects within the pilot departments, continued to use the paper system in FY08.

The questions on the online forms are identical to the current paper forms during the pilot phase, in order to minimize variables when we assess its effectiveness. There will also be the ability to add questions.

An advisory committee was formed to address policy and business process issues in advance of wider adoption.

PHASE 3: ADOPTION AS A PRODUCTION PILOT (FY09) IN PROGRESS

The new system, with added features, is currently being made available as a production pilot to interested departments, although the paper forms will still be available.

FALL 2008 (COMPLETE):

• ran an end-of-term pilot with 9 departments, 53 subjects
• completed unfinished functionality from FY08
• implemented features that enable Institute-wide scalability without developer intervention
• improved reports for students, faculty, and departments
• improved instructor selection on survey UI
• implemented automated email reminders to students by department
• implemented new policy decisions
• finished assessment of Spring 2008 pilot

SPRING 2009 (COMPLETE):

• ran an end-of-term pilot with 16 departments, approximately 250 subjects
• implemented new features: survey, HTML reports; survey search screen

FALL 2009:

• run an end-of-term pilot with 27 departments and approximately 450 subjects
• implement new WTW features: question management, cluster subjects

PHASE 4: FULL ROLLOUT (FY10)

System enhancements. Get all departments on board. Integrate with Student Vision Project and other MIT systems. Revise and maintain policies. Phase out paper-based subject evaluations.

(top of page)

---

 

Who's Teaching What

The improved WTW system will enable teaching data to be entered, reviewed, uploaded, automatically received from core systems, and immediately reflected in the online subject evaluation engine. Teaching data consists of subject, teacher, and student data, links of teachers to sections within a subject, and links of students to those sections.

The structure of subjects in WTW will reflect what's actually happening within a subject — who's teaching lectures, recitations, labs, etc. and who is enrolled in each section. Department administrators will continue to be the primary users of WTW.

WTW takes into account not only the format of sections within subjects (i.e., lecture, recitation, lab), but also the roles that teachers play within those sections. A teacher can have more than one role within a subject, e.g. lecturer and recitation instructor. If primary instruction occurs in a section other than a lecture — e.g., there are only recitations, not lectures — the results can be reported separately.

The system will be able to accommodate configurations that currently can be created only by manual intervention, such as sections of a single subject that are independent of each other and function as separate subjects. It will also be easy to see links between joint and "meets with" subjects.

WTW users will have considerable flexibility in working with data. In 2008-09, they'll be able to:

• Work with all types of data (student, subject, section, teacher)
• Enter data via an upload of a tab-delimited text file, if it's in the proper format
• Define custom section format types and teaching roles
• In the case of multiple teachers, tell the system to make the names appear in a particular order on evaluations and reports
• Filter data and edit the filtered set
• Assign students and teachers to sections of a subject
• Make mass changes to data
• Use a quick-entry system which displays a list of teacher names that begin with the text already entered
• Select subjects and teachers to be evaluated
• Group subjects and sections for evaluation and reporting purposes
• Pick the survey questions to appear on evaluations for a subject
• Run reports on subjects and teachers

 

There will be checks to make sure that entered data is valid. There will be a status indicator to tell whether a survey has gone through the proper validation and review, and if it's ready to be posted online.

(top of page)

---

Subject Evaluation

The evaluation process in 2008-09 will be as follows:

Data from Who's Teaching What is integrated into the subject evaluation system.

• user information – Kerberos ID, name (for user authentication purposes)
• user roles as pertains to subject evaluation – e.g., who can evaluate, who can see what reports
• subject and section structure – e.g. subject number, department, section format
• teaching assignments – subject number, Kerberos ID, section, role
• section assignments – students assigned to sections of a subject (optional)
• survey questions
• survey availability window

Students are prompted to complete surveys.

When surveys are opened, the system sends an email to each student who is expected to complete a survey. The email directs the student to web.mit.edu/subjectevaluation, where they can read instructions. From there, they click to access the survey, which is personalized for them through certificate authentication. The personalized site contains links to the survey and shows them:

• Their subjects for which they have not yet completed surveys
• Their subjects for which they have completed surveys

Students complete surveys.

1. They first select a subject to evaluate.
2. They pick instructor(s) to evaluate. If they have been assigned to particular sections of a subject, they can only choose from the instructors of those sections.
3. They answer questions about their chosen instructor(s) (in the Quality of Teaching category), about the subject as a whole, and about the section, as well as any specific questions that were submitted by the department.
4. They submit the evaluation and that subject is flagged as Completed.

Reminders are sent to non-respondents about surveys they didn't complete.

Response rates are monitored by the Office of Faculty Support and communicated to departments and instructors.

Survey results are presented to department administrators and individual instructors.

Once the instructors have submitted their grades for the subject, anonymous results are presented to departments and instructors in an HTML report. These results are both quantitative and qualitative and include open-ended comments about the instructor, subject and section.

Survey results are presented to students and other members of the MIT community.

Students and others with MIT certificates will continue to be able to view quantitative results (no open-ended comments) by subject on the web. Beginning in 2009, they will also have the capability to search for any subject evaluation result that has been published online for the MIT community, regardless of where the evaluation was offered (departmental online evaluation systems as well as Institute-wide paper and online systems).

(top of page)

 

 

A Joint Project of the Office of the Dean for Undergraduate Education (DUE) and Information Services & Technology (IS&T)