EXECUTIVE SUMMARY 

LESSONS LEARNED

Findings from Ten Formative Assessments
of Educational Initiatives at MIT (2000-2003)

Educational Innovation at MIT

In 1999, MIT received two generous grants that allowed it to embark on a wide scale series of innovations in undergraduate education. The first was from then chairman of the MIT Corporation, Alex d'Arbeloff, and his wife, Brit d'Arbeloff; they created the d'Arbeloff Fund for Excellence in Education. The d'Arbeloff grants have been devoted primarily to strengthening the first-year experience at MIT. The second grant, from the Microsoft Corporation, funded iCampus, a five-year, $25 million research alliance whose purpose is to improve higher education through the use of information technology. Since 1999, MIT faculty, staff, and students have undertaken approximately forty experiments in educational innovation supported by these two sources of funding.

The Teaching and Learning Laboratory (TLL) was asked to manage the assessment of these initiatives. Of course, the Institute has evaluated its educational efforts throughout its history, but it wanted these new initiatives to be studied in a more systematic way. We have undertaken that work over the last four years in collaboration with MIT faculty, administrators, students, and assessment and evaluation consultants. Of the ten research projects undertaken, six have been completed and four are in their second or third years.

Types of Findings

Following the lead of the newly formed Center for the Advancement of Scholarship on Engineering Education (CASEE), we have grouped these ten projects into "strands."  A strand is a line of inquiry that a number of individual projects can contribute to. Although itis difficult to categorize ten distinct projects, as we have reviewed them over the last several months, we have come to see they can be placed in one of two strands: (1) those that used active learning pedagogies; and (2) those that focused on educational technology. (Appendix A to the report provides a description of each individual project.)

This report, then, summarizes the most important findings from the educational initiatives MIT has undertaken over the last several years. It should be noted that we have not described every finding for every project; we are only reporting the findings that we believe are the most striking, and that have the most relevance for undergraduate education in science, engineering, and technology. We should also make clear that the initiatives listed in Appendix A do not encompass all of the activities that are being carried out at the Institute to strengthen undergraduate education. A number of other initiatives are currently underway, and several others are in the planning stage.

As we have done this work over the last four years, we have also identified what we believe are several "best practices" for the design, implementation, and assessment of reforms in teaching and learning in higher education; the report includes a description of these.

The major findings in each category, then, are as follows.

Findings Related to the Use of Active Learning

• The use of active learning pedagogies resulted in increased learning gains in two courses that specifically measured learning.
  
  
• Students need to be prepared and instructors need to be trained for the change from lecture-based classes to those that employ active learning pedagogies.
  
  
• Preliminary finding:  Some elements of active learning may be more appropriate for some students than for others, and better for some cognitive tasks than for others.

Findings Related to the Use of Educational Technology

• The most successful educational technologies have met a specific instructional need that has been unmet or poorly met by traditional media.
  
  
• Too much technology can be detrimental.
  
  
• There are important relationships between educational technology and the learning environments in which they operate:  (a) the use of educational technology has been most effective when there are strong connections between the technology, the learning goals, and pedagogical methods; (b) the same technology will have differential effectiveness depending on the educational context within which it is embedded; (c) educational technology exerts its impact by changing the properties of information in the learning environment.

Findings Related to Design, Implementation, and Assessment

• Design: Educational innovation can be thought of as a design problem as innovators wish to improve upon instructional practice but must work within a set of constraints.  Best practices to accomplish this goal include:  (a) formulate objectives in terms of learning outcomes; (b) research what is already known in relation to the innovation; (c) identify constraints; (d) plan for the pull of the status quo.
  
  
• Implementation: Educational innovation is an iterative process.
  
  
• Assessment: (a) Differences between research in the "hard" and "soft" sciences need to be made explicit in order to aid faculty in understanding what educational assessment can and cannot accomplish; (b) the full range of assessment methodologies should be used because the educational environment is "noisy"; (c) assessment should be formative in line with the philosophy that educational innovation is an iterative process.

Next Steps

We believe the studies we have done over the past several years have set the stage for a second phase of research at MIT into pedagogical innovation, educational technology, and how improvements in those two areas impact learning. The conclusion to this report lays out the priorities for research and assessment as we move forward.