Education Report
Concept Questions
Watch on TechTV

Bringing Interactive Learning to 8.012 Lectures

an MIT Alumni Class Funds Project

In 2008, I was awarded funding from the MIT Alumni Classes of 1951, 1955, 1972 and 1999 Funds to incorporate interactive teaching methods into the first-year MIT introductory physics course, 8.012 Physics I (classical mechanics). This advanced, calculus-based course, part of the MIT curriculum for nearly 40 years, is designed for the brightest and most engaged first-year students at MIT who score sufficiently high on the Physics placement exam and/or AP Physics exam to qualify for enrollment. This course is also commonly taught by junior Physics faculty as part of the department's demonstration teaching requirements. As junior faculty (including myself) frequently come to MIT with little or no teaching experience, and are under concurrent pressure to build up their research groups, apply for grants, and generally meet expectations for tenure, it is no surprise that we tend to revert to traditional and familiar lecture-style formats. It was therefore my goal to introduce interactive teaching methods to 8.012 that could function in a lecture environment and be easily adopted by future instructors.

What was the implementation?

A total of 30 new concepttests were designed, with the aim of challenging the conceptual understanding while fostering discussion in the classroom. Questions were designed following two methodologies: (1) 3-question sequences of increasing complexity and (2) single challenge questions of sufficient difficulty to trigger disagreement and discussion. In a typical discussion sequence, the students are polled and if the results show near-unanimity for the correct response (say >65-75%) then the lecture moves on to the next topic. If there is a broader mix, then students are instructed to "convince their neighbors" of their solution by explaining their physical reasoning. The students are then re-polled to see if there is a change in the overall consensus. If a disparity remains, additional tactics such as advocacy (having individual students advocate for their given answers), ruling out incorrect options (if votes are split between three or more choices) and as a last resort guided discussion (steering students to the appropriate concepts) are used. The students by and large arrived at the correct conceptual idea with little or no input from the me.

To gauge the efficacy of the program, students took the Mechanics Baseline Test (MBT) at the beginning and end of the semester. I also tracked test scores through the semester and compared to participation in the lecture discussions.

What were the results?

One of the objectives of the interactive learning was to engage the students in participating in lecture discussions, by making participation a small part of their final grade (2.5%). Traditional lecture attendance has been known to taper off to <50% in previous semester. As shown in the figure at left, the participation rate through the clickers dropped to roughly this number by mid-semester, reflecting either comparable levels of lecture attendance or loss of/forgetting clickers. In any case, it appears that the use of the clickers, along with grade enticement, had little effect in keeping attendance much above prior levels.

As described in the report, the change in student's responses after first and final polling showed a mix of learning results, ranging from strongly positive (students initially answering incorrectly changing to a correct answer) to modestly negative (students initially answering correctly changing to an incorrect answer). In general, the negative conversions coincided with questions perceived to be "tricky", and hence probably poorly constructed. However, 10/11 of the concept questions which required re-polling (all others had nearly correct on first response) showed a net gain in correct responses, including two cases where the incorrect answer was initially more popular. On average students overcame a majority in favor of an incorrect answer on the first poll to a majority in favor of the correct answer on the second poll, and in no case was an initially popular correct answer overturned. These statistics suggest that the students' peer discussions were reasonably effective in guiding their conceptual learning.

Comparison of MBT test scores were mixed, with average correct answers (out of 26) changing from 19.3 to 20.5 (median scores changed from 20 to 21). This corresponds to a normalized gain statistic (; Crouch & Mazur 2001) of only 0.17, comparable to the gains realized in traditional lectures and much below the = 0.48 found in other peer-instruction courses. This results suggests that the interactive elements resulted in little improvement in conceptual learning; however, it is essential to take into account the fact that the pre-class MBT scores (74%) were comparable to post-class scores for students in other calculus-based, peer instruction courses (71-79%). The MBT may simply not provide a sufficiently challenging benchmark for measuring the conceptual gain of a typical 8.012 student.

Did participation in the concept questions correlate with other proficiency metrics, such as performance on exams? The figure to the left indicates that the correlation is weak. Average exam scores compared to lecture participation (left) shows a very weak inverse correlation (Pearson rank coefficient r = -0.10). Improvement in exam performance (3rd exam score minus 1st exam score) shows a slightly more significant positive correlation with participation (r = 0.17), with the students showing the largest improvement in exam scores participating the most. Yet the mean change in exam score was essentially identical between frequent (-2±16%) and infrequent participants (-9%±9%). Hence, there is a suggestion of exam performance tracking with participation in in-class concept discussions, but the correlation is weak.

What does it all mean?

Based on the statistical results, student feedback and personal experience in class, it is clear that concept questions are an invaluable and easily adoptable addition to the traditional lecture format. One to two questions per lecture allows for rapid assessment of the students' understanding of key concepts and facilitates peer instruction and learning. However, it is also clear from the analysis of conversion rates that concept questions alone are not sufficient for conceptual learning. The rare occasions when peer discussion fails to correct, and even obfuscates, basic physical concepts indicates that focused lecture and recitation periods are still required. Such problems occur especially when concept questions are misleading, and these can exasperate students already challenged by a difficult topic. That said, the frequency with which the students converged on correct ideas during the peer discussions, their retention of concepts that they themselves "discovered", and the ability to recognize gross misconceptions among large groups of students make concept questions an effective tool in the lecture environment.

The PRS technology employed for this investigation satisfied its objectives, was broadly adopted by the students, and provided quantitative data for assessment. However, the low PRS participation rates in the second half of the semester and student concerns over cost and misplacement of the RF transmitters suggests that an alternative to "clickers" should be explored. Many programs are now incorporating colored flash cards for voting (e.g., Meltzer & Manivannan 1996) that are cheap to print and distribute, easy to store in notebooks or textbooks, and trivial to replace. Paper flash cards still provide the ability for anonymous voting (e.g., by holding the card close to the body) and immediate instructor feedback, although they do not (in their current implementation) provide real-time feedback for the students nor attendance and other quantitative data for evaluation.

Finally, while the short-term metrics for concept learning - conversion rates in re-polling - suggest that this study was modestly successful in improving students' understanding of classical physics concepts, the long-term metrics - correlation between exam to PRS participation and the MBT exam - were ambiguous. In the case of the MBT exam, this is likely due to the high level of proficiency with which the 8.012 students began the semester, making large improvements unlikely. The lack of comparable statistics from prior semesters of instruction also prevents an assessment as to whether the concept questions specifically improved learning for 8.012 students over traditional lecture formats. The insensitivity of the MBT for this population, the more advanced of the two Physics baseline assessment tools currently employed, suggests that an even more advanced test may be needed. Perhaps the questions developed as part of this program may seed this effort.

Adam J. Burgasser
University of California San Diego
Center for Astrophysics and Space Science
9500 Gilman Drive, Mail Code 0424
La Jolla, CA 92093, USA

tel: +1 (858) 822 6958
fax: +1 (858) 534 2294

aburgasser [at] ucsd [dot] edu