Doctoral Qualifying Exam

Statistical assessment of bias in the doctoral qualifying exam

"Do factors like gender, U.S./International status, having

studied or not studied at MIT for undergrad, or the rank

of your faculty endorser correlate with student performance

on the qualifying exam?"

In 2003 the doctoral qualifying exam results of the
department of Aeronautics and Astronautics there was a
preponderance of low scores by female candidates. Triggered
by this event, a professional statistician was hired to look
for evidence of bias in the exam results. Using 5 years of
partial data and 2 complete years, the statistician looked
for evidence of bias based on Gender, MIT/non-MIT undergrad,
U.S./International, and rank of faculty endorser. The tests
were nonparametric (median and Kruskal-Wallace) and were
designed to suppress subtleties, minimize assumptions and
detect only major differences.

The number of candidates in some of the sub-categories was
relatively small and the variance in performance amongst all
the candidates was relatively high. The statistically-
significant difference between the scores of men and women
in 2003 was found to originate from low written exam scores
for women despite GRE's that were (statistically) higher.
For the five year period represented by the data, several
examples of statistically-significant differences were
found. For instance in one year, candidates who did their
undergrad study outside MIT fared better than those who did
it at MIT.

However, there were no apparent trends year to year.
Further, taken as a whole, the number of statistically
significant differences was consistent with the variance in
the population. Therefore, the results do not suggest a
strong bias in performance related to any of the factors
that were assessed.

This analysis will continue every year and the results will
be communicated to the stakeholders.

If you would like more information, including a detailed
look at the statistical analyses, you are encouraged to
contact Professor Ian A. Waitz (iaw@mit.edu).