MIT Humans and Automation Lab
Generalizable classes and selection methodologies for human supervisory control metrics
Measuring multiple human-computer system aspects, such as the situational
awareness of the human, can be valuable in diagnosing performance successes
and failures, and identifying effective training and design interventions.
However, choosing an efficient set of metrics for a given human supervisory
control experiment is a challenge. This research aims to develop a principled
approach to evaluate and select the most efficient set of metrics among the
large number of available metrics. As part of this effort, a taxonomy of human
supervisory metrics have been developed, followed by the identification of
metric evaluation criteria that can help determine the quality of a metric
in terms of experimental constraints, comprehensive understanding, construct
validity, statistical efficiency, and measurement technique efficiency. Future
research will build on these evaluation criteria and the generic metric classes
to develop a cost-benefit analysis approach that can be used for metric selection.
Donmez, B., Pina, P. E., & Cummings, M. L. Evaluation criteria for human-automation performance metrics, 2008, In Proceedings of the Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD.
P.E. Pina, M. L. Cummings, J. W. Crandall and M. Della Penna. Identifying Generalizable Metric Classes to Evaluate Human-Robot Teams, Accepted, Metrics for Human-Robot Interaction Workshop at the 3rd Annual Conference on Human-Robot Interaction, Amsterdam, The Nederlands, 2008.
Pina, P., Donmez, B., Cummings, M.L., Selecting Metrics to Evaluate Human Supervisory Control Applications, (HAL2008-04), MIT Humans and Automation Laboratory, Cambridge, MA. (2008)
To complete a high level system acquisition decision, decision makers must process
a large amount of data to determine which system best fits a given mission
or purpose. This project investigates what type of decision support tool could
provide decision makers the greatest understanding of the system acquisition
trade space, thus allowing them to make more informed decisions. As part of
this project, a new configural display named Fan Visualization (FanVis) has
been conceived, designed and developed. FanVis takes the system acquisition
trade space data, and by using emergent features, naturally maps the data for
the decision maker. An experiment has shown improvement in performance using
FanVis over a set of Excel bar and line charts, which represent the tools
used currently in system acquisition decisions.
Related paper:
Massie, A. E., (2009), Designing a Graphical Decision Support Tool to Improve System Acquisition Decision, S. M. Thesis, MIT Aeronautics and Astronautics, Cambridge, MA.
Air Force Office of Scientific Research
