Generalizable classes and selection methodologies for human supervisory control metrics
Measuring multiple human-computer system aspects, such as the situational awareness of the human, can be valuable in diagnosing performance successes and failures, and identifying effective training and design interventions. However, choosing an efficient set of metrics for a given human supervisory control experiment is a challenge. This research aims to develop a principled approach to evaluate and select the most efficient set of metrics among the large number of available metrics. As part of this effort, a taxonomy of human supervisory metrics have been developed, followed by the identification of metric evaluation criteria that can help determine the quality of a metric in terms of experimental constraints, comprehensive understanding, construct validity, statistical efficiency, and measurement technique efficiency. Future research will build on these evaluation criteria and the generic metric classes to develop a cost-benefit analysis approach that can be used for metric selection.
Donmez, B., Pina, P. E., & Cummings, M. L. Evaluation criteria for human-automation performance metrics, 2008, In Proceedings of the Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD.
P.E. Pina, M. L. Cummings, J. W. Crandall and M. Della Penna. Identifying Generalizable Metric Classes to Evaluate Human-Robot Teams, Accepted, Metrics for Human-Robot Interaction Workshop at the 3rd Annual Conference on Human-Robot Interaction, Amsterdam, The Nederlands, 2008.
Pina, P., Donmez, B., Cummings, M.L., Selecting Metrics to Evaluate Human Supervisory Control Applications, (HAL2008-04), MIT Humans and Automation Laboratory, Cambridge, MA. (2008)
To complete a high level system acquisition decision, decision makers must process a large amount of data to determine which system best fits a given mission or purpose. This project investigates what type of decision support tool could provide decision makers the greatest understanding of the system acquisition trade space, thus allowing them to make more informed decisions. As part of this project, a new configural display named Fan Visualization (FanVis) has been conceived, designed and developed. FanVis takes the system acquisition trade space data, and by using emergent features, naturally maps the data for the decision maker. An experiment has shown improvement in performance using FanVis over a set of Excel bar and line charts, which represent the tools used currently in system acquisition decisions.
Massie, A. E., (2009), Designing a Graphical Decision Support Tool to Improve System Acquisition Decision, S. M. Thesis, MIT Aeronautics and Astronautics, Cambridge, MA.