MIT Humans and Automation Lab
Generalizable classes and selection methodologies for human supervisory control metrics
Measuring multiple human-computer system aspects, such as the situational
awareness of the human, can be valuable in diagnosing performance successes
and failures, and identifying effective training and design interventions.
However, choosing an efficient set of metrics for a given human supervisory
control experiment is a challenge. This research aims to develop a principled
approach to evaluate and select the most efficient set of metrics among the
large number of available metrics. As part of this effort, a taxonomy of human
supervisory metrics have been developed, followed by the identification of
metric evaluation criteria that can help determine the quality of a metric
in terms of experimental constraints, comprehensive understanding, construct
validity, statistical efficiency, and measurement technique efficiency. Future
research will build on these evaluation criteria and the generic metric classes
to develop a cost-benefit analysis approach that can be used for metric selection.
Donmez, B., Pina, P. E., & Cummings, M. L. Evaluation criteria for human-automation performance metrics, 2008, In Proceedings of the Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD.
P.E. Pina, M. L. Cummings, J. W. Crandall and M. Della Penna. Identifying Generalizable Metric Classes to Evaluate Human-Robot Teams, Accepted, Metrics for Human-Robot Interaction Workshop at the 3rd Annual Conference on Human-Robot Interaction, Amsterdam, The Nederlands, 2008.
Pina, P., Donmez, B., Cummings, M.L., Selecting Metrics to Evaluate Human Supervisory Control Applications, (HAL2008-04), MIT Humans and Automation Laboratory, Cambridge, MA. (2008)
The "Tracking Resource Allocation Cognitive Strategies" tool
(TRACS) allows for post-hoc visualization of the cognitive steps exhibited
by a human operator while interacting with a multivariate resource allocation
decision-support interface. This tool was applied to both mission planning
for multi-criteria resource allocation for military strikes, and also multi-variable
geospatial path planning problems for astronaut moon traversals. Both domains
involve a human operator interacting with an automated decision-support system
in order to find a solution to a complex planning problem involving multivariate
and constrained optimization for a cost function. With the help of TRACS, clear
patterns of behavior were identified that could be correlated to performance
in both applications.
Related papers:
S. Bruni, Y. Boussemart, M.L. Cummings, and S. Haro . Visualizing Cognitive Strategies in Time-Critical Mission Replanning. In Proceedings of HSIS 2007: ASNE Human Systems Integration Symposium, March 19-21, 2007, Annapolis, MD, USA, 2007.
Bruni, S., Marquez, J., Brzezinski, A., & Cummings, M.L., Visualizing Operators’ Cognitive Strategies In Multivariate Optimization, Proceedings of HFES 2006: 50th Annual Meeting of the Human Factors and Ergonomic Society, San Francisco, CA, USA, October 16-20, 2006.
Bruni, S., & Cummings, M.L., Tracking Resource Allocation Cognitive Strategies for Strike Planning, COGIS 2006 - Cognitive Systems with Interactive Sensors, Paris, France, 2006.
The Cooper-Harper Scale is a 10 point quasi-subjective scale developed in
1957. The scale was developed to get standardized reporting from test pilots
on the controllability of the aircraft being evaluated. The focus of MCH-UVD
project is to develop a similar scale for evaluating unmanned vehicle displays.
The objective is to have a standard tool for troops in the field to identify
unmanned vehicle display deficiencies. Modified Cooper Harper Scale is intended
to serve as an instrument for the DoD to compare displays when acquiring
new software and to provide designers with further guidelines on what the
troops in the field need. An experiment is underway to validate this scale
using unmanned ground vehicle (UGV) and unmanned aerial vehicle (UAV) simulators,
in collaboration with the Robotics Institute at Carnegie Mellon University
and Aurora Flight Sciences Corporation.
Sponsored by U.S. Army Aberdeen Test Center
In order to increase its experimental capabilities, operational demonstration
abilities, and outreach efforts, the Humans and Automation Laboratory is
designing and building the Mobile Advanced Command and Control Station
(MACCS). This mobile unit, a large cargo van, will feature a multi-operator
replica of the U.S. Navy’s Multi-Modal WorkStation (MMWS). Alongside
this advanced computer equipment, this vehicle will include a reconfigurable
seating area to accommodate for a variety of experimental and meeting settings.
MACCS will be equipped with the latest satellite internet and positioning
technologies, allowing for realistic, joint operations research as well
as teaming scenarios.
Sponsored by the Office of Naval Research
HACT is an innovative taxonomy aimed at providing a new information-processing model for collaborative human-computer decision-making, defining specific collaboration roles (moderator, generator, and decider) and their characteristics, and representing collaboration in a direct-perception visualization. HACT can be both used to describe and compare command and control system(s) from a collaboration standpoint. Future research based on HACT will incorporate design trade-off characterizations in order to provide system designer with a cost-benefit analysis tool.
S. Bruni, J.J. Marquez, A. Brzezinski, C. Nehme and Y. Boussemart (2007). Introducing a Human-Automation Collaboration Taxonomy (HACT) in Command and Control Decision-Support Systems, Accepted, 12th International Command and Control Research and Technology Symposium, Newport, RI, June, 2007.
To complete a high level system acquisition decision, decision makers must process
a large amount of data to determine which system best fits a given mission
or purpose. This project investigates what type of decision support tool could
provide decision makers the greatest understanding of the system acquisition
trade space, thus allowing them to make more informed decisions. As part of
this project, a new configural display named Fan Visualization (FanVis) has
been conceived, designed and developed. FanVis takes the system acquisition
trade space data, and by using emergent features, naturally maps the data for
the decision maker. An experiment has shown improvement in performance using
FanVis over a set of Excel bar and line charts, which represent the tools
used currently in system acquisition decisions.
Related paper:
Massie, A. E., (2009), Designing a Graphical Decision Support Tool to Improve System Acquisition Decision, S. M. Thesis, MIT Aeronautics and Astronautics, Cambridge, MA.
Air Force Office of Scientific Research
HACT is an innovative taxonomy aimed at providing a new information-processing model for collaborative human-computer decision-making, defining specific collaboration roles (moderator, generator, and decider) and their characteristics, and representing collaboration in a direct-perception visualization. HACT can be both used to describe and compare command and control system(s) from a collaboration standpoint. Future research based on HACT will incorporate design trade-off characterizations in order to provide system designer with a cost-benefit analysis tool.
S. Bruni, J.J. Marquez, A. Brzezinski, C. Nehme and Y. Boussemart (2007). Introducing a Human-Automation Collaboration Taxonomy (HACT) in Command and Control Decision-Support Systems, Accepted, 12th International Command and Control Research and Technology Symposium, Newport, RI, June, 2007.
Sponsored by Boeing Phantom Works
