Research Projects

Generalizable classes and selection methodologies for human supervisory control metrics

MetricsMeasuring multiple human-computer system aspects, such as the situational awareness of the human, can be valuable in diagnosing performance successes and failures, and identifying effective training and design interventions. However, choosing an efficient set of metrics for a given human supervisory control experiment is a challenge. This research aims to develop a principled approach to evaluate and select the most efficient set of metrics among the large number of available metrics. As part of this effort, a taxonomy of human supervisory metrics have been developed, followed by the identification of metric evaluation criteria that can help determine the quality of a metric in terms of experimental constraints, comprehensive understanding, construct validity, statistical efficiency, and measurement technique efficiency. Future research will build on these evaluation criteria and the generic metric classes to develop a cost-benefit analysis approach that can be used for metric selection.

Donmez, B., Pina, P. E., & Cummings, M. L. Evaluation criteria for human-automation performance metrics, 2008, In Proceedings of the Performance Metrics for Intelligent Systems Workshop, Gaithersburg, MD.

P.E. Pina, M. L. Cummings, J. W. Crandall and M. Della Penna. Identifying Generalizable Metric Classes to Evaluate Human-Robot Teams, Accepted, Metrics for Human-Robot Interaction Workshop at the 3rd Annual Conference on Human-Robot Interaction, Amsterdam, The Nederlands, 2008.

Pina, P., Donmez, B., Cummings, M.L., Selecting Metrics to Evaluate Human Supervisory Control Applications, (HAL2008-04), MIT Humans and Automation Laboratory, Cambridge, MA. (2008)

Tracking Resource Allocation Cognitive Strategies (TRACS)

The "Tracking Resource Allocation Cognitive Strategies" tool (TRACS) allows for post-hoc visualization of the cognitive steps exhibited by a human operator while interacting with a multivariate resource allocation decision-support interface. This tool was applied to both mission planning for multi-criteria resource allocation for military strikes, and also multi-variable geospatial path planning problems for astronaut moon traversals. Both domains involve a human operator interacting with an automated decision-support system in order to find a solution to a complex planning problem involving multivariate and constrained optimization for a cost function. With the help of TRACS, clear patterns of behavior were identified that could be correlated to performance in both applications.

Related papers:

S. Bruni, Y. Boussemart, M.L. Cummings, and S. Haro . Visualizing Cognitive Strategies in Time-Critical Mission Replanning. In Proceedings of HSIS 2007: ASNE Human Systems Integration Symposium, March 19-21, 2007, Annapolis, MD, USA, 2007.

Bruni, S., Marquez, J., Brzezinski, A., & Cummings, M.L., Visualizing Operators’ Cognitive Strategies In Multivariate Optimization, Proceedings of HFES 2006: 50th Annual Meeting of the Human Factors and Ergonomic Society, San Francisco, CA, USA, October 16-20, 2006.

Bruni, S., & Cummings, M.L., Tracking Resource Allocation Cognitive Strategies for Strike Planning, COGIS 2006 - Cognitive Systems with Interactive Sensors, Paris, France, 2006.

Office of Naval Research

back to top

Modified Cooper-Harper for Unmanned Vehicle Displays (MCH-UVD)

Modified Cooper Harper for Unmanned Vehicle DisplaysThe Cooper-Harper Scale is a 10 point quasi-subjective scale developed in 1957. The scale was developed to get standardized reporting from test pilots on the controllability of the aircraft being evaluated. The focus of MCH-UVD project is to develop a similar scale for evaluating unmanned vehicle displays. The objective is to have a standard tool for troops in the field to identify unmanned vehicle display deficiencies. Modified Cooper Harper Scale is intended to serve as an instrument for the DoD to compare displays when acquiring new software and to provide designers with further guidelines on what the troops in the field need. An experiment is underway to validate this scale using unmanned ground vehicle (UGV) and unmanned aerial vehicle (UAV) simulators, in collaboration with the Robotics Institute at Carnegie Mellon University and Aurora Flight Sciences Corporation.

Sponsored by U.S. Army Aberdeen Test Center

back to top

Mobile Advanced Command and Control Station

Mobile Advanced Command and Control StationIn order to increase its experimental capabilities, operational demonstration abilities, and outreach efforts, the Humans and Automation Laboratory is designing and building the Mobile Advanced Command and Control Station (MACCS). This mobile unit, a large cargo van, will feature a multi-operator replica of the U.S. Navy’s Multi-Modal WorkStation (MMWS). Alongside this advanced computer equipment, this vehicle will include a reconfigurable seating area to accommodate for a variety of experimental and meeting settings. MACCS will be equipped with the latest satellite internet and positioning technologies, allowing for realistic, joint operations research as well as teaming scenarios.

Sponsored by the Office of Naval Research

back to top

Human-Automation Collaboration Taxonomy (HACT)

HACT is an innovative taxonomy aimed at providing a new information-processing model for collaborative human-computer decision-making, defining specific collaboration roles (moderator, generator, and decider) and their characteristics, and representing collaboration in a direct-perception visualization. HACT can be both used to describe and compare command and control system(s) from a collaboration standpoint. Future research based on HACT will incorporate design trade-off characterizations in order to provide system designer with a cost-benefit analysis tool.

S. Bruni, J.J. Marquez, A. Brzezinski, C. Nehme and Y. Boussemart (2007). Introducing a Human-Automation Collaboration Taxonomy (HACT) in Command and Control Decision-Support Systems, Accepted, 12th International Command and Control Research and Technology Symposium, Newport, RI, June, 2007.

Office of Naval Research

 

 

 

 

back to top

System Acquisition Decision Support (FanVis)

To complete a high level system acquisition decision, decision makers must process a large amount of data to determine which system best fits a given mission or purpose. This project investigates what type of decision support tool could provide decision makers the greatest understanding of the system acquisition trade space, thus allowing them to make more informed decisions. As part of this project, a new configural display named Fan Visualization (FanVis) has been conceived, designed and developed. FanVis takes the system acquisition trade space data, and by using emergent features, naturally maps the data for the decision maker. An experiment has shown improvement in performance using FanVis over a set of Excel bar and line charts, which represent the tools used currently in system acquisition decisions.

Related paper:

Massie, A. E., (2009), Designing a Graphical Decision Support Tool to Improve System Acquisition Decision, S. M. Thesis, MIT Aeronautics and Astronautics, Cambridge, MA.

Air Force Office of Scientific Research

back to top

Predictive Models of Supervisory Teams

2 undergraduate students working on the eyetrackergazepath over an interfaceHidden Markov Model of an operator controlling multiple unmanned vehicles HACT is an innovative taxonomy aimed at providing a new information-processing model for collaborative human-computer decision-making, defining specific collaboration roles (moderator, generator, and decider) and their characteristics, and representing collaboration in a direct-perception visualization. HACT can be both used to describe and compare command and control system(s) from a collaboration standpoint. Future research based on HACT will incorporate design trade-off characterizations in order to provide system designer with a cost-benefit analysis tool.

S. Bruni, J.J. Marquez, A. Brzezinski, C. Nehme and Y. Boussemart (2007). Introducing a Human-Automation Collaboration Taxonomy (HACT) in Command and Control Decision-Support Systems, Accepted, 12th International Command and Control Research and Technology Symposium, Newport, RI, June, 2007.

Sponsored by Boeing Phantom Works

back to top