I.A.2. Title of Proposal:

Assessment & Circumvention of Temporal Asynchronies  
in Weapon Handling Virtual Environments

 Thomas E. von Wiegand

I.B. Capable Manpower FNC Information

I.B.1. The proposed research has direct impact on the usability and appropriateness of VIRTE training of skills in which weapon handling is performed. Visual search coupled with weapon aiming is an essential component of the skills covered in Close Quarters Battle training. This combination of actions, for which it is desired to create in the user an engrained automatic behavior that does not degrade under stress, represents a particular challenge to current and forseeable VE systems due to limitations in tracking, display, and processing speed. System latencies and jitter are present in all VE systems, and are particularly troublesome in Head-Mounted Display (HMD) -based installations. Notwithstanding these potential limitations, there are many advantages of HMD-based systems, and to the extent that these temporal artifacts are well characterized and controlled, we can expect to achieve positive training transfer for infantry tasks using the planned VIRTE hardware configuration.

 

I.B.2. The creation of an immersive VE system is a challenging application of sensor, computation, and display technology. Although processor speed, system architecture and memory availability predictably improve with each new generation of products, and while sensors and displays continue to show modest improvement, in the course of the VIRTE program (and beyond the end of this decade) we will not be in a position to simply ignore the effects of system latencies and other systematic uncertainties. However, the technology has progressed to the point where it is reasonable to perform tasks in a realtime immersive VE that have a critical sensorimotor component, if the limitations are appropriately recognized and managed. In most cases this does not mean pouring money into systems in order to make a marginal improvement over COTS equipment, but instead, to "get smart" about how the system is actually used, and to determine which aspects of the training are most likely to suffer. Given this informed analysis, a set of tested guidelines can be provided for apportioning limited resources to match the proven needs of the task. Our technical approach to accomplish this goal will be to utilize observations of weapon handling behavior outside of the immersive VE system, and compare how this behavior degrades within VE presentations of varying fidelity. Using special apparatus that we have constructed during the precursor to this program ("Realism in VEs") we will be able to explore a parameter space that, for certain task components, is arbitrarily close to perfect system performance (zero latency and jitter). This approach, combined with various more conventioinal VE presentation measurements, will enable us to provide specific guidelines on how weapon handling skills and training transfer degrade as a function of temporal asynchronies present in VIRTE training hardware. These guidelines will enable VIRTE system designers to proactively apportion computational resources to minimize degradation of training effectiveness.

 

I.C. Technical Information

I.C.1 Requirement / Problem / Deficiency

The operational deficiency is that training of tasks such as those which involve the execution of coordinated movements under (primarily) visual control is a particularly difficult goal to accomplish using current and predicted VE equipment. The causal factors are twofold: 1. tracking technologies, like most other sensor/display technologies, progress more slowly than other areas of computer technology (such as processor speed and storage size), 2. when executing tasks involving coordinated search and response movements, humans are keenly sensitive to temporal asynchronies arising from processing delays and sensing/display delays and errors. The costs of ignoring the deficiency can take many forms. In the most straightforward form, it can mean that training that uses the completed VIRTE system will fail to have any effect, and may in fact result in a negative training outcome (where the user actually becomes worse at performing the task in the real world after experiencing the VE training system). A more subtle but nevertheless likely outcome of ignoring the deficiency is that other (somewhat unrelated) skills that utilize similar coordinated movements, (e.g., marksmanship skills,) can be inadvertently degraded by exposure to the training system. S&T is required to characterize the problem and identify solutions in a manner that is directly relevant to the infantry tasks planned for VIRTE Demo II. Although related work exists that speaks to some aspects of the problem, application of this work would be sketchy at best, and would miss crucial aspects that are central to the success of VIRTE as a system suitable for tasks involving weapon handling. The use of immersive VEs to train tasks involving sensorimotor action (such as weapon handling under stress) is a novel domain, subsequently there are no existing documented standards to follow. However, given the expected large-scale exposure of trainees to these systems the results of the study are likely to become important reference information for VIRTE and other future systems.

 

 

I.C.2 Technical Background

A previous VIRTE grant to our lab entitled "Realism in Virtual Environments" has provided funding to develop instrumentation and software that will be used in the research proposed here. Although the currently proposed work represents a change in scope from that which was previously planned, it is in large part coincident with the past goals, and is more specifically tuned to particular needs of Demo II. The main contributions from this past work are outlined in section 1.E.2, and are covered in previous presentations and communications submitted to the program sponsor.

` Previous general work on temporal asynchronies in visual displays covers the effects of decreased frame rate, lag, and spatial jitter. These effects result in temporal sampling artifacts, including jerky motion, reversal of motion, multiple images, shimmering edges and many others [Crow, 1977; Edgar & Bex, 1995; Watt 1989]. Performance on moving target detection, recognition, and identification tasks [Swartz, Wallace & Tkacz, 1992] and on tracking tasks [Ellis et al. 1999] is hindered by temporal asynchronies. Although this research is of importance to our work, the results are decidedly task-dependent. Fortunately, these existing results can be put to use in guiding our work on the current (CQB) task of interest.

 

I.C.3 Technical Approach

The technical approach is designed specifically to mesh with the particular task domain chosen for VIRTE Demo II, and to address issues related to the hardware configuration slated for use in Demo II. There are at least two consequences of this tightly-knit approach: 1. The development of experimental stimuli and presentation methodology must remain flexible enough to accommodate possible changes posed by the designers of the Demo II system, 2. The experiments should be designed such that even if a dramatic change in architecture was dictated by the Demo II system designers, the results accrued to that point would still have relevance (thereby preventing wasted effort). We are addressing item 1 through our ongoing attempts to be responsive to, and informed about, the evolving state of the VIRTE system and its application domain. This communication will primarily happen through Dr. Templeman, but will also include direct contact as needed with end users. Item 2 represents a challenge from the standpoint that the work is designed to mesh with the Demo II needs as closely as possible. However, we feel that the results of the study will have applicability beyond the specifics of the guiding application, and will be useful in a variety of Navy/USMC training tasks in which visual-motor coordination is an important component.

With the previous principles in mind, we plan to perform a series of experiments evaluating the effect of latency and jitter on weapons handling performance in a room clearing scenario. The goal of these experiments is to determine tolerable levels of these temporal asynchronies for guidence in development of Demo II. For example, if a tracking system does not meet the thresholds required for proper performance an alternative system can be sought or developed. If no such system can be found, the reduction in performance due to this technological limitation will be well documented and can be circumvented through judicious modification of the task to accommodate the technological limitation.

The experimental task will mimic the weapon handling activities of a soldier in Close Quarters Battle. A subject will look through an opening (i.e., a door or window) into an area populated with friendly personnel, enemy targets, and various obstacles, then address the enemy targets. This scenario will be further refined with respect to the location and number of the targets until a sufficiently representative experimental task is achieved.

In order to effectively evaluate the role of latency and jitter in the Demo II simulator, we need to isolate these factors from other potential confounding elements. For example, one comparison will be to determine whether the latency threshold is higher for a low-resolution high-FOV HMD-based system than for a high-resolution, low-FOV projection-based system. To isolate the characteristics in which we are interested, we will compare a small-footprint (i.e., HMD-based) system (SF) with a large-footprint system that better approaches certain (target-oriented) real-world characteristics (RW'). The RW' system allows us to find baseline values for our measures, which we can then compare with those obtained in the SF system. Furthermore, we will degrade the RW' (DRW') system to study only latency or jitter, thus removing artifacts due to other system limitations (i.e., FOV, resolution, etc.).

In addition to the effects of latency and jitter on performance in the SF, DRW', and RW' systems, we will evaluate the degree to which latency and jitter result in negative transfer. If a subject trains in the SF or DRW' system (containing some level of latency and hitter), and then performs in the RW' well below the standard for a RW'-trained subject, we can conclude that the level of latency and jitter in the system is problematic. Thus, we are not only examining the effects of performance resulting from technological limitations, but also the effects on training transfer.

The effectiveness of the simulation will be measured in three ways: (1) the speed and accuracy with which the enemy targets are addressed, (2) the psychophysiological reactions of the subject, and (3) post-test questionnaires and rating scales. In this way, we are evaluating not only performance, but also the realism of the simulator.

One of the first steps in determining the effects of temporal asynchronies is to define the kinds of latency and jitter that are present in the system. Two types of latency issues need to be addressed: (1) latency that results in "lag" and (2) latency that results in a low frame rate (a third effect involves a second-order temporal sampling artifact where the frame rate can "jump"). Type (1) latency is generally caused by end-to-end system latency. Type (2) latency is generally due to the computational requirements of the simulator. Jitter has amplitude and frequency components, as well as the potential for other spatial bias. All of these effects need to be characterized for the tracking system to be used in Demo II. Furthermore, the interaction between these effects will be examined in some detail in order to determine reasonable threshold values.

Most importantly, although much of the stated approach implies a precipitous decline in human performance below certain values of system performance, we expect the data to reveal subtleties and interactions that can be exploited with the goal of circumventing unavoidable system limitations. We realize that a simplistic "go / no-go" identification is not always useful, but we fully expect that our results will go beyond this binary distinction and lead to useful methods of preserving the training goals in the face of system limitations.

 

I.C.4 Dual-Use

Research in this area has many implications for VE training of non-military tasks, and we plan to publish the results in the appropriate journal. Although we have not identified particular industrial partners, the work is likely to be of interest to commercial firms involved in VE training, tracking, and simulation.

 

I.C.6 S&T Product Descriptions and Exit Criteria

The overall exit criteria for the project consist of reports to the sponsor detailing the effects on training transfer of temporal asynchronies in the context of weapon handling tasks being trained in an immersive VE. There are a number of phases to the proposed work leading to observable milestones, in particular: 1. development of performance metrics appropriate to evaluate the cognitive, motor, and skill level of the task; 2. development of physiological metrics relevant to the task, including experimental validation; 3. an analysis of the temporal performance of trackers and display systems and creation of representative models of the performance (generator functions); 4. three phases of software development for representing scenarios with increasing verisimilitude and ability to elicit cognitive involvement and emotional response; 5. training transfer experiments (utilizing software from item 4) in which comparisons are made among "SF" (immersive VE) and different versions of "RW."

 

 

 

References

 

Crow, F. 1977. "The Aliasing Problem in Computer-Generated Shaded Images." Comm. of the ACM, 20(11).

 

Edgar, G. & Bex, P. 1995. "Vision and Displays." in K. Carr & R. England (Eds.), Simulated and Virtual Realities: Elements of Perception, pp. 85-101. London: Taylor & Francis.

 

Watt, A. 1989.  Fundamentals of Three-Dimensional Computer Graphics. New York: Addison-Wesley Publishing.

 

Swartz, M., Wallace, D. & Tkacz, S. 1992. "The Influence of Frame Rate and Resolution Reduction on Human Performance." Proc. Human Factors Society 36th Annual Meeting, pp. 1440-1444.

 

Ellis, S., Adelstein, B., Baumeler, S., Jense, G., & Jacoby, R. 1999. "Sensor Spatial Distortion, Visual Latency, and Update Rate Effects on 3D Tracking in Virtual Environments." in Proceedings of VR '99, pp. 218-221.



Assessment & Circumvention of Temporal Asynchronies in Weapon Handling Virtual Environments
Late @ 10:43 AM, July 17, 2001