Design and Development of a Prototype Heads-Up Display: Supporting Context-Aware, Semi-Automated, Hands-Free Medical Documentation

Military and civilian medical personnel across all echelons of medical care play a critical role in evaluating, caring for, and treating casualties. Accurate medical documentation is critical to effective, coordinated care and positive patient outcomes. We describe our prototype, Context-Aware Procedure Support Tools and User Interfaces for Rapid and Effective Workflows (CAPTURE). Leveraging human factors and usercentered design methods, and advanced artificial intelligence and computer vision capabilities, CAPTURE was designed to enable Tactical Combat Causality Care (TCCC) providers to more efficiently and effectively input critical medical information through hands-free interaction techniques and semiautomated data capture methods. We designed and prototyped a heads-up display that incorporates: multimodal interfaces, including augmented reality-based methods for input and information display to support visual image capture and heads-up interaction; post-care documentation support (e.g., artifacts to support post-care review and documentation); context-aware active and passive data capture methods, specifically natural language interpretation using systemic functional grammars; and computer vision technologies for semi-automated data capture capabilities. During the course of this project we encountered challenges towards effective design which fall into three main categories: (1) challenges related to designing novel multimodal interfaces; (2) technical challenges related to software and hardware development to meet design needs; and (3) challenges as a result of domain characteristics and operational constraints. We discuss how we addressed some of these challenges and provide additional considerations necessary for future research regarding next generation technology design for medical documentation in the field.