Health AI needs meaningful human involvement: lessons from war

Artificial intelligence (AI) continues to show its potential for innovation in healthcare. As this transformation occurs, robust methods are needed to assess when and how AI should function autonomously. The scalability of AI use in healthcare will often conflict with the maintenance of human oversight, and an appropriate balance must be sought. We draw on ongoing debates in the military domain to explain why the key to effective AI lies in ‘meaningful human involvement’ and key considerations in attaining it.

The goals of combat contrast sharply with the goals of healthcare. There are nonetheless salient parallels for shared approaches to weighing autonomy. The fundamental questions in both domains are, “When, how and by whom should decisions be made?”. Both arenas demand judgement of acceptable risk to life; the careful delineation of responsibility and accountability; and, ultimately, a decision about how much autonomy should be afforded to nonhuman systems.