Commentary: controlling the demon: autonomous agents and the urgent need for controls

Abstract

In “Avoiding adverse autonomous agent actions,” Hancock (This issue) argues that controlling and exploiting autonomous systems represents one of the fundamental challenges of the 21 century. His parting shot is the disquieting and challenging observation that, with autonomous agents, we may be creating a new “peak predator” from which there will be no recovery of human control. The next generation of Artificial Intelligence (AI), Artificial General Intelligence (AGI) could see the idea of a new technological peak predator become reality. AGI will possess the capacity to learn, evolve and modify its functional capabilities and could quickly become intellectually superior to humans (Bostrom, 2014). Though estimates on when AGI will appear vary, the exact time of arrival is perhaps a moot point. What is more important, as Hancock alludes to, is that work is required immediately to ensure that the impact on humanity is positive rather than negative (Salmon et al., 2021). Should we take a reactive approach and only focus our efforts once AGI is created, it will already be too late (Bostrom, 2014). The first AGI system will quickly become uncontrollable.