Addressing the risks of generative AI for the judiciary: The accountability framework(s) under the EU AI Act

Abstract

The rapid advancements in natural language processing, particularly the development of generative large language models (LLMs), have renewed interest in using artificial intelligence (AI) for judicial decision-making. While these technological breakthroughs present new possibilities for legal automation, they also raise concerns about over-reliance and automation bias. Drawing insights from the COMPAS case, this paper examines the implications of deploying generative LLMs in the judicial domain. It identifies the persistent factors that contributed to an accountability gap when AI systems were previously used for judicial decision-making. To address these risks, the paper analyses the relevant provisions of the EU Artificial Intelligence Act, outlining a comprehensive accountability framework based on the regulation's risk-based approach. The paper concludes that the successful integration of generative LLMs in judicial decision-making requires a holistic approach addressing cognitive biases. By emphasising shared responsibility and the imperative of AI literacy across the AI value chain, the regulatory framework can help mitigate the risks of automation bias and preserve the rule of law.