Representations and consequences of race in AI systems

Abstract

Race is directly or indirectly incorporated into many AI systems. These systems, which automate typically human tasks, are used across various domains such as predictive policing, disease detection, government resource allocation, and loan approvals. However, these tools have been criticized for handling race insensitively or inaccurately. Despite the prevalent use of race in these AI systems, it is often not properly defined. It is treated as an obvious concept and represented as fixed categories, which fail to fully incorporate the social meaning surrounding race. Thus, in this review article, we define race and discuss how it is represented in AI systems. We also explore the consequences of such representations and offer recommendations on how to incorporate race more appropriately in these systems.