Create a Crowd Emotion Detection Framework With Ecological Validity

Abstract

Ecological validity remains essential for generalizing scientific research into real-world applications. However, current methods for crowd emotion detection lack ecological validity due to limited diversity samples in datasets. This paper proposes a crowd emotion detection framework that improves the ecological validity of models from both dataset and methodological perspectives. Firstly, we develop an Emotional Crowd Generator script within Grand Theft Auto V to generate a large-scale and diverse synthetic emotional crowd dataset, named Emotional-GTA (E-GTA). Secondly, we utilize prior features to enhance the model's generalization ability, especially for rare samples. Building on this, we introduce a dual-driven Graph-based Prior Feature and Image Fusion Network (GPIFN), which further strengthens our model's ecological validity from a methodological perspective. We propose a graphical representation that effectively constructs the Crowd Image Graph (CIG) and the Crowd Prior Features Graph (CPG). The CIG represents crowds from the perspective of the image features, while the CPG represents them from the perspective of prior features. We then design a dual-stream network GPIFN that extracts image features from the CIG and prior features from the CPG. Additionally, we design an Image and Prior Features Fusion Module (IPFM) that efficiently merges image and prior features while maintaining the original stream features. Our experiments demonstrate that both E-GTA and GPIFN greatly enhance ecological validity in real-world scenarios. Our framework achieves state-of-the-art results on real-world datasets: UMN and Violent-Flows.