ChatGPT and CLT: Investigating differences in multimodal processing

Abstract

Drawing on construal level theory, recent studies have demonstrated that ChatGPT interprets text inputs from an abstract perspective. However, as ChatGPT has evolved into a multimodal tool, this research examines whether ChatGPT's abstraction bias extends to image-based prompts. In a pre-registered study utilising hierarchical letters, ChatGPT predominantly associated these images with local rather than global letters, suggesting a concrete bias when analysing images. This starkly contrasts human participants who predominantly identified the same images with the global letters, indicating that humans and ChatGPT significantly diverge in image interpretations. Furthermore, while humans generally perceive ChatGPT to be more concrete in image processing, there is a notable discrepancy between this perception and the actual level of concreteness exhibited by ChatGPT in handling image-based tasks. These findings provide insights into the distinct cognitive behaviours of LLMs compared to humans, contributing to an emerging understanding of LLM cognition in the context of multimodal inputs.