Investigating students’ cognitive processes in generative AI-assisted digital multimodal composing and traditional writing

Recently, generative artificial intelligence (AI)-powered chatbots such as ChatGPT and Bing Chat have garnered increasing attention on a global scale. Previous studies have focused mostly on the influence of generative AI on writing while few researchers have investigated how generative AI can facilitate students' multimodal writing process. To fill in this gap, we explored the generative AI-assisted composing processes of two groups of English as a foreign language (EFL) writers over two weeks in this qualitative study. One group completed a multimodal PowerPoint (PPT) project, and the other group completed a traditional argumentative essay project. Our data consist of students’ screen recordings with think-aloud protocols, final multimodal texts, and post-project interviews. Our analysis showed different patterns in text production across the two groups. Students in the PPT group tended to construct more bridge texts and examples to corroborate their sub-claims in the hierarchical order. They also inclined to borrow the summarized search results from the Bing Chat to expand texts for their PPT slides. With regard to image generation for PPT slides, descriptions of AI images from ChatGPT were used as effective prompts to generate AI images from Bing Image Creator. Moreover, students were interested in producing and refining AI images following the recommended prompts by Bing Chat. They also evaluated these AI images from different perspectives. We conclude the study with a discussion on the pedagogical implications and suggestions for further study.