Pub Date : 2024-01-27DOI: 10.1177/26349795241230966
James Paul Gee, Qing Archer Zhang
This was an invited paper meant to accompany a paper by Bill Cope and Mary Kalantzis.
这是一篇特邀论文，旨在配合比尔-科普和玛丽-卡兰茨斯的论文。
{"title":"Reflections on Cope and Kalantzis: Generative Artificial Intelligence and humans","authors":"James Paul Gee, Qing Archer Zhang","doi":"10.1177/26349795241230966","DOIUrl":"https://doi.org/10.1177/26349795241230966","url":null,"abstract":"This was an invited paper meant to accompany a paper by Bill Cope and Mary Kalantzis.","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"160 4","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-01-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140492601","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-01-20DOI: 10.1177/26349795241228473
Friendred (Youhong) Peng
The traditional construct of dance performances has predominantly positioned the audience as passive observers with little to no interaction with the unfolding narrative on stage. However, the advent of digital technology has paved the way for a more immersive and interactive performative experience. This paper elucidates the transformation of this dynamic through the lens of an interactive dance performance piece, “Moving Photon”, which endeavors to blur the entrenched demarcations between performers and spectators. By melding choreographic improvisation, interactive installations, and physiological sensing, “Moving Photon” cultivates a co-constitutive interaction, challenging conventional performative frameworks. Through two distinct configurations, performance and installation setting, the piece endeavors to transition spectators from passive observers to active collaborators in the narrative. The performance setting leverages real-time electroencephalogram (EEG) data from audience members to influence the kinetic dynamics of the installation, thus directly impacting the dancer's improvisational choreography. Conversely, the setup of the interactive installation invites participants to influence the musical score and the installation's robotic movements through motion-sensing devices, fostering a tangible feedback loop. The reflections gathered from initial exhibitions reveal a heightened sense of connectivity and shared authorship of the performance among both the performer and audience members. This innovative exploration signifies a step forward towards fostering more collaborative and multimodal engagements in digital art performances.
{"title":"Transforming spectator into collaborator during a digitalised dance performance","authors":"Friendred (Youhong) Peng","doi":"10.1177/26349795241228473","DOIUrl":"https://doi.org/10.1177/26349795241228473","url":null,"abstract":"The traditional construct of dance performances has predominantly positioned the audience as passive observers with little to no interaction with the unfolding narrative on stage. However, the advent of digital technology has paved the way for a more immersive and interactive performative experience. This paper elucidates the transformation of this dynamic through the lens of an interactive dance performance piece, “Moving Photon”, which endeavors to blur the entrenched demarcations between performers and spectators. By melding choreographic improvisation, interactive installations, and physiological sensing, “Moving Photon” cultivates a co-constitutive interaction, challenging conventional performative frameworks. Through two distinct configurations, performance and installation setting, the piece endeavors to transition spectators from passive observers to active collaborators in the narrative. The performance setting leverages real-time electroencephalogram (EEG) data from audience members to influence the kinetic dynamics of the installation, thus directly impacting the dancer's improvisational choreography. Conversely, the setup of the interactive installation invites participants to influence the musical score and the installation's robotic movements through motion-sensing devices, fostering a tangible feedback loop. The reflections gathered from initial exhibitions reveal a heightened sense of connectivity and shared authorship of the performance among both the performer and audience members. This innovative exploration signifies a step forward towards fostering more collaborative and multimodal engagements in digital art performances.","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"8 46","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-01-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139523918","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-28DOI: 10.1177/26349795231221699
B. Cope, M. Kalantzis
This paper analyzes the scope of Artificial Intelligence (AI) from the perspective of a multimodal grammar. Its focal point is Generative AI, a technology that puts so-called Large Language Models to work. The first part of the paper analyzes Generative AI, based as it is on the statistical probability of one token (a word or part of a word) following another. If the relation of tokens is meaningful, this is circumstantial and no more, because its mechanisms of statistical analysis eschew any theory of meaning. This is the case not only for the written text that Generative AI leverages, but by extension image and multimodal forms of meaning that it can generate. The AI can only work with non-textual forms of meaning after applying language labels, and to that extent is captive not only to the limits of probabilistic statistics but the limits of written language as well. While acknowledging gains arising from the brute statistical power of Generative AI, in its second part the paper goes on to map what is lost in its statistical and text-bound approaches to multimodal meaning-making. Our measure of these gains and losses is guided by the concept of grammar, defined here as a theory of the elemental patterns of meaning in the world—not just written text and speech, but also image, space, object, body, and sound. Ironically, a good deal of what is lost by Generative AI is computable. The third and final part of the paper briefly discusses educational applications of Generative AI. Given both its power and intrinsic limitations, we have been experimenting with the application of Generative AI in educational settings and the ways it might be put to pedagogical use. How does a grammatical analysis help us to identify the scope of worthwhile application? Finally, if more of human experience is computable than can be captured in text-bound AI, how might it be possible at the level of code to create a synthesis in which grammatical and multimodal approaches complement Generative AI?
{"title":"A multimodal grammar of artificial intelligence: Measuring the gains and losses in generative artificial intelligence","authors":"B. Cope, M. Kalantzis","doi":"10.1177/26349795231221699","DOIUrl":"https://doi.org/10.1177/26349795231221699","url":null,"abstract":"This paper analyzes the scope of Artificial Intelligence (AI) from the perspective of a multimodal grammar. Its focal point is Generative AI, a technology that puts so-called Large Language Models to work. The first part of the paper analyzes Generative AI, based as it is on the statistical probability of one token (a word or part of a word) following another. If the relation of tokens is meaningful, this is circumstantial and no more, because its mechanisms of statistical analysis eschew any theory of meaning. This is the case not only for the written text that Generative AI leverages, but by extension image and multimodal forms of meaning that it can generate. The AI can only work with non-textual forms of meaning after applying language labels, and to that extent is captive not only to the limits of probabilistic statistics but the limits of written language as well. While acknowledging gains arising from the brute statistical power of Generative AI, in its second part the paper goes on to map what is lost in its statistical and text-bound approaches to multimodal meaning-making. Our measure of these gains and losses is guided by the concept of grammar, defined here as a theory of the elemental patterns of meaning in the world—not just written text and speech, but also image, space, object, body, and sound. Ironically, a good deal of what is lost by Generative AI is computable. The third and final part of the paper briefly discusses educational applications of Generative AI. Given both its power and intrinsic limitations, we have been experimenting with the application of Generative AI in educational settings and the ways it might be put to pedagogical use. How does a grammatical analysis help us to identify the scope of worthwhile application? Finally, if more of human experience is computable than can be captured in text-bound AI, how might it be possible at the level of code to create a synthesis in which grammatical and multimodal approaches complement Generative AI?","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"39 3","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139150413","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-21DOI: 10.1177/26349795231222917
Julia A Gorham, Natalie A. Amgott
Research has long touted and recently confirmed the importance of intercultural communicative competence (ICC) — the ability to appropriately communicate with those from diverse sociocultural and linguistic backgrounds — in second language (L2) teaching and learning. Additionally, while multimodal course materials have contributed to learners’ ICC, the link between students’ embodied modes (gestures, facial expressions, body movements) and ICC remains under-explored. Building on ICC and the social semiotic theory of multimodality, this study blends multimodal transcription and conversation analysis to examine how 73 university undergraduate learners of L2 French used embodied modes in 188 asynchronous video reflections and how these modes accompanied demonstrations of ICC skills, knowledge, and attitudes. Findings indicate that gestures and facial expressions indicated students’ demonstrations of the ICC skills of observing, analyzing/interpreting, evaluating, relating, listening, questioning, researching, and problematizing. Further, embodied modes demonstrated students’ retrieval and communication of cultural and sociolinguistic knowledge. This study has implications for online language learning and enhancing ICC through virtual multimodal assignments.
{"title":"Embodying intercultural communicative competence in French L2 video reflections: Illustrating skills, knowledge, and attitudes","authors":"Julia A Gorham, Natalie A. Amgott","doi":"10.1177/26349795231222917","DOIUrl":"https://doi.org/10.1177/26349795231222917","url":null,"abstract":"Research has long touted and recently confirmed the importance of intercultural communicative competence (ICC) — the ability to appropriately communicate with those from diverse sociocultural and linguistic backgrounds — in second language (L2) teaching and learning. Additionally, while multimodal course materials have contributed to learners’ ICC, the link between students’ embodied modes (gestures, facial expressions, body movements) and ICC remains under-explored. Building on ICC and the social semiotic theory of multimodality, this study blends multimodal transcription and conversation analysis to examine how 73 university undergraduate learners of L2 French used embodied modes in 188 asynchronous video reflections and how these modes accompanied demonstrations of ICC skills, knowledge, and attitudes. Findings indicate that gestures and facial expressions indicated students’ demonstrations of the ICC skills of observing, analyzing/interpreting, evaluating, relating, listening, questioning, researching, and problematizing. Further, embodied modes demonstrated students’ retrieval and communication of cultural and sociolinguistic knowledge. This study has implications for online language learning and enhancing ICC through virtual multimodal assignments.","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"77 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138952870","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-12-18DOI: 10.1177/26349795231223678
Chikelue Chris Akabuike
As a process-oriented studio exploration, “The Ruler and the Ruled” is a generic name of a body of work that engages a joinery method to upcycle wood-off-cuts. It creates visual imagery that metaphorically interprets the consequences of Nigeria’s socio-political settings. Genesis of Disintegration is an art piece selected for discussion. It conceptually addresses the plights of the downtrodden. The use of wood offcuts effectively exploits upcycle in a creative interpretation of joinery processes distinct as practised by other artists. This reflection attempts to validate how wood offcuts form part of the ongoing dialogue of the downtrodden. Focusing on joinery and its aesthetic effects, emerging forms are metaphorically interpreted while their formal features are conceptually analyzed using theories and models of visual social semiotics. Significant socio-political issues were raised that addressed the plight of the downtrodden and its implications.
{"title":"Wood-off-cuts as a creative resource: A metaphorical reflection on “The Ruler and The Ruled” sculpture project I","authors":"Chikelue Chris Akabuike","doi":"10.1177/26349795231223678","DOIUrl":"https://doi.org/10.1177/26349795231223678","url":null,"abstract":"As a process-oriented studio exploration, “The Ruler and the Ruled” is a generic name of a body of work that engages a joinery method to upcycle wood-off-cuts. It creates visual imagery that metaphorically interprets the consequences of Nigeria’s socio-political settings. Genesis of Disintegration is an art piece selected for discussion. It conceptually addresses the plights of the downtrodden. The use of wood offcuts effectively exploits upcycle in a creative interpretation of joinery processes distinct as practised by other artists. This reflection attempts to validate how wood offcuts form part of the ongoing dialogue of the downtrodden. Focusing on joinery and its aesthetic effects, emerging forms are metaphorically interpreted while their formal features are conceptually analyzed using theories and models of visual social semiotics. Significant socio-political issues were raised that addressed the plight of the downtrodden and its implications.","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"100 ","pages":""},"PeriodicalIF":0.0,"publicationDate":"2023-12-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139174300","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-08-31DOI: 10.1177/26349795231200643
Ida Melander
{"title":"Book review: Digital communication and media linguistics","authors":"Ida Melander","doi":"10.1177/26349795231200643","DOIUrl":"https://doi.org/10.1177/26349795231200643","url":null,"abstract":"","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"344 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-08-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116456189","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-08-09DOI: 10.1177/26349795231191023
L. Malafouris, R. Carnegie, Miranda Creswell, A. Iliopoulos, M. Koukouti, Wendy Ross
Perspectival Kinaesthetic Imaging is being proposed as a method designed to facilitate the heightened sensitivity needed for the anthropological study of the relationship between making and thinking, during the creative engagement with form-generating materials. Technically, this objective is achieved through the juxtaposition of perspectival view points on the process of making. We follow the ways of the hand using a combination of multimodal visual captures (i.e., photography, video, observational drawing and mobile eye-tracking). Each of these multimodal visual captures affords a specific spatio-temporal perspective from which to identify and observe morphogenetic events of interest (e.g. creative gestures and modes of enactive signification). The basic idea is that the juxtaposition of different media affects how we observe and what can be observed by enabling the discovery of connections and material relations that are often obscured when seen from a single perspectival point.
{"title":"Perspectival Kinaesthetic Imaging","authors":"L. Malafouris, R. Carnegie, Miranda Creswell, A. Iliopoulos, M. Koukouti, Wendy Ross","doi":"10.1177/26349795231191023","DOIUrl":"https://doi.org/10.1177/26349795231191023","url":null,"abstract":"Perspectival Kinaesthetic Imaging is being proposed as a method designed to facilitate the heightened sensitivity needed for the anthropological study of the relationship between making and thinking, during the creative engagement with form-generating materials. Technically, this objective is achieved through the juxtaposition of perspectival view points on the process of making. We follow the ways of the hand using a combination of multimodal visual captures (i.e., photography, video, observational drawing and mobile eye-tracking). Each of these multimodal visual captures affords a specific spatio-temporal perspective from which to identify and observe morphogenetic events of interest (e.g. creative gestures and modes of enactive signification). The basic idea is that the juxtaposition of different media affects how we observe and what can be observed by enabling the discovery of connections and material relations that are often obscured when seen from a single perspectival point.","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-08-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128791277","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-08-08DOI: 10.1177/26349795231194746
Ricardo O'Nascimento
Wearable haptic artefacts are more and more present in our everyday life. They increasingly mediate our perception, transforming the ways in which we interact with ourselves, the environment and others. Wearable haptic artefacts can sense the body and its surroundings, impacting our subjectivity by shaping perceptions, actions, and behaviours. Such ability to influence our senses is helpful in creating engaging multisensory experiences. In my work, I explore the technological mediation between the human body and its surroundings by facilitating new relationships among the senses. This article presents a wearable haptic artefact in the shape of a glove called “I am feeling blue… and red”, which was designed to enable users to experience colour through touch. The goal is to suggest a mode of colour experience and break preconceptions one might have towards the relationship there is between touch and colours. Through an autoethnographic account, the article reflects on the artefact’s design process and user interaction. It proposes strategies for developing unique multisensory experiences using wearable haptic artefacts. These strategies can assist artists and designers who want to create captivating interactions by connecting the senses of sight and touch.
{"title":"“I am feeling blue… and red”: Exploring alternative ways of feeling colours through a wearable haptic artefact","authors":"Ricardo O'Nascimento","doi":"10.1177/26349795231194746","DOIUrl":"https://doi.org/10.1177/26349795231194746","url":null,"abstract":"Wearable haptic artefacts are more and more present in our everyday life. They increasingly mediate our perception, transforming the ways in which we interact with ourselves, the environment and others. Wearable haptic artefacts can sense the body and its surroundings, impacting our subjectivity by shaping perceptions, actions, and behaviours. Such ability to influence our senses is helpful in creating engaging multisensory experiences. In my work, I explore the technological mediation between the human body and its surroundings by facilitating new relationships among the senses. This article presents a wearable haptic artefact in the shape of a glove called “I am feeling blue… and red”, which was designed to enable users to experience colour through touch. The goal is to suggest a mode of colour experience and break preconceptions one might have towards the relationship there is between touch and colours. Through an autoethnographic account, the article reflects on the artefact’s design process and user interaction. It proposes strategies for developing unique multisensory experiences using wearable haptic artefacts. These strategies can assist artists and designers who want to create captivating interactions by connecting the senses of sight and touch.","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-08-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114143281","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2023-07-25DOI: 10.1177/26349795231188727
Emilie Munch Nicolaisen, Gitte Rasmussen
This paper focuses on interdisciplinary trials as a multiparty and multiactivity setting in which different professions are needed for students with mobility disabilities to try out robot-assisted gait training. Through an ethnomethodological and conversation analytic approach to multimodal interaction analysis, we investigate how a research team collaborates with the employees at a sports high school to get students who are wheelchair users to walk. The paper provides an analysis of the professionals’ employment of multimodal resources for managing deontic rights when organizing interdisciplinary trials. It examines how the professionals initiate and negotiate their own and others’ actions necessary for preparing the student for walking. Based on video recordings, the paper focuses on the professionals’ deontic claims concerning co-participants actions and analyzes their temporal and spatial contingencies. Building on prior research on deontics, this paper contributes to our understanding of the interactive accomplishment of deontic authority and its sensitivity to contextual features.
{"title":"Multimodal methods for managing deontic rights in interdisciplinary trials","authors":"Emilie Munch Nicolaisen, Gitte Rasmussen","doi":"10.1177/26349795231188727","DOIUrl":"https://doi.org/10.1177/26349795231188727","url":null,"abstract":"This paper focuses on interdisciplinary trials as a multiparty and multiactivity setting in which different professions are needed for students with mobility disabilities to try out robot-assisted gait training. Through an ethnomethodological and conversation analytic approach to multimodal interaction analysis, we investigate how a research team collaborates with the employees at a sports high school to get students who are wheelchair users to walk. The paper provides an analysis of the professionals’ employment of multimodal resources for managing deontic rights when organizing interdisciplinary trials. It examines how the professionals initiate and negotiate their own and others’ actions necessary for preparing the student for walking. Based on video recordings, the paper focuses on the professionals’ deontic claims concerning co-participants actions and analyzes their temporal and spatial contingencies. Building on prior research on deontics, this paper contributes to our understanding of the interactive accomplishment of deontic authority and its sensitivity to contextual features.","PeriodicalId":153235,"journal":{"name":"Multimodality & Society","volume":"96 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-07-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116883589","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：