Aaron Einbond, Thibaut Carpentier, Diemo Schwarz, Jean Bresson
The situated spatial presence of musical instruments has been well studied in the fields of acoustics and music perception research, but so far it has not been the focus of human-AI interaction. We respond critically to this trend by seeking to reembody interactive electronics using data derived from natural acoustic phenomena. Two musical works, composed for human soloist and computer-generated live electronics, are intended to situate the listener in an immersive sonic environment in which real and virtual sources blend seamlessly. To do so, we experimented with two contrasting reproduction setups: a surrounding Ambisonic loudspeaker dome and a compact spherical loudspeaker array for radiation synthesis. A large database of measured radiation patterns of orchestral instruments served as a training set for machine learning models to control spatially rich 3-D patterns for electronic sounds. These are exploited during performance in response to live sounds captured with a spherical microphone array and used to train computer models of improvisation and to trigger corpus-based spatial synthesis. We show how AI techniques are useful to utilize complex, multidimensional, spatial data in the context of computer-assisted composition and human-computer interactive improvisation.
{"title":"Embodying Spatial Sound Synthesis with AI in Two Compositions for Instruments and 3-D Electronics","authors":"Aaron Einbond, Thibaut Carpentier, Diemo Schwarz, Jean Bresson","doi":"10.1162/comj_a_00664","DOIUrl":"https://doi.org/10.1162/comj_a_00664","url":null,"abstract":"The situated spatial presence of musical instruments has been well studied in the fields of acoustics and music perception research, but so far it has not been the focus of human-AI interaction. We respond critically to this trend by seeking to reembody interactive electronics using data derived from natural acoustic phenomena. Two musical works, composed for human soloist and computer-generated live electronics, are intended to situate the listener in an immersive sonic environment in which real and virtual sources blend seamlessly. To do so, we experimented with two contrasting reproduction setups: a surrounding Ambisonic loudspeaker dome and a compact spherical loudspeaker array for radiation synthesis. A large database of measured radiation patterns of orchestral instruments served as a training set for machine learning models to control spatially rich 3-D patterns for electronic sounds. These are exploited during performance in response to live sounds captured with a spherical microphone array and used to train computer models of improvisation and to trigger corpus-based spatial synthesis. We show how AI techniques are useful to utilize complex, multidimensional, spatial data in the context of computer-assisted composition and human-computer interactive improvisation.","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"29 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-01-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139496833","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Gérard Assayag, Laurent Bonnasse-Gahot, Joakim Borg
Somax2 is an AI-based multiagent system for human-machine coimprovisation that generates stylistically coherent streams while continuously listening and adapting to musicians or other agents. The model on which it is based can be used with little configuration to interact with humans in full autonomy, but it also allows fine real-time control of its generative processes and interaction strategies, closer in this case to a “smart” digital instrument. An offspring of the Omax system, conceived at the Institut de Recherche et Coordination Acoustique/Musique, the Somax2 environment is part of the European Research Council Raising Cocreativity in Cyber-Human Musicianship (REACH) project, which studies distributed creativity as a general template for symbiotic interaction between humans and digital systems. It fosters mixed musical reality involving cocreative AI agents. The REACH project puts forward the idea that cocreativity in cyber-human systems results from the emergence of complex joint behavior, produced by interaction and featuring cross-learning mechanisms. Somax2 is a first step toward this ideal, and already shows life-size achievements. This article describes Somax2 extensively, from its theoretical model to its system architecture, through its listening and learning strategies, representation spaces, and interaction policies.
{"title":"Cocreative Interaction: Somax2 and the REACH Project","authors":"Gérard Assayag, Laurent Bonnasse-Gahot, Joakim Borg","doi":"10.1162/comj_a_00662","DOIUrl":"https://doi.org/10.1162/comj_a_00662","url":null,"abstract":"Somax2 is an AI-based multiagent system for human-machine coimprovisation that generates stylistically coherent streams while continuously listening and adapting to musicians or other agents. The model on which it is based can be used with little configuration to interact with humans in full autonomy, but it also allows fine real-time control of its generative processes and interaction strategies, closer in this case to a “smart” digital instrument. An offspring of the Omax system, conceived at the Institut de Recherche et Coordination Acoustique/Musique, the Somax2 environment is part of the European Research Council Raising Cocreativity in Cyber-Human Musicianship (REACH) project, which studies distributed creativity as a general template for symbiotic interaction between humans and digital systems. It fosters mixed musical reality involving cocreative AI agents. The REACH project puts forward the idea that cocreativity in cyber-human systems results from the emergence of complex joint behavior, produced by interaction and featuring cross-learning mechanisms. Somax2 is a first step toward this ideal, and already shows life-size achievements. This article describes Somax2 extensively, from its theoretical model to its system architecture, through its listening and learning strategies, representation spaces, and interaction policies.","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"15 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-01-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139482224","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Machine learning (ML) deals with algorithms able to learn from data, with the primary aim of finding optimum solutions to perform tasks autonomously. In recent years there has been development in integrating ML algorithms with live coding practices, raising questions about what to optimize or automate, the agency of the algorithms, and in which parts of the ML processes one might intervene midperformance. Live coding performance practices typically involve conversational interaction with algorithmic processes in real time. In analyzing systems integrating live coding and ML, we consider the musical and performative implications of the “moment of intervention” in the ML model and workflow, and the channels for real-time intervention. We propose a framework for analysis, through which we reflect on the domain-specific algorithms and practices being developed that combine these two practices.
机器学习（ML）涉及能够从数据中学习的算法，其主要目的是找到最佳解决方案来自主执行任务。近年来，将 ML 算法与实时编码实践结合在一起的做法得到了发展，但也提出了一些问题，如哪些内容需要优化或自动化、算法的作用以及在执行过程中可以干预 ML 流程的哪些部分。实时编码性能实践通常涉及与算法过程的实时对话交互。在分析集成了现场编码和 ML 的系统时，我们会考虑 ML 模型和工作流程中 "干预时刻 "的音乐和表演含义，以及实时干预的渠道。我们提出了一个分析框架，通过该框架，我们思考了结合这两种实践而开发的特定领域算法和实践。
{"title":"Live Coding Machine Learning: Finding the Moments of Intervention in Autonomous Processes","authors":"Iván Paz, Shelly Knottsy","doi":"10.1162/comj_a_00663","DOIUrl":"https://doi.org/10.1162/comj_a_00663","url":null,"abstract":"Machine learning (ML) deals with algorithms able to learn from data, with the primary aim of finding optimum solutions to perform tasks autonomously. In recent years there has been development in integrating ML algorithms with live coding practices, raising questions about what to optimize or automate, the agency of the algorithms, and in which parts of the ML processes one might intervene midperformance. Live coding performance practices typically involve conversational interaction with algorithmic processes in real time. In analyzing systems integrating live coding and ML, we consider the musical and performative implications of the “moment of intervention” in the ML model and workflow, and the channels for real-time intervention. We propose a framework for analysis, through which we reflect on the domain-specific algorithms and practices being developed that combine these two practices.","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"24 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-01-16","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"139482307","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Çağrı Erdem, Benedikte Wallace, Kyrre Glette, Alexander Refsum Jensenius
Abstract In this article we introduce the coadaptive audiovisual instrument CAVI. This instrument uses deep learning to generate control signals based on muscle and motion data of a performer's actions. The generated signals control time-based live sound-processing modules. How does a performer perceive such an instrument? Does it feel like a machine learning-based musical tool? Or is it an actor with the potential to become a musical partner? We report on an evaluation of CAVI after it had been used in two public performances. The evaluation is based on interviews with the performers, audience questionnaires, and the creator's self-analysis. Our findings suggest that the perception of CAVI as a tool or actor correlates with the performer's sense of agency. The perceived agency changes throughout a performance based on several factors, including perceived musical coordination, the balance between surprise and familiarity, a “common sense,” and the physical characteristics of the performance setting.
{"title":"Tool or Actor? Expert Improvisers' Evaluation of a Musical AI “Toddler”","authors":"Çağrı Erdem, Benedikte Wallace, Kyrre Glette, Alexander Refsum Jensenius","doi":"10.1162/comj_a_00657","DOIUrl":"https://doi.org/10.1162/comj_a_00657","url":null,"abstract":"Abstract In this article we introduce the coadaptive audiovisual instrument CAVI. This instrument uses deep learning to generate control signals based on muscle and motion data of a performer's actions. The generated signals control time-based live sound-processing modules. How does a performer perceive such an instrument? Does it feel like a machine learning-based musical tool? Or is it an actor with the potential to become a musical partner? We report on an evaluation of CAVI after it had been used in two public performances. The evaluation is based on interviews with the performers, audience questionnaires, and the creator's self-analysis. Our findings suggest that the perception of CAVI as a tool or actor correlates with the performer's sense of agency. The perceived agency changes throughout a performance based on several factors, including perceived musical coordination, the balance between surprise and familiarity, a “common sense,” and the physical characteristics of the performance setting.","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"57 25","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135091944","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Abstract In this article we address the role of machine learning (ML) in the composition of two new musical works for acoustic instruments and electronics through autoethnographic reflection on the experience. Our study poses the key question of how ML shapes, and is in turn shaped by, the aesthetic commitments characterizing distinctive compositional practices. Further, we ask how artistic research in these practices can be informed by critical themes from humanities scholarship on material engagement and critical data studies. Through these frameworks, we consider in what ways the interaction with ML algorithms as part of the compositional process differs from that with other music technology tools. Rather than focus on narrowly conceived ML algorithms, we take into account the heterogeneous assemblage brought into play: from composers, performers, and listeners to loudspeakers, microphones, and audio descriptors. Our analysis focuses on a deconstructive critique of data as being contingent on the decisions and material conditions involved in the data creation process. It also explores how interaction among the human and nonhuman collaborators in the ML assemblage has significant similarities to—as well as differences from—existing models of material engagement. Tracking the creative process of composing these works, we uncover the aesthetic implications of the many nonlinear collaborative decisions involved in composing the assemblage.
{"title":"Composing the Assemblage: Probing Aesthetic and Technical Dimensions of Artistic Creation with Machine Learning","authors":"Artemi-Maria Gioti, Aaron Einbond, Georgina Born","doi":"10.1162/comj_a_00658","DOIUrl":"https://doi.org/10.1162/comj_a_00658","url":null,"abstract":"Abstract In this article we address the role of machine learning (ML) in the composition of two new musical works for acoustic instruments and electronics through autoethnographic reflection on the experience. Our study poses the key question of how ML shapes, and is in turn shaped by, the aesthetic commitments characterizing distinctive compositional practices. Further, we ask how artistic research in these practices can be informed by critical themes from humanities scholarship on material engagement and critical data studies. Through these frameworks, we consider in what ways the interaction with ML algorithms as part of the compositional process differs from that with other music technology tools. Rather than focus on narrowly conceived ML algorithms, we take into account the heterogeneous assemblage brought into play: from composers, performers, and listeners to loudspeakers, microphones, and audio descriptors. Our analysis focuses on a deconstructive critique of data as being contingent on the decisions and material conditions involved in the data creation process. It also explores how interaction among the human and nonhuman collaborators in the ML assemblage has significant similarities to—as well as differences from—existing models of material engagement. Tracking the creative process of composing these works, we uncover the aesthetic implications of the many nonlinear collaborative decisions involved in composing the assemblage.","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"57 24","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-11-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135091945","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Finite State Machines with Data Paths in Visual Languages for Music","authors":"Tiago Fernandes Tavares;José Eduardo Fornari","doi":"10.1162/COMJ_a_00688","DOIUrl":"10.1162/COMJ_a_00688","url":null,"abstract":"","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"47 3","pages":"35-49"},"PeriodicalIF":0.4,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142223861","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Products of Interest","authors":"","doi":"10.1162/COMJ_r_00690","DOIUrl":"https://doi.org/10.1162/COMJ_r_00690","url":null,"abstract":"","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"47 3","pages":"71-85"},"PeriodicalIF":0.4,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143430562","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Esteban Gutiérrez;Christopher Haworth;Rodrigo F. Cádiz
{"title":"Generating Sonic Phantoms with Quadratic Difference Tone Spectrum Synthesis","authors":"Esteban Gutiérrez;Christopher Haworth;Rodrigo F. Cádiz","doi":"10.1162/COMJ_a_00687","DOIUrl":"10.1162/COMJ_a_00687","url":null,"abstract":"","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"47 3","pages":"19-34"},"PeriodicalIF":0.4,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142223863","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"About This Issue","authors":"","doi":"10.1162/COMJ_e_00689","DOIUrl":"https://doi.org/10.1162/COMJ_e_00689","url":null,"abstract":"","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"47 3","pages":"1-1"},"PeriodicalIF":0.4,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143430498","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Using Music Features for Managing Revisions and Variants of Musical Scores","authors":"Paul Grünbacher;Rudolf Hanl;Lukas Linsbauer","doi":"10.1162/COMJ_a_00691","DOIUrl":"https://doi.org/10.1162/COMJ_a_00691","url":null,"abstract":"","PeriodicalId":50639,"journal":{"name":"Computer Music Journal","volume":"47 3","pages":"50-68"},"PeriodicalIF":0.4,"publicationDate":"2023-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143430518","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：