VRmonic: A VR Piano Playing Form Trainer

Ethan Matzek, Tyler Yankee, Odin Kohler, Tracy Lipke-Perry, N. Banerjee, Sean Banerjee
{"title":"VRmonic: A VR Piano Playing Form Trainer","authors":"Ethan Matzek, Tyler Yankee, Odin Kohler, Tracy Lipke-Perry, N. Banerjee, Sean Banerjee","doi":"10.1109/AIxVR59861.2024.00056","DOIUrl":null,"url":null,"abstract":"Improper hand form during practice and performance can lead to long-term injuries in musicians of all levels. We present a demonstration of VRmonic, a novel immersive VR-based piano trainer for learning correct hand form for piano by overlaying the playing form of an expert oracle. VRmonic consists of a library of 48 scales comprising of the major, and natural, harmonic, and melodic minor scales for each of the 12 notes. We obtain correct hand form by using an automatic approach to detect hand keypoints from recordings of an expert pianist collected using an overhead RGB-D mounted camera. During training, the user can use VRmonic to load in a prior recording of themselves playing a scale and compare their hand form against the expert oracle. During playback, we display virtual oracle and user hand overlap via a color change, controlled by our form tolerance threshold slider that allows the user to determine how aggressively VRmonic should check incorrect form. VRmonic enables everyday users to practice piano playing with continuous feedback while mitigating long-term physical injury.","PeriodicalId":518749,"journal":{"name":"2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)","volume":"41 7-8","pages":"330-334"},"PeriodicalIF":0.0000,"publicationDate":"2024-01-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"2024 IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/AIxVR59861.2024.00056","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Improper hand form during practice and performance can lead to long-term injuries in musicians of all levels. We present a demonstration of VRmonic, a novel immersive VR-based piano trainer for learning correct hand form for piano by overlaying the playing form of an expert oracle. VRmonic consists of a library of 48 scales comprising of the major, and natural, harmonic, and melodic minor scales for each of the 12 notes. We obtain correct hand form by using an automatic approach to detect hand keypoints from recordings of an expert pianist collected using an overhead RGB-D mounted camera. During training, the user can use VRmonic to load in a prior recording of themselves playing a scale and compare their hand form against the expert oracle. During playback, we display virtual oracle and user hand overlap via a color change, controlled by our form tolerance threshold slider that allows the user to determine how aggressively VRmonic should check incorrect form. VRmonic enables everyday users to practice piano playing with continuous feedback while mitigating long-term physical injury.
查看原文
分享 分享
微信好友 朋友圈 QQ好友 复制链接
本刊更多论文
VRmonic:VR 钢琴弹奏形式训练器
在练习和演奏过程中,不正确的手形可能会导致各种水平的音乐家长期受伤。我们展示了 VRmonic,这是一款基于虚拟现实的新型钢琴训练器,通过叠加专家的弹奏形式来学习正确的钢琴手形。VRmonic 包含一个由 48 个音阶组成的音阶库,每个音阶有 12 个音符,包括大调音阶、自然音阶、和声音阶和旋律小调音阶。我们采用一种自动方法,从钢琴演奏专家使用架空 RGB-D 摄像机采集的录音中检测手部关键点,从而获得正确的手形。在训练过程中,用户可以使用 VRmonic 加载自己先前弹奏音阶的录音,并将自己的手形与专家手形进行比较。在回放过程中,我们会通过颜色变化来显示虚拟手形和用户手形的重叠,用户可以通过手形容忍度阈值滑块来决定 VRmonic 检查错误手形的力度。VRmonic 可让日常用户通过持续反馈练习钢琴演奏,同时减轻长期的身体伤害。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 去求助
来源期刊
自引率
0.00%
发文量
0
期刊最新文献
Multimedia Retrieval in Mixed Reality: Leveraging Live Queries for Immersive Experiences TransUser's: A Transformer Based Salient Object Detection for Users Experience Generation in 360° Videos Optimizing Robotic Automatic Suturing Through VR-Enhanced Data Generation for Reinforcement Learning Algorithms Interactive Digital Twins for Online Planning and Worker Safety in Intralogistics and Production Equipment Assembly Recognition for Augmented Reality Guidance
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
已复制链接
已复制链接
快去分享给好友吧!
我知道了
×
扫码分享
扫码分享
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1