Pub Date : 2021-02-18DOI: 10.1007/s12193-021-00367-x
Yutaro Yagi, Shogo Okada, Shota Shiobara, Sota Sugimura
Presentation skills assessment is one of the central challenges of multimodal modeling. Presentation skills are composed of verbal and nonverbal skill components, but because people demonstrate their presentation skills in a variety of manners, the observed multimodal features vary widely. Due to the differences in features, when test data samples are generated on different training data sample distributions, in many cases, the prediction accuracy of the skills degrades. In machine learning theory, this problem in which training (source) data are biased is known as instance selection bias or covariate shift. To solve this problem, this paper presents an instance weighting adaptation method that is applied to estimate the presentation skills of each participant from multimodal (verbal and nonverbal) features. For this purpose, we collect a novel multimodal presentation dataset that includes audio signal data, body motion sensor data, and text data of the speech content for participants observed in 58 presentation sessions. The dataset also includes both verbal and nonverbal presentation skills, which are assessed by two external experts from a human resources department. We extract multimodal features, such as spoken utterances, acoustic features, and the amount of body motion, to estimate the presentation skills. We propose two approaches, early fusing and late fusing, for the regression models based on multimodal instance weighting adaptation. The experimental results show that the early fusing regression model with instance weighting adaptation achieved (rho =0.39) for the Pearson correlation, which presents the regression accuracy for the clarity of presentation goal elements. In the maximum case, the accuracy (correlation coefficient) is improved from (-0.34) to +0.35 by instance weighting adaptation.
{"title":"Predicting multimodal presentation skills based on instance weighting domain adaptation","authors":"Yutaro Yagi, Shogo Okada, Shota Shiobara, Sota Sugimura","doi":"10.1007/s12193-021-00367-x","DOIUrl":"https://doi.org/10.1007/s12193-021-00367-x","url":null,"abstract":"<p>Presentation skills assessment is one of the central challenges of multimodal modeling. Presentation skills are composed of verbal and nonverbal skill components, but because people demonstrate their presentation skills in a variety of manners, the observed multimodal features vary widely. Due to the differences in features, when test data samples are generated on different training data sample distributions, in many cases, the prediction accuracy of the skills degrades. In machine learning theory, this problem in which training (source) data are biased is known as instance selection bias or covariate shift. To solve this problem, this paper presents an instance weighting adaptation method that is applied to estimate the presentation skills of each participant from multimodal (verbal and nonverbal) features. For this purpose, we collect a novel multimodal presentation dataset that includes audio signal data, body motion sensor data, and text data of the speech content for participants observed in 58 presentation sessions. The dataset also includes both verbal and nonverbal presentation skills, which are assessed by two external experts from a human resources department. We extract multimodal features, such as spoken utterances, acoustic features, and the amount of body motion, to estimate the presentation skills. We propose two approaches, early fusing and late fusing, for the regression models based on multimodal instance weighting adaptation. The experimental results show that the early fusing regression model with instance weighting adaptation achieved <span>(rho =0.39)</span> for the Pearson correlation, which presents the regression accuracy for the clarity of presentation goal elements. In the maximum case, the accuracy (correlation coefficient) is improved from <span>(-0.34)</span> to +0.35 by instance weighting adaptation.</p>","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"33 1","pages":""},"PeriodicalIF":2.9,"publicationDate":"2021-02-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"138508789","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-01-26DOI: 10.1007/s12193-020-00363-7
Jun He, Xiaocui Yu, Bo Sun, Lejun Yu
{"title":"Facial expression and action unit recognition augmented by their dependencies on graph convolutional networks","authors":"Jun He, Xiaocui Yu, Bo Sun, Lejun Yu","doi":"10.1007/s12193-020-00363-7","DOIUrl":"https://doi.org/10.1007/s12193-020-00363-7","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"15 1","pages":"429 - 440"},"PeriodicalIF":2.9,"publicationDate":"2021-01-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1007/s12193-020-00363-7","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"45781026","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-01-25DOI: 10.1007/s12193-020-00358-4
K. Renuga Devi, H. Hannah Inbarani
{"title":"Neighborhood based decision theoretic rough set under dynamic granulation for BCI motor imagery classification","authors":"K. Renuga Devi, H. Hannah Inbarani","doi":"10.1007/s12193-020-00358-4","DOIUrl":"https://doi.org/10.1007/s12193-020-00358-4","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"44 1","pages":"301 - 321"},"PeriodicalIF":2.9,"publicationDate":"2021-01-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"72827662","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-01-21DOI: 10.1007/s12193-020-00356-6
Minha Lee, Gale M. Lucas, J. Gratch
{"title":"Comparing mind perception in strategic exchanges: human-agent negotiation, dictator and ultimatum games","authors":"Minha Lee, Gale M. Lucas, J. Gratch","doi":"10.1007/s12193-020-00356-6","DOIUrl":"https://doi.org/10.1007/s12193-020-00356-6","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"9 1","pages":"201 - 214"},"PeriodicalIF":2.9,"publicationDate":"2021-01-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1007/s12193-020-00356-6","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"52688271","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-01-17DOI: 10.1007/s12193-020-00361-9
Usman Malik, Mukesh Barange, Julien Saunier, A. Pauchet
{"title":"A novel focus encoding scheme for addressee detection in multiparty interaction using machine learning algorithms","authors":"Usman Malik, Mukesh Barange, Julien Saunier, A. Pauchet","doi":"10.1007/s12193-020-00361-9","DOIUrl":"https://doi.org/10.1007/s12193-020-00361-9","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"15 1","pages":"175 - 188"},"PeriodicalIF":2.9,"publicationDate":"2021-01-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1007/s12193-020-00361-9","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"42477170","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-01-02DOI: 10.1007/s12193-020-00357-5
Mohan S Zalake, F. Tavassoli, Kyle A. Duke, T. George, François Modave, J. Neil, Janice L. Krieger, B. Lok
{"title":"Internet-based tailored virtual human health intervention to promote colorectal cancer screening: design guidelines from two user studies","authors":"Mohan S Zalake, F. Tavassoli, Kyle A. Duke, T. George, François Modave, J. Neil, Janice L. Krieger, B. Lok","doi":"10.1007/s12193-020-00357-5","DOIUrl":"https://doi.org/10.1007/s12193-020-00357-5","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"15 1","pages":"147 - 162"},"PeriodicalIF":2.9,"publicationDate":"2021-01-02","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1007/s12193-020-00357-5","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"43848179","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-24DOI: 10.1007/s12193-020-00353-9
Daniel P. Davison, F. Wijnen, V. Charisi, J. van der Meij, D. Reidsma, V. Evers
{"title":"Words of encouragement: how praise delivered by a social robot changes children’s mindset for learning","authors":"Daniel P. Davison, F. Wijnen, V. Charisi, J. van der Meij, D. Reidsma, V. Evers","doi":"10.1007/s12193-020-00353-9","DOIUrl":"https://doi.org/10.1007/s12193-020-00353-9","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"15 1","pages":"61 - 76"},"PeriodicalIF":2.9,"publicationDate":"2020-11-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1007/s12193-020-00353-9","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"52688114","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-13DOI: 10.1007/s12193-020-00352-w
G. Ince, R. Yorganci, A. Ozkul, Taha Berkay Duman, Hatice Köse
{"title":"An audiovisual interface-based drumming system for multimodal human–robot interaction","authors":"G. Ince, R. Yorganci, A. Ozkul, Taha Berkay Duman, Hatice Köse","doi":"10.1007/s12193-020-00352-w","DOIUrl":"https://doi.org/10.1007/s12193-020-00352-w","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"15 1","pages":"413 - 428"},"PeriodicalIF":2.9,"publicationDate":"2020-11-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1007/s12193-020-00352-w","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"41529631","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-11-06DOI: 10.1007/s12193-020-00350-y
T. Bickmore, Everlyne Kimani, Ameneh Shamekhi, Prasanth Murali, Dhaval Parmar, H. Trinh
{"title":"Virtual agents as supporting media for scientific presentations","authors":"T. Bickmore, Everlyne Kimani, Ameneh Shamekhi, Prasanth Murali, Dhaval Parmar, H. Trinh","doi":"10.1007/s12193-020-00350-y","DOIUrl":"https://doi.org/10.1007/s12193-020-00350-y","url":null,"abstract":"","PeriodicalId":17529,"journal":{"name":"Journal on Multimodal User Interfaces","volume":"15 1","pages":"131 - 146"},"PeriodicalIF":2.9,"publicationDate":"2020-11-06","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://sci-hub-pdf.com/10.1007/s12193-020-00350-y","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"44575952","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":3,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
扫码关注我们
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号