Pub Date : 2022-06-01DOI: 10.48550/arXiv.2206.00167
A. Papangelis, Nicole Chartier, Pankaj Rajan, J. Hirschberg, Dilek Z. Hakkani-Tür
User ratings play a significant role in spoken dialogue systems. Typically, such ratings tend to be averaged across all users and then utilized as feedback to improve the system or personalize its behavior. While this method can be useful to understand broad, general issues with the system and its behavior, it does not take into account differences between users that affect their ratings. In this work, we conduct a study to better understand how people rate their interactions with conversational agents. One macro-level characteristic that has been shown to cor-relate with how people perceive their inter-personal communication is personality [1, 2, 12]. We specifically focus on agreeableness and extraversion as variables that may explain variation in ratings and therefore provide a more meaningful signal for training or personalization. In order to elicit those personality traits during an interaction with a conversational agent, we designed and validated a fictional story, grounded in prior work in psychology. We then implemented the story into an experimental conversational agent that allowed users to opt-in to hearing the story. Our results suggest that for human-conversational agent interactions, extraversion may play a role in user ratings, but more data is needed to determine if the relationship is significant. Agreeableness, on the other hand, plays a statistically significant role in conversation ratings: users who are more agreeable are more likely to provide a higher rating for their interaction. In addition, we found that users who opted to hear the story were, in general, more likely to rate their conversational experience higher than those who did not.
{"title":"Understanding How People Rate Their Conversations","authors":"A. Papangelis, Nicole Chartier, Pankaj Rajan, J. Hirschberg, Dilek Z. Hakkani-Tür","doi":"10.48550/arXiv.2206.00167","DOIUrl":"https://doi.org/10.48550/arXiv.2206.00167","url":null,"abstract":"User ratings play a significant role in spoken dialogue systems. Typically, such ratings tend to be averaged across all users and then utilized as feedback to improve the system or personalize its behavior. While this method can be useful to understand broad, general issues with the system and its behavior, it does not take into account differences between users that affect their ratings. In this work, we conduct a study to better understand how people rate their interactions with conversational agents. One macro-level characteristic that has been shown to cor-relate with how people perceive their inter-personal communication is personality [1, 2, 12]. We specifically focus on agreeableness and extraversion as variables that may explain variation in ratings and therefore provide a more meaningful signal for training or personalization. In order to elicit those personality traits during an interaction with a conversational agent, we designed and validated a fictional story, grounded in prior work in psychology. We then implemented the story into an experimental conversational agent that allowed users to opt-in to hearing the story. Our results suggest that for human-conversational agent interactions, extraversion may play a role in user ratings, but more data is needed to determine if the relationship is significant. Agreeableness, on the other hand, plays a statistically significant role in conversation ratings: users who are more agreeable are more likely to provide a higher rating for their interaction. In addition, we found that users who opted to hear the story were, in general, more likely to rate their conversational experience higher than those who did not.","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"38 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133260592","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Design Guidelines for Developing Systems for Dialogue System Competitions","authors":"Kazunori Komatani, Ryu Takeda, Keisuke Nakashima, Mikio Nakano","doi":"10.1527/tjsai.37-3_ids-b","DOIUrl":"https://doi.org/10.1527/tjsai.37-3_ids-b","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"47 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2022-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115134746","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-10-15DOI: 10.1007/978-981-19-5538-9_7
Lena Reed, Cecilia Li, Angela Ramirez, Liren Wu, M. Walker
{"title":"Jurassic is (almost) All You Need: Few-Shot Meaning-to-Text Generation for Open-Domain Dialogue","authors":"Lena Reed, Cecilia Li, Angela Ramirez, Liren Wu, M. Walker","doi":"10.1007/978-981-19-5538-9_7","DOIUrl":"https://doi.org/10.1007/978-981-19-5538-9_7","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"51 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124656754","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-10-05DOI: 10.1007/978-981-19-5538-9_21
Chen Zhang, L. F. D’Haro, Yiming Chen, Thomas Friedrichs, Haizhou Li
{"title":"Investigating the Impact of Pre-trained Language Models on Dialog Evaluation","authors":"Chen Zhang, L. F. D’Haro, Yiming Chen, Thomas Friedrichs, Haizhou Li","doi":"10.1007/978-981-19-5538-9_21","DOIUrl":"https://doi.org/10.1007/978-981-19-5538-9_21","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-10-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126596349","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-09-17DOI: 10.1007/978-981-19-5538-9_6
S. Pandey, Svetlana Stoyanchev, R. Doddipatla
{"title":"Towards Handling Unconstrained User Preferences in Dialogue","authors":"S. Pandey, Svetlana Stoyanchev, R. Doddipatla","doi":"10.1007/978-981-19-5538-9_6","DOIUrl":"https://doi.org/10.1007/978-981-19-5538-9_6","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"194 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-09-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122934497","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-09-07DOI: 10.1007/978-981-19-5538-9_5
Ye Liu, Wolfgang Maier, W. Minker, Stefan Ultes
{"title":"Empathetic Dialogue Generation with Pre-trained RoBERTa-GPT2 and External Knowledge","authors":"Ye Liu, Wolfgang Maier, W. Minker, Stefan Ultes","doi":"10.1007/978-981-19-5538-9_5","DOIUrl":"https://doi.org/10.1007/978-981-19-5538-9_5","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-09-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117019411","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-04-30DOI: 10.1007/978-981-19-5538-9_1
Peng Liu, Kun Li, H. Meng
{"title":"Out-of-Scope Domain and Intent Classification through Hierarchical Joint Modeling","authors":"Peng Liu, Kun Li, H. Meng","doi":"10.1007/978-981-19-5538-9_1","DOIUrl":"https://doi.org/10.1007/978-981-19-5538-9_1","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-04-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129818963","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2021-03-11DOI: 10.1007/978-981-15-9323-9_13
Nicolas Wagner, Matthias Kraus, Niklas Rach, W. Minker
{"title":"How to Address Humans: System Barge-In in Multi-user HRI","authors":"Nicolas Wagner, Matthias Kraus, Niklas Rach, W. Minker","doi":"10.1007/978-981-15-9323-9_13","DOIUrl":"https://doi.org/10.1007/978-981-15-9323-9_13","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"39 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2021-03-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123069644","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-25DOI: 10.1007/978-981-15-8395-7_31
D. Griol, Zoraida Callejas Carrión
{"title":"Conversational Systems Research in Spain: A Scientometric Approach","authors":"D. Griol, Zoraida Callejas Carrión","doi":"10.1007/978-981-15-8395-7_31","DOIUrl":"https://doi.org/10.1007/978-981-15-8395-7_31","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134028649","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2020-10-25DOI: 10.1007/978-981-15-8395-7_13
R. Justo, Leila Ben Letaifa, J. M. Olaso, Asier López-Zorrilla, M. deVelasco, A. Vázquez, M. I. Torres
{"title":"A Spanish Corpus for Talking to the Elderly","authors":"R. Justo, Leila Ben Letaifa, J. M. Olaso, Asier López-Zorrilla, M. deVelasco, A. Vázquez, M. I. Torres","doi":"10.1007/978-981-15-8395-7_13","DOIUrl":"https://doi.org/10.1007/978-981-15-8395-7_13","url":null,"abstract":"","PeriodicalId":200919,"journal":{"name":"International Workshop on Spoken Dialogue Systems Technology","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2020-10-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132863797","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：