Spoken and multimodal dialogue systems typically make use of confidence scores to choose among (or reject) a speech recognizer's N-best hypotheses for a particular utterance. We argue that it is beneficial to instead choose among a list of candidate system responses. We propose a novel method in which a confidence score for each response is derived from a classifier trained on acoustic and lexical features emitted by the recognizer, as well as features culled from the generation of the candidate response itself. Our response-based method yields statistically significant improvements in F-measure over a baseline in which hypotheses are chosen based on recognition confidence scores only.
{"title":"Response-Based Confidence Annotation for Spoken Dialogue Systems","authors":"A. Gruenstein","doi":"10.3115/1622064.1622067","DOIUrl":"https://doi.org/10.3115/1622064.1622067","url":null,"abstract":"Spoken and multimodal dialogue systems typically make use of confidence scores to choose among (or reject) a speech recognizer's N-best hypotheses for a particular utterance. We argue that it is beneficial to instead choose among a list of candidate system responses. We propose a novel method in which a confidence score for each response is derived from a classifier trained on acoustic and lexical features emitted by the recognizer, as well as features culled from the generation of the candidate response itself. Our response-based method yields statistically significant improvements in F-measure over a baseline in which hypotheses are chosen based on recognition confidence scores only.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2008-06-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123744274","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Evaluating a dialogue system is seen as a major challenge within the dialogue research community. Due to the very nature of the task, most of the evaluation methods need a substantial amount of human involvement. Following the tradition in machine translation, summarization and discourse coherence modeling, we introduce the the idea of evaluation understudy for dialogue coherence models. Following (Lapata, 2006), we use the information ordering task as a testbed for evaluating dialogue coherence models. This paper reports findings about the reliability of the information ordering task as applied to dialogues. We find that simple n-gram co-occurrence statistics similar in spirit to BLEU (Papineni et al., 2001) correlate very well with human judgments for dialogue coherence
评估对话系统被视为对话研究界的一个主要挑战。由于任务的性质，大多数评估方法需要大量的人力参与。继机器翻译、摘要和语篇连贯建模的传统之后，我们引入了评价替代的思想来建立对话连贯模型。接下来(Lapata, 2006)，我们使用信息排序任务作为评估对话一致性模型的测试平台。本文报道了应用于对话的信息排序任务的可靠性研究结果。我们发现，在精神上与BLEU相似的简单n-gram共现统计(Papineni et al.， 2001)与人类对对话连贯性的判断非常相关
{"title":"Evaluation Understudy for Dialogue Coherence Models","authors":"Sudeep Gandhe, D. Traum","doi":"10.3115/1622064.1622098","DOIUrl":"https://doi.org/10.3115/1622064.1622098","url":null,"abstract":"Evaluating a dialogue system is seen as a major challenge within the dialogue research community. Due to the very nature of the task, most of the evaluation methods need a substantial amount of human involvement. Following the tradition in machine translation, summarization and discourse coherence modeling, we introduce the the idea of evaluation understudy for dialogue coherence models. Following (Lapata, 2006), we use the information ordering task as a testbed for evaluating dialogue coherence models. This paper reports findings about the reliability of the information ordering task as applied to dialogues. We find that simple n-gram co-occurrence statistics similar in spirit to BLEU (Papineni et al., 2001) correlate very well with human judgments for dialogue coherence","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2008-06-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128828106","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A dialogue system can present itself and/or address the user as an active agent by means of linguistic constructions in personal style, or suppress agentivity by using impersonal style. We compare system evaluation judgments and input style alignment of users interacting with an in-car dialogue system generating output in personal vs. impersonal style. Although our results are consistent with earlier findings obtained with simulated systems, the effects are weaker.
{"title":"The Effect of Dialogue System Output Style Variation on Users’ Evaluation Judgments and Input Style","authors":"Ivana Kruijff-Korbayová, O. Kukina","doi":"10.3115/1622064.1622101","DOIUrl":"https://doi.org/10.3115/1622064.1622101","url":null,"abstract":"A dialogue system can present itself and/or address the user as an active agent by means of linguistic constructions in personal style, or suppress agentivity by using impersonal style. We compare system evaluation judgments and input style alignment of users interacting with an in-car dialogue system generating output in personal vs. impersonal style. Although our results are consistent with earlier findings obtained with simulated systems, the effects are weaker.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2008-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128050528","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper investigates the problems facing modelling agents' beliefs in Discourse Representation Theory (DRT) and presents a viable solution in the form of a dialogue-based DRT representation of beliefs. Integrating modelling dialogue interaction into DRT allows modelling agents' beliefs, intentions and mutual beliefs. Furthermore, it is one of the aims of the paper to account for the important notion of agents' varying degrees of belief in different contexts.
{"title":"DRT Representation of Degrees of Belief","authors":"Yafa Al-Raheb","doi":"10.3115/1654595.1654607","DOIUrl":"https://doi.org/10.3115/1654595.1654607","url":null,"abstract":"This paper investigates the problems facing modelling agents' beliefs in Discourse Representation Theory (DRT) and presents a viable solution in the form of a dialogue-based DRT representation of beliefs. Integrating modelling dialogue interaction into DRT allows modelling agents' beliefs, intentions and mutual beliefs. Furthermore, it is one of the aims of the paper to account for the important notion of agents' varying degrees of belief in different contexts.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122557003","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper we deal with learning and forgetting of speech commands in speech dialogue systems. We discuss two mathematical models for learning and four models for forgetting. Furthermore, we describe the experiments used to determine the learning and forgetting curve in our environment. Our findings are compared to the theoretical models and based on this we deduce which models best describe learning and forgetting in our automotive environment. The resulting models are used to develop an adaptive help system for a speech dialogue system. The system provides only relevant context specific information.
{"title":"Adaptive Help for Speech Dialogue Systems Based on Learning and Forgetting of Speech Commands","authors":"A. Hof, E. Hagen, Alexander Huber","doi":"10.3115/1654595.1654597","DOIUrl":"https://doi.org/10.3115/1654595.1654597","url":null,"abstract":"In this paper we deal with learning and forgetting of speech commands in speech dialogue systems. We discuss two mathematical models for learning and four models for forgetting. Furthermore, we describe the experiments used to determine the learning and forgetting curve in our environment. Our findings are compared to the theoretical models and based on this we deduce which models best describe learning and forgetting in our automotive environment. The resulting models are used to develop an adaptive help system for a speech dialogue system. The system provides only relevant context specific information.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"291 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133178243","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Integration of new utterances into context is a central task in any model for rational (human-machine) dialogues in natural language. In this paper, a pragmatics-first approach to specifying the meaning of utterances in terms of plans is presented. A rational dialogue is driven by the reaction of dialogue participants on how they find their expectations on changes in the environment satisfied by their observations of the outcome of performed actions. We present a computational model for this view on dialogues and illustrate it with examples from a real-world application.
{"title":"Tracing Actions Helps in Understanding Interactions","authors":"Bernd Ludwig","doi":"10.3115/1654595.1654609","DOIUrl":"https://doi.org/10.3115/1654595.1654609","url":null,"abstract":"Integration of new utterances into context is a central task in any model for rational (human-machine) dialogues in natural language. In this paper, a pragmatics-first approach to specifying the meaning of utterances in terms of plans is presented. A rational dialogue is driven by the reaction of dialogue participants on how they find their expectations on changes in the environment satisfied by their observations of the outcome of performed actions. We present a computational model for this view on dialogues and illustrate it with examples from a real-world application.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"51 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132141960","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper presents an extension to the Reference Domain Theory (Salmon-Alt, 2001) in order to solve plural references. While this theory doesn't take plural reference into account in its original form, this paper shows how several entities can be grouped together by building a new domain and how they can be accessed later on. We introduce the notion of super-domain, representing the access structure to all the plural referents of a given type.
{"title":"Resolution of Referents Groupings in Practical Dialogues","authors":"Alexandre Denis, Guillaume Pitel, M. Quignard","doi":"10.3115/1654595.1654608","DOIUrl":"https://doi.org/10.3115/1654595.1654608","url":null,"abstract":"This paper presents an extension to the Reference Domain Theory (Salmon-Alt, 2001) in order to solve plural references. While this theory doesn't take plural reference into account in its original form, this paper shows how several entities can be grouped together by building a new domain and how they can be accessed later on. We introduce the notion of super-domain, representing the access structure to all the plural referents of a given type.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121687457","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Leuski, Ronakkumar Patel, D. Traum, Brandon Kennedy
In this paper, we describe methods for building and evaluation of limited domain question-answering characters. Several classification techniques are tested, including text classification using support vector machines, language-model based retrieval, and cross-language information retrieval techniques, with the latter having the highest success rate. We also evaluated the effect of speech recognition errors on performance with users, finding that retrieval is robust until recognition reaches over 50% WER.
{"title":"Building Effective Question Answering Characters","authors":"A. Leuski, Ronakkumar Patel, D. Traum, Brandon Kennedy","doi":"10.3115/1654595.1654600","DOIUrl":"https://doi.org/10.3115/1654595.1654600","url":null,"abstract":"In this paper, we describe methods for building and evaluation of limited domain question-answering characters. Several classification techniques are tested, including text classification using support vector machines, language-model based retrieval, and cross-language information retrieval techniques, with the latter having the highest success rate. We also evaluated the effect of speech recognition errors on performance with users, finding that retrieval is robust until recognition reaches over 50% WER.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"77 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129134682","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Deciding what is the content of an utterance in dialogue is a potentially tricky business: should it be an entity computed using (solely/primarily) grammatical information or is it determined by recognition of participant intention using domain level inference? The decisions one makes on this score play a crucial role in any model of the interaction involved in grounding an utterance. Integrating the clarificatory potential of an utterance into the grounding process transforms the issue of content recognition into a more concrete issue: grammatically determined content has markedly distinct clarificatory potential from content determined using domain level inference. This leads to a new challenge: how to integrate the two types of content in such a way that both enables their distinct clarificatory potential to be maintained and allows content determined by domain level inference to feature in grounding. My talk will address this challenge.
{"title":"Content Recognition in Dialogue","authors":"J. Ginzburg","doi":"10.3115/1654595.1654603","DOIUrl":"https://doi.org/10.3115/1654595.1654603","url":null,"abstract":"Deciding what is the content of an utterance in dialogue is a potentially tricky business: should it be an entity computed using (solely/primarily) grammatical information or is it determined by recognition of participant intention using domain level inference? The decisions one makes on this score play a crucial role in any model of the interaction involved in grounding an utterance. Integrating the clarificatory potential of an utterance into the grounding process transforms the issue of content recognition into a more concrete issue: grammatically determined content has markedly distinct clarificatory potential from content determined using domain level inference. This leads to a new challenge: how to integrate the two types of content in such a way that both enables their distinct clarificatory potential to be maintained and allows content determined by domain level inference to feature in grounding. My talk will address this challenge.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"426 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2006-07-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127604119","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：