Rudolf Kadlec, Jindřich Libovický, Jan Macek, Jan Kleindienst
Accurate dialog state tracking is crucial for the design of an efficient spoken dialog system. Until recently, quantitative comparison of different state tracking methods was difficult. However the 2013 Dialog State Tracking Challenge (DSTC) introduced a common dataset and metrics that allow to evaluate the performance of trackers on a standardized task. In this paper we present our belief tracker based on the Hidden Information State (HIS) model with an adjusted user model component. Further, we report the results of our tracker on test3 dataset from DSTC. Our tracker is competitive with trackers submitted to DSTC, even without training it achieves the best results in L2 metrics and it performs between second and third place in accuracy. After adjusting the tracker using the provided data it outperformed the other submissions also in accuracy and yet improved in L2. Additionally we present preliminary results on another two datasets, test1 and test2, used in the DSTC. Strong performance in L2 metric means that our tracker produces well calibrated hypotheses probabilities.
准确的对话状态跟踪是设计高效口语对话系统的关键。直到最近，对不同状态跟踪方法的定量比较还是很困难的。然而，2013年对话状态跟踪挑战(DSTC)引入了一个通用的数据集和指标，允许评估跟踪器在标准化任务中的性能。本文提出了一种基于HIS (Hidden Information State)模型的信念跟踪器，该模型具有调整后的用户模型组件。此外，我们报告了我们的跟踪器在DSTC的test3数据集上的结果。我们的跟踪器与提交给DSTC的跟踪器相比具有竞争力，即使没有经过培训，它在L2指标中也取得了最好的结果，并且在准确性方面表现在第二到第三名之间。在使用提供的数据调整跟踪器后，它在准确性上也优于其他提交，但在L2中有所提高。此外，我们还介绍了DSTC中使用的另外两个数据集test1和test2的初步结果。L2指标的强劲表现意味着我们的跟踪器产生了校准良好的假设概率。
{"title":"IBM’s Belief Tracker: Results On Dialog State Tracking Challenge Datasets","authors":"Rudolf Kadlec, Jindřich Libovický, Jan Macek, Jan Kleindienst","doi":"10.3115/v1/W14-0202","DOIUrl":"https://doi.org/10.3115/v1/W14-0202","url":null,"abstract":"Accurate dialog state tracking is crucial for the design of an efficient spoken dialog system. Until recently, quantitative comparison of different state tracking methods was difficult. However the 2013 Dialog State Tracking Challenge (DSTC) introduced a common dataset and metrics that allow to evaluate the performance of trackers on a standardized task. In this paper we present our belief tracker based on the Hidden Information State (HIS) model with an adjusted user model component. Further, we report the results of our tracker on test3 dataset from DSTC. Our tracker is competitive with trackers submitted to DSTC, even without training it achieves the best results in L2 metrics and it performs between second and third place in accuracy. After adjusting the tracker using the provided data it outperformed the other submissions also in accuracy and yet improved in L2. Additionally we present preliminary results on another two datasets, test1 and test2, used in the DSTC. Strong performance in L2 metric means that our tracker produces well calibrated hypotheses probabilities.","PeriodicalId":198983,"journal":{"name":"DM@EACL","volume":"41 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123316772","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Navigation of blind people is different from the navigation of sighted people and there is also difference when the blind person is recovering from getting lost. In this paper we focus on qualitative analysis of dialogs between lost blind person and navigator, which is done through the mobile phone. The research was done in two outdoor and one indoor location. The analysis revealed several areas where the dialog model must focus on detailed information, like evaluation of instructions provided by blind person and his/her ability to reliably locate navigation points.
{"title":"Navigation Dialog of Blind People: Recovery from Getting Lost","authors":"J. Vystrcil, I. Maly, Jan Balata, Z. Míkovec","doi":"10.3115/v1/W14-0210","DOIUrl":"https://doi.org/10.3115/v1/W14-0210","url":null,"abstract":"Navigation of blind people is different from the navigation of sighted people and there is also difference when the blind person is recovering from getting lost. In this paper we focus on qualitative analysis of dialogs between lost blind person and navigator, which is done through the mobile phone. The research was done in two outdoor and one indoor location. The analysis revealed several areas where the dialog model must focus on detailed information, like evaluation of instructions provided by blind person and his/her ability to reliably locate navigation points.","PeriodicalId":198983,"journal":{"name":"DM@EACL","volume":"53 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129705124","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Mobile Internet access via smartphones puts demands on in-car infotainment systems, as more and more drivers like to access the Internet while driving. Spoken dialog systems support the user by less distracting interaction than visual/hapticbased dialog systems. To develop an intuitive and usable spoken dialog system, an extensive analysis of the interaction concept is necessary. We conducted a Wizard of Oz study to investigate how users will carry out tasks which involve multiple applications in a speech-only, user-initiative infotainment system while driving. Results show that users are not aware of different applications and use anaphoric expressions in task switches. Speaking styles vary and depend on type of task and dialog state. Users interact efficiently and provide multiple semantic concepts in one utterance. This sets high demands for future spoken dialog systems.
{"title":"In-Car Multi-Domain Spoken Dialogs: A Wizard of Oz Study","authors":"Sven Reichel, U. Ehrlich, A. Berton, M. Weber","doi":"10.3115/v1/W14-0201","DOIUrl":"https://doi.org/10.3115/v1/W14-0201","url":null,"abstract":"Mobile Internet access via smartphones puts demands on in-car infotainment systems, as more and more drivers like to access the Internet while driving. Spoken dialog systems support the user by less distracting interaction than visual/hapticbased dialog systems. To develop an intuitive and usable spoken dialog system, an extensive analysis of the interaction concept is necessary. We conducted a Wizard of Oz study to investigate how users will carry out tasks which involve multiple applications in a speech-only, user-initiative infotainment system while driving. Results show that users are not aware of different applications and use anaphoric expressions in task switches. Speaking styles vary and depend on type of task and dialog state. Users interact efficiently and provide multiple semantic concepts in one utterance. This sets high demands for future spoken dialog systems.","PeriodicalId":198983,"journal":{"name":"DM@EACL","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2014-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130423831","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：