W. Morgan, Pi-Chuan Chang, Surabhi Gupta, Jason M. Brenier
Identification of action items in meeting recordings can provide immediate access to salient information in a medium notoriously difficult to search and summarize. To this end, we use a maximum entropy model to automatically detect action item-related utterances from multi-party audio meeting recordings. We compare the effect of lexical, temporal, syntactic, semantic, and prosodic features on system performance. We show that on a corpus of action item annotations on the ICSI meeting recordings, characterized by high imbalance and low inter-annotator agreement, the system performs at an F measure of 31.92%. While this is low compared to better-studied tasks on more mature corpora, the relative usefulness of the features towards this task is indicative of their usefulness on more consistent annotations, as well as to related tasks.
{"title":"Automatically Detecting Action Items in Audio Meeting Recordings","authors":"W. Morgan, Pi-Chuan Chang, Surabhi Gupta, Jason M. Brenier","doi":"10.3115/1654595.1654614","DOIUrl":"https://doi.org/10.3115/1654595.1654614","url":null,"abstract":"Identification of action items in meeting recordings can provide immediate access to salient information in a medium notoriously difficult to search and summarize. To this end, we use a maximum entropy model to automatically detect action item-related utterances from multi-party audio meeting recordings. We compare the effect of lexical, temporal, syntactic, semantic, and prosodic features on system performance. We show that on a corpus of action item annotations on the ICSI meeting recordings, characterized by high imbalance and low inter-annotator agreement, the system performs at an F measure of 31.92%. While this is low compared to better-studied tasks on more mature corpora, the relative usefulness of the features towards this task is indicative of their usefulness on more consistent annotations, as well as to related tasks.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"1663 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126895191","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Dialog systems for mobile robots operating in the real world should enable mixed-initiative dialog style, handle multi-modal information involved in the communication and be relatively independent of the domain knowledge. Most dialog systems developed for mobile robots today, however, are often system-oriented and have limited capabilities. We present an agent-based dialog model that are specially designed for human-robot interaction and provide evidence for its efficiency with our implemented system.
{"title":"A computational model of multi-modal grounding for human robot interaction","authors":"Shuyin Li, B. Wrede, G. Sagerer","doi":"10.3115/1654595.1654626","DOIUrl":"https://doi.org/10.3115/1654595.1654626","url":null,"abstract":"Dialog systems for mobile robots operating in the real world should enable mixed-initiative dialog style, handle multi-modal information involved in the communication and be relatively independent of the domain knowledge. Most dialog systems developed for mobile robots today, however, are often system-oriented and have limited capabilities. We present an agent-based dialog model that are specially designed for human-robot interaction and provide evidence for its efficiency with our implemented system.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"34 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116799552","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In recent years, the development of intelligent tutoring dialogue systems has become more prevalent, in an attempt to close the performance gap between human and computer tutors. Tutoring applications differ in many ways, however, from the types of applications for which spoken dialogue systems are typically developed. This talk will illustrate some of the opportunities and challenges in this area, focusing on issues such as affective reasoning, discourse and dialogue analysis, and performance evaluation.
{"title":"Discourse and Dialogue Processing in Spoken Intelligent Tutoring Systems","authors":"D. Litman","doi":"10.3115/1654595.1654624","DOIUrl":"https://doi.org/10.3115/1654595.1654624","url":null,"abstract":"In recent years, the development of intelligent tutoring dialogue systems has become more prevalent, in an attempt to close the performance gap between human and computer tutors. Tutoring applications differ in many ways, however, from the types of applications for which spoken dialogue systems are typically developed. This talk will illustrate some of the opportunities and challenges in this area, focusing on issues such as affective reasoning, discourse and dialogue analysis, and performance evaluation.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"50 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125904484","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A problem in dialogue research is that of finding and managing expectations. Adjacency pair theory has widespread acceptance, but traditional classification features (in particular, 'previous-tag' type features) do not exploit this information optimally. We suggest a method of dialogue segmentation that verifies adjacency pairs and allows us to use dialogue-level information within the entire segment and not just the previous utterance. We also use the X2 test for statistical significance as 'noise reduction' to refine a list of pairs. Together, these methods can be used to extend expectation beyond the traditional classification features.
{"title":"Empirical Verification of Adjacency Pairs Using Dialogue Segmentation","authors":"T. D. Midgley, S. Harrison, C. MacNish","doi":"10.3115/1654595.1654615","DOIUrl":"https://doi.org/10.3115/1654595.1654615","url":null,"abstract":"A problem in dialogue research is that of finding and managing expectations. Adjacency pair theory has widespread acceptance, but traditional classification features (in particular, 'previous-tag' type features) do not exploit this information optimally. We suggest a method of dialogue segmentation that verifies adjacency pairs and allows us to use dialogue-level information within the entire segment and not just the previous utterance. We also use the X2 test for statistical significance as 'noise reduction' to refine a list of pairs. Together, these methods can be used to extend expectation beyond the traditional classification features.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"6 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129229207","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We present a probabilistic approach for the interpretation of arguments that casts the selection of an interpretation as a model selection task. In selecting the best model, our formalism balances conflicting factors: model complexity against data fit, and structure complexity against belief reasonableness. We first describe our basic formalism, which considers interpretations comprising inferential relations, and then show how our formalism is extended to suppositions that account for the beliefs in an argument, and justifications that account for the inferences in an interpretation. Our evaluations with users show that the interpretations produced by our system are acceptable, and that there is strong support for the postulated suppositions and justifications.
{"title":"Balancing Conflicting Factors in Argument Interpretation","authors":"Ingrid Zukerman, M. Niemann, Sarah George","doi":"10.3115/1654595.1654621","DOIUrl":"https://doi.org/10.3115/1654595.1654621","url":null,"abstract":"We present a probabilistic approach for the interpretation of arguments that casts the selection of an interpretation as a model selection task. In selecting the best model, our formalism balances conflicting factors: model complexity against data fit, and structure complexity against belief reasonableness. We first describe our basic formalism, which considers interpretations comprising inferential relations, and then show how our formalism is extended to suppositions that account for the beliefs in an argument, and justifications that account for the inferences in an interpretation. Our evaluations with users show that the interpretations produced by our system are acceptable, and that there is strong support for the postulated suppositions and justifications.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"17 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120969768","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We present a dialogue manager for "Call for Fire" training dialogues. We describe the training environment, the domain, the features of its novel information state-based dialogue manager, the system it is a part of, and preliminary evaluation results.
{"title":"An Information State-Based Dialogue Manager for Call for Fire Dialogues","authors":"Antonio Roque, D. Traum","doi":"10.3115/1654595.1654613","DOIUrl":"https://doi.org/10.3115/1654595.1654613","url":null,"abstract":"We present a dialogue manager for \"Call for Fire\" training dialogues. We describe the training environment, the domain, the features of its novel information state-based dialogue manager, the system it is a part of, and preliminary evaluation results.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"7 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114417113","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We consider here the task of linear the-matic segmentation of text documents, by using features based on word distributions in the text. For this task, a typical and often implicit assumption in previous studies is that a document has just one topic and therefore many algorithms have been tested and have shown encouraging results on artificial data sets, generated by putting together parts of different documents. We show that evaluation on synthetic data is potentially misleading and fails to give an accurate evaluation of the performance on real data. Moreover, we provide a critical review of existing evaluation metrics in the literature and we propose an improved evaluation metric.
{"title":"An Analysis of Quantitative Aspects in the Evaluation of Thematic Segmentation Algorithms","authors":"Maria Georgescul, Alexander Clark, S. Armstrong","doi":"10.3115/1654595.1654622","DOIUrl":"https://doi.org/10.3115/1654595.1654622","url":null,"abstract":"We consider here the task of linear the-matic segmentation of text documents, by using features based on word distributions in the text. For this task, a typical and often implicit assumption in previous studies is that a document has just one topic and therefore many algorithms have been tested and have shown encouraging results on artificial data sets, generated by putting together parts of different documents. We show that evaluation on synthetic data is potentially misleading and fails to give an accurate evaluation of the performance on real data. Moreover, we provide a critical review of existing evaluation metrics in the literature and we propose an improved evaluation metric.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"42 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123191250","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ben Wellner, J. Pustejovsky, Catherine Havasi, Anna Rumshisky, R. Saurí
In this paper we consider the problem of identifying and classifying discourse coherence relations. We report initial results over the recently released Discourse GraphBank (Wolf and Gibson, 2005). Our approach considers, and determines the contributions of, a variety of syntactic and lexico-semantic features. We achieve 81% accuracy on the task of discourse relation type classification and 70% accuracy on relation identification.
本文主要研究语篇连贯关系的识别和分类问题。我们报告了最近发布的Discourse GraphBank的初步结果(Wolf and Gibson, 2005)。我们的方法考虑并决定了各种句法和词汇语义特征的贡献。我们在篇章关系类型分类任务上达到了81%的准确率，在关系识别任务上达到了70%的准确率。
{"title":"Classification of Discourse Coherence Relations: An Exploratory Study using Multiple Knowledge Sources","authors":"Ben Wellner, J. Pustejovsky, Catherine Havasi, Anna Rumshisky, R. Saurí","doi":"10.3115/1654595.1654618","DOIUrl":"https://doi.org/10.3115/1654595.1654618","url":null,"abstract":"In this paper we consider the problem of identifying and classifying discourse coherence relations. We report initial results over the recently released Discourse GraphBank (Wolf and Gibson, 2005). Our approach considers, and determines the contributions of, a variety of syntactic and lexico-semantic features. We achieve 81% accuracy on the task of discourse relation type classification and 70% accuracy on relation identification.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"81 4","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"120933031","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We present a first analysis of inter-annotator agreement for the DIT++ tagset of dialogue acts, a comprehensive, layered, multidimensional set of 86 tags. Within a dimension or a layer, subsets of tags are often hierarchically organised. We argue that especially for such highly structured annotation schemes the well-known kappa statistic is not an adequate measure of inter-annotator agreement. Instead, we propose a statistic that takes the structural properties of the tagset into account, and we discuss the application of this statistic in an annotation experiment. The experiment shows promising agreement scores for most dimensions in the tagset and provides useful insights into the usability of the annotation scheme, but also indicates that several additional factors influence annotator agreement. We finally suggest that the proposed approach for measuring agreement per dimension can be a good basis for measuring annotator agreement over the dimensions of a multidimensional annotation scheme.
{"title":"Measuring annotator agreement in a complex hierarchical dialogue act annotation scheme","authors":"J. Geertzen, H. Bunt","doi":"10.3115/1654595.1654619","DOIUrl":"https://doi.org/10.3115/1654595.1654619","url":null,"abstract":"We present a first analysis of inter-annotator agreement for the DIT++ tagset of dialogue acts, a comprehensive, layered, multidimensional set of 86 tags. Within a dimension or a layer, subsets of tags are often hierarchically organised. We argue that especially for such highly structured annotation schemes the well-known kappa statistic is not an adequate measure of inter-annotator agreement. Instead, we propose a statistic that takes the structural properties of the tagset into account, and we discuss the application of this statistic in an annotation experiment. The experiment shows promising agreement scores for most dimensions in the tagset and provides useful insights into the usability of the annotation scheme, but also indicates that several additional factors influence annotator agreement. We finally suggest that the proposed approach for measuring agreement per dimension can be a good basis for measuring annotator agreement over the dimensions of a multidimensional annotation scheme.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"55 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"126841369","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
In this paper, we explain a rapid development method of multimodal dialogue sys-tem using MIML (Multimodal Interaction Markup Language), which defines dialogue patterns between human and various types of interactive agents. The feature of this language is three-layered description of agent-based interactive systems which separates task level description, interaction description and device dependent realization. MIML has advantages in high-level interaction description, modality extensibility and compatibility with standardized technologies.
{"title":"Multimodal Dialog Description Language for Rapid System Development","authors":"Masahiro Araki, K. Tachibana","doi":"10.3115/1654595.1654616","DOIUrl":"https://doi.org/10.3115/1654595.1654616","url":null,"abstract":"In this paper, we explain a rapid development method of multimodal dialogue sys-tem using MIML (Multimodal Interaction Markup Language), which defines dialogue patterns between human and various types of interactive agents. The feature of this language is three-layered description of agent-based interactive systems which separates task level description, interaction description and device dependent realization. MIML has advantages in high-level interaction description, modality extensibility and compatibility with standardized technologies.","PeriodicalId":426429,"journal":{"name":"SIGDIAL Workshop","volume":"18 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2009-07-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125143402","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：