首页 > 最新文献

Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval最新文献

英文 中文
Modeling user's receptiveness over time for recommendation 建模用户对推荐的接受程度
Wei Chen, W. Hsu, M. Lee
Existing recommender systems model user interests and the social influences independently. In reality, user interests may change over time, and as the interests change, new friends may be added while old friends grow apart and the new friendships formed may cause further interests change. This complex interaction requires the joint modeling of user interest and social relationships over time. In this paper, we propose a probabilistic generative model, called Receptiveness over Time Model (RTM), to capture this interaction. We design a Gibbs sampling algorithm to learn the receptiveness and interest distributions among users over time. The results of experiments on a real world dataset demonstrate that RTM-based recommendation outperforms the state-of-the-art recommendation methods. Case studies also show that RTM is able to discover the user interest shift and receptiveness change over time
现有的推荐系统独立地模拟用户兴趣和社会影响。在现实中,用户的兴趣可能会随着时间的推移而变化,随着兴趣的变化,可能会增加新朋友,而老朋友可能会疏远,新友谊的形成可能会导致兴趣的进一步变化。这种复杂的交互需要对用户兴趣和社交关系进行联合建模。在本文中,我们提出了一个概率生成模型,称为随时间接受模型(RTM),以捕捉这种相互作用。我们设计了一个Gibbs抽样算法来学习用户之间随时间的接受度和兴趣分布。在真实数据集上的实验结果表明,基于rtm的推荐优于最先进的推荐方法。案例研究还表明,RTM能够发现用户兴趣的变化和接受程度随时间的变化
{"title":"Modeling user's receptiveness over time for recommendation","authors":"Wei Chen, W. Hsu, M. Lee","doi":"10.1145/2484028.2484047","DOIUrl":"https://doi.org/10.1145/2484028.2484047","url":null,"abstract":"Existing recommender systems model user interests and the social influences independently. In reality, user interests may change over time, and as the interests change, new friends may be added while old friends grow apart and the new friendships formed may cause further interests change. This complex interaction requires the joint modeling of user interest and social relationships over time. In this paper, we propose a probabilistic generative model, called Receptiveness over Time Model (RTM), to capture this interaction. We design a Gibbs sampling algorithm to learn the receptiveness and interest distributions among users over time. The results of experiments on a real world dataset demonstrate that RTM-based recommendation outperforms the state-of-the-art recommendation methods. Case studies also show that RTM is able to discover the user interest shift and receptiveness change over time","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"123 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121751108","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 24
Task-aware query recommendation 任务感知查询推荐
H. Feild, James Allan
When generating query recommendations for a user, a natural approach is to try and leverage not only the user's most recently submitted query, or reference query, but also information about the current search context, such as the user's recent search interactions. We focus on two important classes of queries that make up search contexts: those that address the same information need as the reference query (on-task queries), and those that do not (off-task queries). We analyze the effects on query recommendation performance of using contexts consisting of only on-task queries, only off-task queries, and a mix of the two. Using TREC Session Track data for simulations, we demonstrate that on-task context is helpful on average but can be easily overwhelmed when off-task queries are interleaved---a common situation according to several analyses of commercial search logs. To minimize the impact of off-task queries on recommendation performance, we consider automatic methods of identifying such queries using a state of the art search task identification technique. Our experimental results show that automatic search task identification can eliminate the effect of off-task queries in a mixed context. We also introduce a novel generalized model for generating recommendations over a search context. While we only consider query text in this study, the model can handle integration over arbitrary user search behavior, such as page visits, dwell times, and query abandonment. In addition, it can be used for other types of recommendation, including personalized web search.
在为用户生成查询建议时,一种自然的方法是不仅尝试利用用户最近提交的查询或参考查询,而且还利用有关当前搜索上下文的信息,例如用户最近的搜索交互。我们将重点关注构成搜索上下文的两类重要查询:处理与参考查询相同信息需求的查询(任务上查询)和不处理相同信息需求的查询(任务下查询)。我们分析了使用仅包含任务内查询、任务外查询和两者混合的上下文对查询推荐性能的影响。使用TREC Session Track数据进行模拟,我们证明了任务内上下文通常是有帮助的,但是当任务外查询交叉时,上下文很容易被淹没——根据对商业搜索日志的一些分析,这是一种常见的情况。为了尽量减少任务外查询对推荐性能的影响,我们考虑使用最先进的搜索任务识别技术自动识别此类查询的方法。实验结果表明,自动搜索任务识别可以消除混合环境下任务外查询的影响。我们还介绍了一种新的通用模型,用于在搜索上下文中生成推荐。虽然我们在本研究中只考虑查询文本,但该模型可以处理任意用户搜索行为的集成,如页面访问、停留时间和查询放弃。此外,它还可以用于其他类型的推荐,包括个性化的网络搜索。
{"title":"Task-aware query recommendation","authors":"H. Feild, James Allan","doi":"10.1145/2484028.2484069","DOIUrl":"https://doi.org/10.1145/2484028.2484069","url":null,"abstract":"When generating query recommendations for a user, a natural approach is to try and leverage not only the user's most recently submitted query, or reference query, but also information about the current search context, such as the user's recent search interactions. We focus on two important classes of queries that make up search contexts: those that address the same information need as the reference query (on-task queries), and those that do not (off-task queries). We analyze the effects on query recommendation performance of using contexts consisting of only on-task queries, only off-task queries, and a mix of the two. Using TREC Session Track data for simulations, we demonstrate that on-task context is helpful on average but can be easily overwhelmed when off-task queries are interleaved---a common situation according to several analyses of commercial search logs. To minimize the impact of off-task queries on recommendation performance, we consider automatic methods of identifying such queries using a state of the art search task identification technique. Our experimental results show that automatic search task identification can eliminate the effect of off-task queries in a mixed context. We also introduce a novel generalized model for generating recommendations over a search context. While we only consider query text in this study, the model can handle integration over arbitrary user search behavior, such as page visits, dwell times, and query abandonment. In addition, it can be used for other types of recommendation, including personalized web search.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"42 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122054396","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 47
RecSys for distributed events: investigating the influence of recommendations on visitor plans 分布式事件的RecSys:调查推荐对访问者计划的影响
Richard Schaller, Morgan Harvey, David Elsweiler
Distributed events are collections of events taking place within a small area over the same time period and relating to a single topic. There are often a large number of events on offer and the times in which they can be visited are heavily constrained, therefore the task of choosing events to visit and in which order can be very difficult. In this work we investigate how visitors can be assisted by means of a recommender system via 2 large-scale naturalistic studies (n=860 and n=1047). We show that a recommender system can influence users to select events that result in tighter and more compact routes, thus allowing users to spend less time travelling and more time visiting events.
分布式事件是在同一时间段内与单个主题相关的小区域内发生的事件的集合。通常会有大量的活动可供选择,并且可以访问的时间受到严重限制,因此选择访问活动和顺序的任务可能非常困难。在这项工作中,我们通过两项大规模的自然主义研究(n=860和n=1047)调查了如何通过推荐系统帮助游客。我们展示了一个推荐系统可以影响用户选择事件,从而导致更紧凑和更紧凑的路线,从而允许用户花费更少的时间旅行和更多的时间参观事件。
{"title":"RecSys for distributed events: investigating the influence of recommendations on visitor plans","authors":"Richard Schaller, Morgan Harvey, David Elsweiler","doi":"10.1145/2484028.2484119","DOIUrl":"https://doi.org/10.1145/2484028.2484119","url":null,"abstract":"Distributed events are collections of events taking place within a small area over the same time period and relating to a single topic. There are often a large number of events on offer and the times in which they can be visited are heavily constrained, therefore the task of choosing events to visit and in which order can be very difficult. In this work we investigate how visitors can be assisted by means of a recommender system via 2 large-scale naturalistic studies (n=860 and n=1047). We show that a recommender system can influence users to select events that result in tighter and more compact routes, thus allowing users to spend less time travelling and more time visiting events.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"XCV 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131394161","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 17
Finding knowledgeable groups in enterprise corpora 在企业语料库中寻找知识渊博的群体
Shangsong Liang, M. de Rijke
The task of finding groups is a natural extension of search tasks aimed at retrieving individual entities. We introduce a group finding task: given a query topic, find knowledgeable groups that have expertise on that topic. We present four general strategies to this task. The models are formalized using generative language models. Two of the models aggregate expertise scores of the experts in the same group for the task, one locates documents associated with experts in the group and then determines how closely the documents are associated with the topic, whilst the remaining model directly estimates the degree to which a group is a knowledgeable group for a given topic. We construct a test collections based on the TREC 2005 and 2006 Enterprise collections. We find significant differences between different ways of estimating the association between a topic and a group. Experiments show that our knowledgeable group finding models achieve high absolute scores.
查找组的任务是旨在检索单个实体的搜索任务的自然扩展。我们引入了一个组查找任务:给定一个查询主题,查找具有该主题专业知识的知识组。为此,我们提出了四项总体策略。这些模型使用生成语言模型形式化。其中两个模型汇总同一组中专家的专业知识分数,一个定位与组中专家相关的文档,然后确定文档与主题的关联程度,而其余模型直接估计一个组在多大程度上是给定主题的知识组。我们基于TREC 2005和2006 Enterprise集合构建了一个测试集合。我们发现估算主题和群体之间关联的不同方法之间存在显著差异。实验表明,我们的知识群发现模型获得了很高的绝对分数。
{"title":"Finding knowledgeable groups in enterprise corpora","authors":"Shangsong Liang, M. de Rijke","doi":"10.1145/2484028.2484109","DOIUrl":"https://doi.org/10.1145/2484028.2484109","url":null,"abstract":"The task of finding groups is a natural extension of search tasks aimed at retrieving individual entities. We introduce a group finding task: given a query topic, find knowledgeable groups that have expertise on that topic. We present four general strategies to this task. The models are formalized using generative language models. Two of the models aggregate expertise scores of the experts in the same group for the task, one locates documents associated with experts in the group and then determines how closely the documents are associated with the topic, whilst the remaining model directly estimates the degree to which a group is a knowledgeable group for a given topic. We construct a test collections based on the TREC 2005 and 2006 Enterprise collections. We find significant differences between different ways of estimating the association between a topic and a group. Experiments show that our knowledgeable group finding models achieve high absolute scores.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"70 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131930042","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 14
Pursuing insights about healthcare utilization via geocoded search queries 通过地理编码搜索查询了解医疗保健利用情况
Shuang-Hong Yang, Ryen W. White, E. Horvitz
Mobile devices provide people with a conduit to the rich infor-mation resources of the Web. With consent, the devices can also provide streams of information about search activity and location that can be used in population studies and real-time assistance. We analyzed geotagged mobile queries in a privacy-sensitive study of potential transitions from health information search to in-world healthcare utilization. We note differences in people's health infor-mation seeking before, during, and after the appearance of evidence that a medical facility has been visited. We find that we can accu-rately estimate statistics about such potential user engagement with healthcare providers. The findings highlight the promise of using geocoded search for sensing and predicting activities in the world.
移动设备为人们提供了访问Web丰富信息资源的渠道。在征得用户同意的情况下,这些设备还可以提供有关搜索活动和位置的信息流,用于人口研究和实时援助。我们在一项隐私敏感的研究中分析了地理标记的移动查询,研究了从健康信息搜索到全球医疗保健利用的潜在转变。我们注意到,人们在访问医疗机构的证据出现之前、期间和之后寻求健康信息的差异。我们发现,我们可以准确地估计这些潜在用户与医疗保健提供者互动的统计数据。这些发现强调了使用地理编码搜索来感知和预测世界上的活动的前景。
{"title":"Pursuing insights about healthcare utilization via geocoded search queries","authors":"Shuang-Hong Yang, Ryen W. White, E. Horvitz","doi":"10.1145/2484028.2484147","DOIUrl":"https://doi.org/10.1145/2484028.2484147","url":null,"abstract":"Mobile devices provide people with a conduit to the rich infor-mation resources of the Web. With consent, the devices can also provide streams of information about search activity and location that can be used in population studies and real-time assistance. We analyzed geotagged mobile queries in a privacy-sensitive study of potential transitions from health information search to in-world healthcare utilization. We note differences in people's health infor-mation seeking before, during, and after the appearance of evidence that a medical facility has been visited. We find that we can accu-rately estimate statistics about such potential user engagement with healthcare providers. The findings highlight the promise of using geocoded search for sensing and predicting activities in the world.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"63 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133181233","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 11
Self reinforcement for important passage retrieval 自我强化的重要通道检索
Ricardo Ribeiro, Luís Marujo, David Martins de Matos, J. Neto, A. Gershman, J. Carbonell
In general, centrality-based retrieval models treat all elements of the retrieval space equally, which may reduce their effectiveness. In the specific context of extractive summarization (or important passage retrieval), this means that these models do not take into account that information sources often contain lateral issues, which are hardly as important as the description of the main topic, or are composed by mixtures of topics. We present a new two-stage method that starts by extracting a collection of key phrases that will be used to help centrality-as-relevance retrieval model. We explore several approaches to the integration of the key phrases in the centrality model. The proposed method is evaluated using different datasets that vary in noise (noisy vs clean) and language (Portuguese vs English). Results show that the best variant achieves relative performance improvements of about 31% in clean data and 18% in noisy data.
一般来说,基于中心性的检索模型对检索空间的所有元素都是平等的,这可能会降低其有效性。在抽取摘要(或重要段落检索)的特定上下文中,这意味着这些模型没有考虑到信息源通常包含横向问题,这些问题几乎没有主题的描述重要,或者由主题的混合组成。我们提出了一种新的两阶段方法,首先提取关键短语的集合,这些关键短语将用于帮助“中心即相关性”检索模型。我们探索了几种方法来整合中心性模型中的关键短语。所提出的方法使用不同的数据集进行评估,这些数据集在噪声(嘈杂的vs干净的)和语言(葡萄牙语vs英语)方面各不相同。结果表明,最佳变体在干净数据中实现了31%的相对性能提升,在噪声数据中实现了18%的相对性能提升。
{"title":"Self reinforcement for important passage retrieval","authors":"Ricardo Ribeiro, Luís Marujo, David Martins de Matos, J. Neto, A. Gershman, J. Carbonell","doi":"10.1145/2484028.2484134","DOIUrl":"https://doi.org/10.1145/2484028.2484134","url":null,"abstract":"In general, centrality-based retrieval models treat all elements of the retrieval space equally, which may reduce their effectiveness. In the specific context of extractive summarization (or important passage retrieval), this means that these models do not take into account that information sources often contain lateral issues, which are hardly as important as the description of the main topic, or are composed by mixtures of topics. We present a new two-stage method that starts by extracting a collection of key phrases that will be used to help centrality-as-relevance retrieval model. We explore several approaches to the integration of the key phrases in the centrality model. The proposed method is evaluated using different datasets that vary in noise (noisy vs clean) and language (Portuguese vs English). Results show that the best variant achieves relative performance improvements of about 31% in clean data and 18% in noisy data.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"68 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133722239","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 16
Towards retrieving relevant information graphics 对检索相关信息图形
Zhuo Li, Matthew Stagitis, S. Carberry, Kathleen F. McCoy
Information retrieval research has made significant progress in the retrieval of text documents and images. However, relatively little attention has been given to the retrieval of information graphics (non-pictorial images such as bar charts and line graphs) despite their proliferation in popular media such as newspapers and magazines. Our goal is to build a system for retrieving bar charts and line graphs that reasons about the content of the graphic itself in deciding its relevance to the user query. This paper presents the first steps toward such a system, with a focus on identifying the category of intended message of potentially relevant bar charts and line graphs. Our learned model achieves accuracy higher than 80% on a corpus of collected user queries.
信息检索研究在文本文档和图像检索方面取得了重大进展。然而,尽管诸如报纸和杂志等大众媒体大量使用信息图形(如条形图和线形图等非图画图像),但对其检索的注意相对较少。我们的目标是建立一个检索条形图和线形图的系统,该系统可以在确定图形本身的内容与用户查询的相关性时进行推理。本文介绍了迈向这样一个系统的第一步,重点是确定潜在相关柱状图和线形图的预期信息的类别。我们学习的模型在收集用户查询的语料库上实现了高于80%的准确率。
{"title":"Towards retrieving relevant information graphics","authors":"Zhuo Li, Matthew Stagitis, S. Carberry, Kathleen F. McCoy","doi":"10.1145/2484028.2484164","DOIUrl":"https://doi.org/10.1145/2484028.2484164","url":null,"abstract":"Information retrieval research has made significant progress in the retrieval of text documents and images. However, relatively little attention has been given to the retrieval of information graphics (non-pictorial images such as bar charts and line graphs) despite their proliferation in popular media such as newspapers and magazines. Our goal is to build a system for retrieving bar charts and line graphs that reasons about the content of the graphic itself in deciding its relevance to the user query. This paper presents the first steps toward such a system, with a focus on identifying the category of intended message of potentially relevant bar charts and line graphs. Our learned model achieves accuracy higher than 80% on a corpus of collected user queries.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"362 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133399570","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 13
On the measurement of test collection reliability 关于测试采集可靠性的测量
Julián Urbano, M. Marrero, Diego Martín
The reliability of a test collection is proportional to the number of queries it contains. But building a collection with many queries is expensive, so researchers have to find a balance between reliability and cost. Previous work on the measurement of test collection reliability relied on data-based approaches that contemplated random what if scenarios, and provided indicators such as swap rates and Kendall tau correlations. Generalizability Theory was proposed as an alternative founded on analysis of variance that provides reliability indicators based on statistical theory. However, these reliability indicators are hard to interpret in practice, because they do not correspond to well known indicators like Kendall tau correlation. We empirically established these relationships based on data from over 40 TREC collections, thus filling the gap in the practical interpretation of Generalizability Theory. We also review the computation of these indicators, and show that they are extremely dependent on the sample of systems and queries used, so much that the required number of queries to achieve a certain level of reliability can vary in orders of magnitude. We discuss the computation of confidence intervals for these statistics, providing a much more reliable tool to measure test collection reliability. Reflecting upon all these results, we review a wealth of TREC test collections, arguing that they are possibly not as reliable as generally accepted and that the common choice of 50 queries is insufficient even for stable rankings.
测试集合的可靠性与它包含的查询数量成正比。但是建立一个包含许多查询的集合是昂贵的,因此研究人员必须在可靠性和成本之间找到平衡。之前关于测试收集可靠性测量的工作依赖于基于数据的方法,这些方法考虑了随机的假设情景,并提供了诸如互换利率和肯德尔tau相关性等指标。概括性理论是建立在方差分析基础上的一种替代理论,它提供了基于统计理论的可靠性指标。然而,这些可靠性指标在实践中很难解释,因为它们不对应于众所周知的指标,如肯德尔tau相关。我们根据40多个TREC收集的数据建立了这些关系,从而填补了概括性理论在实际解释中的空白。我们还回顾了这些指标的计算,并表明它们非常依赖于所使用的系统和查询的样本,以至于达到一定程度的可靠性所需的查询数量可能会发生数量级的变化。我们讨论了这些统计的置信区间的计算,提供了一个更可靠的工具来测量测试集合的可靠性。考虑到所有这些结果,我们回顾了大量的TREC测试集合,认为它们可能不像普遍接受的那样可靠,而且通常选择的50个查询甚至不足以实现稳定的排名。
{"title":"On the measurement of test collection reliability","authors":"Julián Urbano, M. Marrero, Diego Martín","doi":"10.1145/2484028.2484038","DOIUrl":"https://doi.org/10.1145/2484028.2484038","url":null,"abstract":"The reliability of a test collection is proportional to the number of queries it contains. But building a collection with many queries is expensive, so researchers have to find a balance between reliability and cost. Previous work on the measurement of test collection reliability relied on data-based approaches that contemplated random what if scenarios, and provided indicators such as swap rates and Kendall tau correlations. Generalizability Theory was proposed as an alternative founded on analysis of variance that provides reliability indicators based on statistical theory. However, these reliability indicators are hard to interpret in practice, because they do not correspond to well known indicators like Kendall tau correlation. We empirically established these relationships based on data from over 40 TREC collections, thus filling the gap in the practical interpretation of Generalizability Theory. We also review the computation of these indicators, and show that they are extremely dependent on the sample of systems and queries used, so much that the required number of queries to achieve a certain level of reliability can vary in orders of magnitude. We discuss the computation of confidence intervals for these statistics, providing a much more reliable tool to measure test collection reliability. Reflecting upon all these results, we review a wealth of TREC test collections, arguing that they are possibly not as reliable as generally accepted and that the common choice of 50 queries is insufficient even for stable rankings.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"28 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114935555","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 42
Report from the NTCIR-10 1CLICK-2 Japanese subtask: baselines, upperbounds and evaluation robustness 来自ntir -10 1CLICK-2日本子任务的报告:基线、上限和评估稳健性
Makoto P. Kato, T. Sakai, Takehiro Yamamoto, Mayu Iwata
The One Click Access Task (1CLICK) of NTCIR requires systems to return a concise multi-document summary of web pages in response to a query which is assumed to have been submitted in a mobile context. Systems are evaluated based on information units (or iUnits), and are required to present important pieces of information first and to minimise the amount of text the user has to read. Using the official Japanese results of the second round of the 1CLICK task from NTCIR-10, we discuss our task setting and evaluation framework. Our analyses show that: (1) Simple baseline methods that leverage search engine snippets or Wikipedia are effective for 'lookup' type queries but not necessarily for other query types; (2) There is still a substantial gap between manual and automatic runs; and (3) Our evaluation metrics are relatively robust to the incompleteness of iUnits.
NTCIR的一键访问任务(1CLICK)要求系统在响应假定已在移动环境中提交的查询时返回一个简洁的多文档网页摘要。系统是基于信息单元(或iUnits)进行评估的,并要求首先呈现重要的信息片段,并尽量减少用户必须阅读的文本数量。使用来自ntir -10的第二轮1CLICK任务的官方日本结果,我们讨论了我们的任务设置和评估框架。我们的分析表明:(1)利用搜索引擎片段或维基百科的简单基线方法对“查找”类型查询有效,但对其他查询类型不一定有效;(2)手动运行与自动运行之间仍有较大差距;(3)我们的评估指标对iunit的不完整性相对稳健。
{"title":"Report from the NTCIR-10 1CLICK-2 Japanese subtask: baselines, upperbounds and evaluation robustness","authors":"Makoto P. Kato, T. Sakai, Takehiro Yamamoto, Mayu Iwata","doi":"10.1145/2484028.2484117","DOIUrl":"https://doi.org/10.1145/2484028.2484117","url":null,"abstract":"The One Click Access Task (1CLICK) of NTCIR requires systems to return a concise multi-document summary of web pages in response to a query which is assumed to have been submitted in a mobile context. Systems are evaluated based on information units (or iUnits), and are required to present important pieces of information first and to minimise the amount of text the user has to read. Using the official Japanese results of the second round of the 1CLICK task from NTCIR-10, we discuss our task setting and evaluation framework. Our analyses show that: (1) Simple baseline methods that leverage search engine snippets or Wikipedia are effective for 'lookup' type queries but not necessarily for other query types; (2) There is still a substantial gap between manual and automatic runs; and (3) Our evaluation metrics are relatively robust to the incompleteness of iUnits.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"512 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116700661","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 6
Match the news: a firefox extension for real-time news recommendation 匹配新闻:一个firefox扩展,用于实时新闻推荐
Margarita Karkali, Dimitris Pontikis, M. Vazirgiannis
We present Match the News, a browser extension for real time news recommendation. Our extension works on the client side to recommend in real time recently published articles that are relevant to the web page the user is currently visiting. Match the News is fed from Google News RSS and applies syntactic matching to find the relevant articles. We implement an innovative weighting function to perform the keyword extraction task, BM25H. With BM25H we extract keywords not only relevant to currently browsed web page, but also novel with respect to the user's recent browsing history. The novelty feature in keyword extraction task results in meaningful news recommendations with regards to the web page the users currently visits. Moreover the extension offers a salient visualization of the terms corresponding to the users recent browsing history making thus the extension a comprehensive tool for real time news recommendation and self assessment.
我们现在匹配的新闻,实时新闻推荐的浏览器扩展。我们的扩展工作在客户端实时推荐最近发表的文章是相关的网页用户目前正在访问。Match the News从Google News RSS提供,并应用语法匹配来查找相关文章。我们实现了一个创新的加权函数来执行关键字提取任务,BM25H。使用BM25H,我们提取的关键字不仅与当前浏览的网页相关,而且与用户最近的浏览历史有关。关键字提取任务中的新颖性功能会对用户当前访问的网页产生有意义的新闻推荐。此外,扩展提供了与用户最近浏览历史相对应的显著可视化术语,从而使扩展成为实时新闻推荐和自我评估的综合工具。
{"title":"Match the news: a firefox extension for real-time news recommendation","authors":"Margarita Karkali, Dimitris Pontikis, M. Vazirgiannis","doi":"10.1145/2484028.2484208","DOIUrl":"https://doi.org/10.1145/2484028.2484208","url":null,"abstract":"We present Match the News, a browser extension for real time news recommendation. Our extension works on the client side to recommend in real time recently published articles that are relevant to the web page the user is currently visiting. Match the News is fed from Google News RSS and applies syntactic matching to find the relevant articles. We implement an innovative weighting function to perform the keyword extraction task, BM25H. With BM25H we extract keywords not only relevant to currently browsed web page, but also novel with respect to the user's recent browsing history. The novelty feature in keyword extraction task results in meaningful news recommendations with regards to the web page the users currently visits. Moreover the extension offers a salient visualization of the terms corresponding to the users recent browsing history making thus the extension a comprehensive tool for real time news recommendation and self assessment.","PeriodicalId":178818,"journal":{"name":"Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval","volume":"29 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-07-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116707759","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 17
期刊
Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval
全部 Acc. Chem. Res. ACS Applied Bio Materials ACS Appl. Electron. Mater. ACS Appl. Energy Mater. ACS Appl. Mater. Interfaces ACS Appl. Nano Mater. ACS Appl. Polym. Mater. ACS BIOMATER-SCI ENG ACS Catal. ACS Cent. Sci. ACS Chem. Biol. ACS Chemical Health & Safety ACS Chem. Neurosci. ACS Comb. Sci. ACS Earth Space Chem. ACS Energy Lett. ACS Infect. Dis. ACS Macro Lett. ACS Mater. Lett. ACS Med. Chem. Lett. ACS Nano ACS Omega ACS Photonics ACS Sens. ACS Sustainable Chem. Eng. ACS Synth. Biol. Anal. Chem. BIOCHEMISTRY-US Bioconjugate Chem. BIOMACROMOLECULES Chem. Res. Toxicol. Chem. Rev. Chem. Mater. CRYST GROWTH DES ENERG FUEL Environ. Sci. Technol. Environ. Sci. Technol. Lett. Eur. J. Inorg. Chem. IND ENG CHEM RES Inorg. Chem. J. Agric. Food. Chem. J. Chem. Eng. Data J. Chem. Educ. J. Chem. Inf. Model. J. Chem. Theory Comput. J. Med. Chem. J. Nat. Prod. J PROTEOME RES J. Am. Chem. Soc. LANGMUIR MACROMOLECULES Mol. Pharmaceutics Nano Lett. Org. Lett. ORG PROCESS RES DEV ORGANOMETALLICS J. Org. Chem. J. Phys. Chem. J. Phys. Chem. A J. Phys. Chem. B J. Phys. Chem. C J. Phys. Chem. Lett. Analyst Anal. Methods Biomater. Sci. Catal. Sci. Technol. Chem. Commun. Chem. Soc. Rev. CHEM EDUC RES PRACT CRYSTENGCOMM Dalton Trans. Energy Environ. Sci. ENVIRON SCI-NANO ENVIRON SCI-PROC IMP ENVIRON SCI-WAT RES Faraday Discuss. Food Funct. Green Chem. Inorg. Chem. Front. Integr. Biol. J. Anal. At. Spectrom. J. Mater. Chem. A J. Mater. Chem. B J. Mater. Chem. C Lab Chip Mater. Chem. Front. Mater. Horiz. MEDCHEMCOMM Metallomics Mol. Biosyst. Mol. Syst. Des. Eng. Nanoscale Nanoscale Horiz. Nat. Prod. Rep. New J. Chem. Org. Biomol. Chem. Org. Chem. Front. PHOTOCH PHOTOBIO SCI PCCP Polym. Chem.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
0
微信
客服QQ
Book学术公众号 扫码关注我们
反馈
×
意见反馈
请填写您的意见或建议
请填写您的手机或邮箱
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
现在去查看 取消
×
提示
确定
Book学术官方微信
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术
文献互助 智能选刊 最新文献 互助须知 联系我们:info@booksci.cn
Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。
Copyright © 2023 Book学术 All rights reserved.
ghs 京公网安备 11010802042870号 京ICP备2023020795号-1