A. Said, Jimmy J. Lin, Alejandro Bellogín, A. D. Vries
During the last decade, recommender systems have become a ubiquitous feature in the online world. Research on systems and algorithms in this area has flourished, leading to novel techniques for personalization and recommendation. The evaluation of recommender systems, however, has not seen similar progress---techniques have changed little since the advent of recommender systems, when evaluation methodologies were "borrowed" from related research areas. As an effort to move evaluation methodology forward, this paper describes a production recommender system infrastructure that allows research systems to be evaluated in situ, by real-world metrics such as user clickthrough. We present an analysis of one month of interactions with this infrastructure and share our findings.
{"title":"A month in the life of a production news recommender system","authors":"A. Said, Jimmy J. Lin, Alejandro Bellogín, A. D. Vries","doi":"10.1145/2513150.2513159","DOIUrl":"https://doi.org/10.1145/2513150.2513159","url":null,"abstract":"During the last decade, recommender systems have become a ubiquitous feature in the online world. Research on systems and algorithms in this area has flourished, leading to novel techniques for personalization and recommendation. The evaluation of recommender systems, however, has not seen similar progress---techniques have changed little since the advent of recommender systems, when evaluation methodologies were \"borrowed\" from related research areas. As an effort to move evaluation methodology forward, this paper describes a production recommender system infrastructure that allows research systems to be evaluated in situ, by real-world metrics such as user clickthrough. We present an analysis of one month of interactions with this infrastructure and share our findings.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"10 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128398206","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Melanie Imhof, Martin Braschler, P. Hansen, Stefan Rietberger
Black box information retrieval (IR) application evaluation allows practitioners to measure the quality of their IR application. Instead of evaluating specific components, e.g. solely the search engine, a complete IR application, including the user's perspective, is evaluated. The evaluation methodology is designed to be applicable to operational IR applications. The black box evaluation methodology could be packaged into an evaluation and monitoring tool, making it usable for industry stakeholders. The tool should lead practitioners through the evaluation process and maintain the test results for the manual and automatic tests. This paper shows that the methodology is generalizable, even though the diversity of IR applications is high. The challenges in automating tests are the simulation of tasks that require intellectual effort and the handling of different visualizations of the same concept.
{"title":"Evaluation for operational IR applications: generalizability and automation","authors":"Melanie Imhof, Martin Braschler, P. Hansen, Stefan Rietberger","doi":"10.1145/2513150.2513160","DOIUrl":"https://doi.org/10.1145/2513150.2513160","url":null,"abstract":"Black box information retrieval (IR) application evaluation allows practitioners to measure the quality of their IR application. Instead of evaluating specific components, e.g. solely the search engine, a complete IR application, including the user's perspective, is evaluated. The evaluation methodology is designed to be applicable to operational IR applications. The black box evaluation methodology could be packaged into an evaluation and monitoring tool, making it usable for industry stakeholders. The tool should lead practitioners through the evaluation process and maintain the test results for the manual and automatic tests. This paper shows that the methodology is generalizable, even though the diversity of IR applications is high. The challenges in automating tests are the simulation of tasks that require intellectual effort and the handling of different visualizations of the same concept.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"68 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124915751","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A major hurdle faced by many information retrieval researchers---especially in academia---is evaluating retrieval systems in the wild. Challenges include tapping into large user bases, collecting user behavior, and modifying a given retrieval system. We outline several options available to researchers to overcome these challenges along with their advantages and disadvantages. We then demonstrate how CrowdLogger, an open-source browser extension for Firefox and Google Chrome, can be used as an in situ evaluation platform.
{"title":"Using CrowdLogger for in situ information retrieval system evaluation","authors":"H. Feild, James Allan","doi":"10.1145/2513150.2513164","DOIUrl":"https://doi.org/10.1145/2513150.2513164","url":null,"abstract":"A major hurdle faced by many information retrieval researchers---especially in academia---is evaluating retrieval systems in the wild. Challenges include tapping into large user bases, collecting user behavior, and modifying a given retrieval system. We outline several options available to researchers to overcome these challenges along with their advantages and disadvantages. We then demonstrate how CrowdLogger, an open-source browser extension for Firefox and Google Chrome, can be used as an in situ evaluation platform.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"2000 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"121074343","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Information Retrieval has a long tradition of being metrics driven. Ranking algorithms are assessed with respect to some utility measure that reflects the likelihood of satisfying an information need. Traditionally these metrics are based on offline judgments. This is very flexible since judgments can be made for any desired output. However, judgments are no better than judgment guidelines and are at some distance from the actual user experience. Modern Web Search engines enjoy an additional resource; existing web search traffic and its attendant wealth of user engagement data. Primarily this signal consists of logged queries and user actions, including clicks and reformulations. I will discuss how this data can be used to derive Web Search quality metrics that have very different properties than traditional offline metrics.
{"title":"Online metrics for web search relevance","authors":"Jan O. Pedersen","doi":"10.1145/2513150.2513165","DOIUrl":"https://doi.org/10.1145/2513150.2513165","url":null,"abstract":"Information Retrieval has a long tradition of being metrics driven. Ranking algorithms are assessed with respect to some utility measure that reflects the likelihood of satisfying an information need. Traditionally these metrics are based on offline judgments. This is very flexible since judgments can be made for any desired output. However, judgments are no better than judgment guidelines and are at some distance from the actual user experience. Modern Web Search engines enjoy an additional resource; existing web search traffic and its attendant wealth of user engagement data. Primarily this signal consists of logged queries and user actions, including clicks and reformulations. I will discuss how this data can be used to derive Web Search quality metrics that have very different properties than traditional offline metrics.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"40 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123606145","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Christian Beutenmüller, Stefan Bordag, Ramin Assadollahi
A "Living Lab" is described as an open innovation space for the cooperation of users, researchers and even companies to participate in a common process to develop innovative solutions. An architecture for a living lab for IR has been proposed in [1]. In this paper we propose a method and system that foregoes the inherent openness of the living lab and implements a private living lab to enable the cooperation between a research department, agile software development, and requirements engineering/quality assurance. This allows the research department to overcome the limitations of the usual approach to gold standard-based evaluation, while preserving its positive aspects. The definition of a private living lab may be seen along the lines of the separation of public and private clouds in cloud computing.
{"title":"A private living lab for requirements based evaluation","authors":"Christian Beutenmüller, Stefan Bordag, Ramin Assadollahi","doi":"10.1145/2513150.2513158","DOIUrl":"https://doi.org/10.1145/2513150.2513158","url":null,"abstract":"A \"Living Lab\" is described as an open innovation space for the cooperation of users, researchers and even companies to participate in a common process to develop innovative solutions. An architecture for a living lab for IR has been proposed in [1]. In this paper we propose a method and system that foregoes the inherent openness of the living lab and implements a private living lab to enable the cooperation between a research department, agile software development, and requirements engineering/quality assurance.\u0000 This allows the research department to overcome the limitations of the usual approach to gold standard-based evaluation, while preserving its positive aspects. The definition of a private living lab may be seen along the lines of the separation of public and private clouds in cloud computing.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"25 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132753030","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper contemplates some of the challenges faced in recruiting and developing a community of contributors (participants/subjects) for a living laboratory for IR evaluation. We briefly review several factors that may affect the efficacy of participant recruiting. The potential benefits of collaboration with respect to recruiting are also discussed briefly.
{"title":"Factors affecting conditions of trust in participant recruiting and retention: a position paper","authors":"Catherine L. Smith","doi":"10.1145/2513150.2513161","DOIUrl":"https://doi.org/10.1145/2513150.2513161","url":null,"abstract":"This paper contemplates some of the challenges faced in recruiting and developing a community of contributors (participants/subjects) for a living laboratory for IR evaluation. We briefly review several factors that may affect the efficacy of participant recruiting. The potential benefits of collaboration with respect to recruiting are also discussed briefly.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"46 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127231868","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Anne Schuth, Katja Hofmann, Shimon Whiteson, M. de Rijke
Online learning to rank methods for IR allow retrieval systems to optimize their own performance directly from interactions with users via click feedback. In the software package Lerot, presented in this paper, we have bundled all ingredients needed for experimenting with online learning to rank for IR. Lerot includes several online learning algorithms, interleaving methods and a full suite of ways to evaluate these methods. In the absence of real users, the evaluation method bundled in the software package is based on simulations of users interacting with the search engine. The software presented here has been used to verify findings of over six papers at major information retrieval venues over the last few years.
{"title":"Lerot: an online learning to rank framework","authors":"Anne Schuth, Katja Hofmann, Shimon Whiteson, M. de Rijke","doi":"10.1145/2513150.2513162","DOIUrl":"https://doi.org/10.1145/2513150.2513162","url":null,"abstract":"Online learning to rank methods for IR allow retrieval systems to optimize their own performance directly from interactions with users via click feedback. In the software package Lerot, presented in this paper, we have bundled all ingredients needed for experimenting with online learning to rank for IR. Lerot includes several online learning algorithms, interleaving methods and a full suite of ways to evaluate these methods. In the absence of real users, the evaluation method bundled in the software package is based on simulations of users interacting with the search engine. The software presented here has been used to verify findings of over six papers at major information retrieval venues over the last few years.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"35 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123635767","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We describe a novel preference-driven search engine (FindiLike) which allows users to find entities of interest based on preferences and also allows users to digest opinions about the retrieved entities easily. FindiLike leverages large amounts of online reviews about various entities, and ranks entities based on how well their associated reviews match a user's preference query (expressed in keywords). FindiLike then uses abstractive summarization techniques to generate concise opinion summaries to enable users to digest the opinions about an entity. We discuss how the system can be extended to support in situ evaluation of two interesting new tasks, i.e., opinion-based entity ranking and abstractive summarization of opinions. The system is currently supporting hotel search and being extended to support in situ evaluation of these two tasks. We will demonstrate the system in the domain of hotel search and show how in situ evaluation can be supported through natural user interaction with the system.
{"title":"FindiLike: a preference driven entity search engine for evaluating entity retrieval and opinion summarization","authors":"Kavita A. Ganesan, ChengXiang Zhai","doi":"10.1145/2513150.2513163","DOIUrl":"https://doi.org/10.1145/2513150.2513163","url":null,"abstract":"We describe a novel preference-driven search engine (FindiLike) which allows users to find entities of interest based on preferences and also allows users to digest opinions about the retrieved entities easily. FindiLike leverages large amounts of online reviews about various entities, and ranks entities based on how well their associated reviews match a user's preference query (expressed in keywords). FindiLike then uses abstractive summarization techniques to generate concise opinion summaries to enable users to digest the opinions about an entity. We discuss how the system can be extended to support in situ evaluation of two interesting new tasks, i.e., opinion-based entity ranking and abstractive summarization of opinions. The system is currently supporting hotel search and being extended to support in situ evaluation of these two tasks. We will demonstrate the system in the domain of hotel search and show how in situ evaluation can be supported through natural user interaction with the system.","PeriodicalId":436800,"journal":{"name":"LivingLab '13","volume":"149 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132213291","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：