{"title":"Qualitative Research in Information Interaction: Data Gathering","authors":"Dana McKay, S. Makri, G. Buchanan","doi":"10.1145/3627508.3638291","DOIUrl":"https://doi.org/10.1145/3627508.3638291","url":null,"abstract":"","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"56 7","pages":"425-426"},"PeriodicalIF":0.0,"publicationDate":"2024-03-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140255922","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"The Dark Matter of Serendipity in Recommender Systems","authors":"Denis Kotkov, A. Medlar, Triin Kask, D. Glowacka","doi":"10.1145/3627508.3638342","DOIUrl":"https://doi.org/10.1145/3627508.3638342","url":null,"abstract":"","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"38 7","pages":"108-118"},"PeriodicalIF":0.0,"publicationDate":"2024-03-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140254926","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This demo paper presents JayBot, an LLM-based chatbot system aimed at enhancing the user experience of prospective and current students, faculty, and staff at a UK university. The objective of JayBot is to provide information to users on general enquiries regarding course modules, duration, fees, entry requirements, lecturers, internship, career paths, course employability and other related aspects. Leveraging the use cases of generative artificial intelligence (AI), the chatbot application was built using OpenAI’s advanced large language model (GPT-3.5 turbo); to tackle issues such as hallucination as well as focus and timeliness of results, an embedding transformer model has been combined with a vector database and vector search. Prompt engineering techniques were employed to enhance the chatbot’s response abilities. Preliminary user studies indicate JayBot’s effectiveness and efficiency. The demo will showcase JayBot in a university admission use case and discuss further application scenarios.
{"title":"JayBot - Aiding University Students and Admission with an LLM-based Chatbot","authors":"Julius Odede, Ingo Frommholz","doi":"10.1145/3627508.3638293","DOIUrl":"https://doi.org/10.1145/3627508.3638293","url":null,"abstract":"This demo paper presents JayBot, an LLM-based chatbot system aimed at enhancing the user experience of prospective and current students, faculty, and staff at a UK university. The objective of JayBot is to provide information to users on general enquiries regarding course modules, duration, fees, entry requirements, lecturers, internship, career paths, course employability and other related aspects. Leveraging the use cases of generative artificial intelligence (AI), the chatbot application was built using OpenAI’s advanced large language model (GPT-3.5 turbo); to tackle issues such as hallucination as well as focus and timeliness of results, an embedding transformer model has been combined with a vector database and vector search. Prompt engineering techniques were employed to enhance the chatbot’s response abilities. Preliminary user studies indicate JayBot’s effectiveness and efficiency. The demo will showcase JayBot in a university admission use case and discuss further application scenarios.","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"14 5","pages":"391-395"},"PeriodicalIF":0.0,"publicationDate":"2024-03-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140255171","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
We present a user study ( 𝑁 = 40) that investigated the role of goal-setting on learning during search. To this end, we developed a tool called the Subgoal Manager (SM). The SM was designed to help searchers break apart a learning-oriented search task into smaller subgoals. The tool enabled participants to add, delete, and modify subgoals; take notes with respect to subgoals; and mark subgoals as completed. During the study, participants completed a single learning-oriented search task and were assigned to one of two sub-goal conditions. In the Subgoals condition, participants had access to the SM; were instructed to develop at least three subgoals before the search session; and could add, delete, and modify subgoals during the search session. In the NoSubgoals condition, participants were not instructed to set subgoals and were simply provided with a text editor to take notes. We investigate the effects of the subgoal condition on: ( RQ1 ) learning and retention and ( RQ2 ) the extent to which participants engaged in specific self-regulated learning (SRL) processes during the search session. Our results found two important trends. First, participants in the Subgoals condition had better learning outcomes, especially with respect to retention. Second, based on a qualitative analysis of participants’ search sessions, participants in the Subgoals condition engaged in more self-regulated learning (SRL) processes. Combined, our results suggest that goal-setting improves learning during search by encouraging and supporting greater engagement with SRL processes.
{"title":"The Effects of Goal-setting on Learning Outcomes and Self-Regulated Learning Processes","authors":"Kelsey Urgo, Jaime Arguello","doi":"10.1145/3627508.3638348","DOIUrl":"https://doi.org/10.1145/3627508.3638348","url":null,"abstract":"We present a user study ( 𝑁 = 40) that investigated the role of goal-setting on learning during search. To this end, we developed a tool called the Subgoal Manager (SM). The SM was designed to help searchers break apart a learning-oriented search task into smaller subgoals. The tool enabled participants to add, delete, and modify subgoals; take notes with respect to subgoals; and mark subgoals as completed. During the study, participants completed a single learning-oriented search task and were assigned to one of two sub-goal conditions. In the Subgoals condition, participants had access to the SM; were instructed to develop at least three subgoals before the search session; and could add, delete, and modify subgoals during the search session. In the NoSubgoals condition, participants were not instructed to set subgoals and were simply provided with a text editor to take notes. We investigate the effects of the subgoal condition on: ( RQ1 ) learning and retention and ( RQ2 ) the extent to which participants engaged in specific self-regulated learning (SRL) processes during the search session. Our results found two important trends. First, participants in the Subgoals condition had better learning outcomes, especially with respect to retention. Second, based on a qualitative analysis of participants’ search sessions, participants in the Subgoals condition engaged in more self-regulated learning (SRL) processes. Combined, our results suggest that goal-setting improves learning during search by encouraging and supporting greater engagement with SRL processes.","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"53 7","pages":"278-290"},"PeriodicalIF":0.0,"publicationDate":"2024-03-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140255514","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A User Study on the Acceptance of Native Advertising in Generative IR","authors":"Ines Zelch, Matthias Hagen, Martin Potthast","doi":"10.1145/3627508.3638316","DOIUrl":"https://doi.org/10.1145/3627508.3638316","url":null,"abstract":"","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"65 3","pages":"142-152"},"PeriodicalIF":0.0,"publicationDate":"2024-03-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140254974","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Alexander Frummet, A. Papenmeier, Maik Fröbe, Johannes Kiesel
{"title":"The Eighth Workshop on Search-Oriented Conversational Artificial Intelligence (SCAI'24)","authors":"Alexander Frummet, A. Papenmeier, Maik Fröbe, Johannes Kiesel","doi":"10.1145/3627508.3638310","DOIUrl":"https://doi.org/10.1145/3627508.3638310","url":null,"abstract":"","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"56 6","pages":"433-435"},"PeriodicalIF":0.0,"publicationDate":"2024-03-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140255029","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Seeking Socially Responsible Consumers: Exploring the Intention-Search-Behaviour Gap","authors":"Leif Azzopardi, F. V. D. Sluis","doi":"10.1145/3627508.3638324","DOIUrl":"https://doi.org/10.1145/3627508.3638324","url":null,"abstract":"","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"52 7","pages":"153-164"},"PeriodicalIF":0.0,"publicationDate":"2024-03-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140255760","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Conversational information-seeking (CIS) is an emerging paradigm for knowledge acquisition and exploratory search. Traditional web search interfaces enable easy exploration of entities, but this is limited in conversational settings due to the limited-bandwidth interface. This paper explore ways to rewrite answers in CIS, so that users can understand them without having to resort to external services or sources. Specifically, we focus on salient entities -- entities that are central to understanding the answer. As our first contribution, we create a dataset of conversations annotated with entities for saliency. Our analysis of the collected data reveals that the majority of answers contain salient entities. As our second contribution, we propose two answer rewriting strategies aimed at improving the overall user experience in CIS. One approach expands answers with inline definitions of salient entities, making the answer self-contained. The other approach complements answers with follow-up questions, offering users the possibility to learn more about specific entities. Results of a crowdsourcing-based study indicate that rewritten answers are clearly preferred over the original ones. We also find that inline definitions tend to be favored over follow-up questions, but this choice is highly subjective, thereby providing a promising future direction for personalization.
{"title":"Towards Self-Contained Answers: Entity-Based Answer Rewriting in Conversational Search","authors":"Ivan Sekuli'c, K. Balog, Fabio Crestani","doi":"10.1145/3627508.3638300","DOIUrl":"https://doi.org/10.1145/3627508.3638300","url":null,"abstract":"Conversational information-seeking (CIS) is an emerging paradigm for knowledge acquisition and exploratory search. Traditional web search interfaces enable easy exploration of entities, but this is limited in conversational settings due to the limited-bandwidth interface. This paper explore ways to rewrite answers in CIS, so that users can understand them without having to resort to external services or sources. Specifically, we focus on salient entities -- entities that are central to understanding the answer. As our first contribution, we create a dataset of conversations annotated with entities for saliency. Our analysis of the collected data reveals that the majority of answers contain salient entities. As our second contribution, we propose two answer rewriting strategies aimed at improving the overall user experience in CIS. One approach expands answers with inline definitions of salient entities, making the answer self-contained. The other approach complements answers with follow-up questions, offering users the possibility to learn more about specific entities. Results of a crowdsourcing-based study indicate that rewritten answers are clearly preferred over the original ones. We also find that inline definitions tend to be favored over follow-up questions, but this choice is highly subjective, thereby providing a promising future direction for personalization.","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"29 5","pages":"209-218"},"PeriodicalIF":0.0,"publicationDate":"2024-03-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140266925","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as"title'',"title+image'', or"title+image+summary''. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users' diverse preferences.
{"title":"The Influence of Presentation and Performance on User Satisfaction","authors":"Kanaad Pathak, Leif Azzopardi, Martin Halvey","doi":"10.1145/3627508.3638335","DOIUrl":"https://doi.org/10.1145/3627508.3638335","url":null,"abstract":"The effectiveness of an IR system is gauged not just by its ability to retrieve relevant results but also by how it presents these results to users; an engaging presentation often correlates with increased user satisfaction. While existing research has delved into the link between user satisfaction, IR performance metrics, and presentation, these aspects have typically been investigated in isolation. Our research aims to bridge this gap by examining the relationship between query performance, presentation and user satisfaction. For our analysis, we conducted a between-subjects experiment comparing the effectiveness of various result card layouts for an ad-hoc news search interface. Drawing data from the TREC WaPo 2018 collection, we centered our study on four specific topics. Within each of these topics, we assessed six distinct queries with varying nDCG values. Our study involved 164 participants who were exposed to one of five distinct layouts containing result cards, such as\"title'',\"title+image'', or\"title+image+summary''. Our findings indicate that while nDCG is a strong predictor of user satisfaction at the query level, there exists no linear relationship between the performance of the query, presentation of results and user satisfaction. However, when considering the total gain on the initial result page, we observed that presentation does play a significant role in user satisfaction (at the query level) for certain layouts with result cards such as, title+image or title+image+summary. Our results also suggest that the layout differences have complex and multifaceted impacts on satisfaction. We demonstrate the capacity to equalize user satisfaction levels between queries of varying performance by changing how results are presented. This emphasizes the necessity to harmonize both performance and presentation in IR systems, considering users' diverse preferences.","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"171 1","pages":"77-86"},"PeriodicalIF":0.0,"publicationDate":"2024-01-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140481178","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-01-29DOI: 10.48550/arXiv.2401.16167
Selina Meyer, David Elsweiler
Conversational agents are increasingly used to address emotional needs on top of information needs. One use case of increasing interest are counselling-style mental health and behaviour change interventions, with large language model (LLM)-based approaches becoming more popular. Research in this context so far has been largely system-focused, foregoing the aspect of user behaviour and the impact this can have on LLM-generated texts. To address this issue, we share a dataset containing text-based user interactions related to behaviour change with two GPT-4-based conversational agents collected in a preregistered user study. This dataset includes conversation data, user language analysis, perception measures, and user feedback for LLM-generated turns, and can offer valuable insights to inform the design of such systems based on real interactions.
{"title":"\"You tell me\": A Dataset of GPT-4-Based Behaviour Change Support Conversations","authors":"Selina Meyer, David Elsweiler","doi":"10.48550/arXiv.2401.16167","DOIUrl":"https://doi.org/10.48550/arXiv.2401.16167","url":null,"abstract":"Conversational agents are increasingly used to address emotional needs on top of information needs. One use case of increasing interest are counselling-style mental health and behaviour change interventions, with large language model (LLM)-based approaches becoming more popular. Research in this context so far has been largely system-focused, foregoing the aspect of user behaviour and the impact this can have on LLM-generated texts. To address this issue, we share a dataset containing text-based user interactions related to behaviour change with two GPT-4-based conversational agents collected in a preregistered user study. This dataset includes conversation data, user language analysis, perception measures, and user feedback for LLM-generated turns, and can offer valuable insights to inform the design of such systems based on real interactions.","PeriodicalId":220434,"journal":{"name":"Conference on Human Information Interaction and Retrieval","volume":"72 3","pages":"411-416"},"PeriodicalIF":0.0,"publicationDate":"2024-01-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"140485792","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：