We contribute an analysis of situated visualizations in motion in video games for different types of data, with a focus on quantitative and categorical data representations. Video games convey a lot of data to players, to help them succeed in the game. These visualizations frequently move across the screen due to camera changes or because the game elements themselves move. Our ultimate goal is to understand how motion factors affect visualization readability in video games and subsequently the players' performance in the game. We started our work by surveying the characteristics of how motion currently influences which kind of data representations in video games. We conducted a systematic review of 160 visualizations in motion in video games and extracted patterns and considerations regarding was what, and how visualizations currently exhibit motion factors in video games.
{"title":"Visualization in Motion in Video Games for Different Types of Data","authors":"Federica Bucchieri, Lijie Yao, Petra Isenberg","doi":"arxiv-2409.07696","DOIUrl":"https://doi.org/arxiv-2409.07696","url":null,"abstract":"We contribute an analysis of situated visualizations in motion in video games\u0000for different types of data, with a focus on quantitative and categorical data\u0000representations. Video games convey a lot of data to players, to help them\u0000succeed in the game. These visualizations frequently move across the screen due\u0000to camera changes or because the game elements themselves move. Our ultimate\u0000goal is to understand how motion factors affect visualization readability in\u0000video games and subsequently the players' performance in the game. We started\u0000our work by surveying the characteristics of how motion currently influences\u0000which kind of data representations in video games. We conducted a systematic\u0000review of 160 visualizations in motion in video games and extracted patterns\u0000and considerations regarding was what, and how visualizations currently exhibit\u0000motion factors in video games.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"18 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183424","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jiashuo Cao, Wujie Gao, Yun Suen Pai, Simon Hoermann, Chen Li, Nilufar Baghaei, Mark Billinghurst
The advent of technology-enhanced interventions has significantly transformed mental health services, offering new opportunities for delivering psychotherapy, particularly in remote settings. This paper reports on a pilot study exploring the use of Virtual Reality (VR) as a medium for remote counselling. The study involved four experienced psychotherapists who evaluated three different virtual environments designed to support remote counselling. Through thematic analysis of interviews and feedback, we identified key factors that could be critical for designing effective virtual environments for counselling. These include the creation of clear boundaries, customization to meet specific therapeutic needs, and the importance of aligning the environment with various therapeutic approaches. Our findings suggest that VR can enhance the sense of presence and engagement in remote therapy, potentially improving the therapeutic relationship. In the paper we also outline areas for future research based on these pilot study results.
{"title":"Explorations in Designing Virtual Environments for Remote Counselling","authors":"Jiashuo Cao, Wujie Gao, Yun Suen Pai, Simon Hoermann, Chen Li, Nilufar Baghaei, Mark Billinghurst","doi":"arxiv-2409.07765","DOIUrl":"https://doi.org/arxiv-2409.07765","url":null,"abstract":"The advent of technology-enhanced interventions has significantly transformed\u0000mental health services, offering new opportunities for delivering\u0000psychotherapy, particularly in remote settings. This paper reports on a pilot\u0000study exploring the use of Virtual Reality (VR) as a medium for remote\u0000counselling. The study involved four experienced psychotherapists who evaluated\u0000three different virtual environments designed to support remote counselling.\u0000Through thematic analysis of interviews and feedback, we identified key factors\u0000that could be critical for designing effective virtual environments for\u0000counselling. These include the creation of clear boundaries, customization to\u0000meet specific therapeutic needs, and the importance of aligning the environment\u0000with various therapeutic approaches. Our findings suggest that VR can enhance\u0000the sense of presence and engagement in remote therapy, potentially improving\u0000the therapeutic relationship. In the paper we also outline areas for future\u0000research based on these pilot study results.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"3 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183422","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
As data visualization gains popularity and projects become more interdisciplinary, there is a growing need for methods that foster creative collaboration and inform diverse audiences about data visualisation. In this paper, we introduce Co-Badge, a 90-minute design activity where participants collaboratively construct visualizations by ideating and prioritizing relevant data types, mapping them to visual variables, and constructing data badges with stationery materials. We conducted three workshops in diverse settings with participants of different backgrounds. Our findings indicate that Co-badge facilitates a playful and engaging way to gain awareness about data visualization design principles without formal training while navigating the challenges of collaboration. Our work contributes to the field of data visualization education for diverse actors. We believe Co-Badge can serve as an engaging activity that introduces basic concepts of data visualization and collaboration.
{"title":"Co-badge: An Activity for Collaborative Engagement with Data Visualization Design Concepts","authors":"Damla Çay, Mary Karyda, Kitti Butter","doi":"arxiv-2409.08175","DOIUrl":"https://doi.org/arxiv-2409.08175","url":null,"abstract":"As data visualization gains popularity and projects become more\u0000interdisciplinary, there is a growing need for methods that foster creative\u0000collaboration and inform diverse audiences about data visualisation. In this\u0000paper, we introduce Co-Badge, a 90-minute design activity where participants\u0000collaboratively construct visualizations by ideating and prioritizing relevant\u0000data types, mapping them to visual variables, and constructing data badges with\u0000stationery materials. We conducted three workshops in diverse settings with\u0000participants of different backgrounds. Our findings indicate that Co-badge\u0000facilitates a playful and engaging way to gain awareness about data\u0000visualization design principles without formal training while navigating the\u0000challenges of collaboration. Our work contributes to the field of data\u0000visualization education for diverse actors. We believe Co-Badge can serve as an\u0000engaging activity that introduces basic concepts of data visualization and\u0000collaboration.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"22 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183414","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Hanqiu Wang, Zihao Zhan, Haoqi Shan, Siqi Dai, Max Panoff, Shuo Wang
The advent and growing popularity of Virtual Reality (VR) and Mixed Reality (MR) solutions have revolutionized the way we interact with digital platforms. The cutting-edge gaze-controlled typing methods, now prevalent in high-end models of these devices, e.g., Apple Vision Pro, have not only improved user experience but also mitigated traditional keystroke inference attacks that relied on hand gestures, head movements and acoustic side-channels. However, this advancement has paradoxically given birth to a new, potentially more insidious cyber threat, GAZEploit. In this paper, we unveil GAZEploit, a novel eye-tracking based attack specifically designed to exploit these eye-tracking information by leveraging the common use of virtual appearances in VR applications. This widespread usage significantly enhances the practicality and feasibility of our attack compared to existing methods. GAZEploit takes advantage of this vulnerability to remotely extract gaze estimations and steal sensitive keystroke information across various typing scenarios-including messages, passwords, URLs, emails, and passcodes. Our research, involving 30 participants, achieved over 80% accuracy in keystroke inference. Alarmingly, our study also identified over 15 top-rated apps in the Apple Store as vulnerable to the GAZEploit attack, emphasizing the urgent need for bolstered security measures for this state-of-the-art VR/MR text entry method.
{"title":"GAZEploit: Remote Keystroke Inference Attack by Gaze Estimation from Avatar Views in VR/MR Devices","authors":"Hanqiu Wang, Zihao Zhan, Haoqi Shan, Siqi Dai, Max Panoff, Shuo Wang","doi":"arxiv-2409.08122","DOIUrl":"https://doi.org/arxiv-2409.08122","url":null,"abstract":"The advent and growing popularity of Virtual Reality (VR) and Mixed Reality\u0000(MR) solutions have revolutionized the way we interact with digital platforms.\u0000The cutting-edge gaze-controlled typing methods, now prevalent in high-end\u0000models of these devices, e.g., Apple Vision Pro, have not only improved user\u0000experience but also mitigated traditional keystroke inference attacks that\u0000relied on hand gestures, head movements and acoustic side-channels. However,\u0000this advancement has paradoxically given birth to a new, potentially more\u0000insidious cyber threat, GAZEploit. In this paper, we unveil GAZEploit, a novel eye-tracking based attack\u0000specifically designed to exploit these eye-tracking information by leveraging\u0000the common use of virtual appearances in VR applications. This widespread usage\u0000significantly enhances the practicality and feasibility of our attack compared\u0000to existing methods. GAZEploit takes advantage of this vulnerability to\u0000remotely extract gaze estimations and steal sensitive keystroke information\u0000across various typing scenarios-including messages, passwords, URLs, emails,\u0000and passcodes. Our research, involving 30 participants, achieved over 80%\u0000accuracy in keystroke inference. Alarmingly, our study also identified over 15\u0000top-rated apps in the Apple Store as vulnerable to the GAZEploit attack,\u0000emphasizing the urgent need for bolstered security measures for this\u0000state-of-the-art VR/MR text entry method.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"11 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183415","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Recent advances in eXplainable AI (XAI) for education have highlighted a critical challenge: ensuring that explanations for state-of-the-art AI models are understandable for non-technical users such as educators and students. In response, we introduce iLLuMinaTE, a zero-shot, chain-of-prompts LLM-XAI pipeline inspired by Miller's cognitive model of explanation. iLLuMinaTE is designed to deliver theory-driven, actionable feedback to students in online courses. iLLuMinaTE navigates three main stages - causal connection, explanation selection, and explanation presentation - with variations drawing from eight social science theories (e.g. Abnormal Conditions, Pearl's Model of Explanation, Necessity and Robustness Selection, Contrastive Explanation). We extensively evaluate 21,915 natural language explanations of iLLuMinaTE extracted from three LLMs (GPT-4o, Gemma2-9B, Llama3-70B), with three different underlying XAI methods (LIME, Counterfactuals, MC-LIME), across students from three diverse online courses. Our evaluation involves analyses of explanation alignment to the social science theory, understandability of the explanation, and a real-world user preference study with 114 university students containing a novel actionability simulation. We find that students prefer iLLuMinaTE explanations over traditional explainers 89.52% of the time. Our work provides a robust, ready-to-use framework for effectively communicating hybrid XAI-driven insights in education, with significant generalization potential for other human-centric fields.
{"title":"From Explanations to Action: A Zero-Shot, Theory-Driven LLM Framework for Student Performance Feedback","authors":"Vinitra Swamy, Davide Romano, Bhargav Srinivasa Desikan, Oana-Maria Camburu, Tanja Käser","doi":"arxiv-2409.08027","DOIUrl":"https://doi.org/arxiv-2409.08027","url":null,"abstract":"Recent advances in eXplainable AI (XAI) for education have highlighted a\u0000critical challenge: ensuring that explanations for state-of-the-art AI models\u0000are understandable for non-technical users such as educators and students. In\u0000response, we introduce iLLuMinaTE, a zero-shot, chain-of-prompts LLM-XAI\u0000pipeline inspired by Miller's cognitive model of explanation. iLLuMinaTE is\u0000designed to deliver theory-driven, actionable feedback to students in online\u0000courses. iLLuMinaTE navigates three main stages - causal connection,\u0000explanation selection, and explanation presentation - with variations drawing\u0000from eight social science theories (e.g. Abnormal Conditions, Pearl's Model of\u0000Explanation, Necessity and Robustness Selection, Contrastive Explanation). We\u0000extensively evaluate 21,915 natural language explanations of iLLuMinaTE\u0000extracted from three LLMs (GPT-4o, Gemma2-9B, Llama3-70B), with three different\u0000underlying XAI methods (LIME, Counterfactuals, MC-LIME), across students from\u0000three diverse online courses. Our evaluation involves analyses of explanation\u0000alignment to the social science theory, understandability of the explanation,\u0000and a real-world user preference study with 114 university students containing\u0000a novel actionability simulation. We find that students prefer iLLuMinaTE\u0000explanations over traditional explainers 89.52% of the time. Our work provides\u0000a robust, ready-to-use framework for effectively communicating hybrid\u0000XAI-driven insights in education, with significant generalization potential for\u0000other human-centric fields.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"1 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183301","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Eike Schneiders, Tina Seabrooke, Joshua Krook, Richard Hyde, Natalie Leesakul, Jeremie Clos, Joel Fischer
Large Language Models (LLMs) are seemingly infiltrating every domain, and the legal context is no exception. In this paper, we present the results of three experiments (total N=288) that investigated lay people's willingness to act upon, and their ability to discriminate between, LLM- and lawyer-generated legal advice. In Experiment 1, participants judged their willingness to act on legal advice when the source of the advice was either known or unknown. When the advice source was unknown, participants indicated that they were significantly more willing to act on the LLM-generated advice. This result was replicated in Experiment 2. Intriguingly, despite participants indicating higher willingness to act on LLM-generated advice in Experiments 1 and 2, participants discriminated between the LLM- and lawyer-generated texts significantly above chance-level in Experiment 3. Lastly, we discuss potential explanations and risks of our findings, limitations and future work, and the importance of language complexity and real-world comparability.
{"title":"Objection Overruled! Lay People can Distinguish Large Language Models from Lawyers, but still Favour Advice from an LLM","authors":"Eike Schneiders, Tina Seabrooke, Joshua Krook, Richard Hyde, Natalie Leesakul, Jeremie Clos, Joel Fischer","doi":"arxiv-2409.07871","DOIUrl":"https://doi.org/arxiv-2409.07871","url":null,"abstract":"Large Language Models (LLMs) are seemingly infiltrating every domain, and the\u0000legal context is no exception. In this paper, we present the results of three\u0000experiments (total N=288) that investigated lay people's willingness to act\u0000upon, and their ability to discriminate between, LLM- and lawyer-generated\u0000legal advice. In Experiment 1, participants judged their willingness to act on\u0000legal advice when the source of the advice was either known or unknown. When\u0000the advice source was unknown, participants indicated that they were\u0000significantly more willing to act on the LLM-generated advice. This result was\u0000replicated in Experiment 2. Intriguingly, despite participants indicating\u0000higher willingness to act on LLM-generated advice in Experiments 1 and 2,\u0000participants discriminated between the LLM- and lawyer-generated texts\u0000significantly above chance-level in Experiment 3. Lastly, we discuss potential\u0000explanations and risks of our findings, limitations and future work, and the\u0000importance of language complexity and real-world comparability.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"23 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183418","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Service robots are increasingly deployed in public spaces, performing functional tasks such as making deliveries. To better integrate them into our social environment and enhance their adoption, we consider integrating social identities within delivery robots along with their functional identity. We conducted a virtual reality-based pilot study to explore people's perceptions and acceptance of delivery robots that perform prosocial behavior. Preliminary findings from thematic analysis of semi-structured interviews illustrate people's ambivalence about dual identity. We discussed the emerging themes in light of social identity theory, framing effect, and human-robot intergroup dynamics. Building on these insights, we propose that the next generation of delivery robots should use peer-based framing, an updated value proposition, and an interactive design that places greater emphasis on expressing intentionality and emotional responses.
{"title":"More than just a Tool: People's Perception and Acceptance of Prosocial Delivery Robots as Fellow Road Users","authors":"Vivienne Bihe Chi, Elise Ulwelling, Kevin Salubre, Shashank Mehrotra, Teruhisa Misu, Kumar Akash","doi":"arxiv-2409.07815","DOIUrl":"https://doi.org/arxiv-2409.07815","url":null,"abstract":"Service robots are increasingly deployed in public spaces, performing\u0000functional tasks such as making deliveries. To better integrate them into our\u0000social environment and enhance their adoption, we consider integrating social\u0000identities within delivery robots along with their functional identity. We\u0000conducted a virtual reality-based pilot study to explore people's perceptions\u0000and acceptance of delivery robots that perform prosocial behavior. Preliminary\u0000findings from thematic analysis of semi-structured interviews illustrate\u0000people's ambivalence about dual identity. We discussed the emerging themes in\u0000light of social identity theory, framing effect, and human-robot intergroup\u0000dynamics. Building on these insights, we propose that the next generation of\u0000delivery robots should use peer-based framing, an updated value proposition,\u0000and an interactive design that places greater emphasis on expressing\u0000intentionality and emotional responses.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"7 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183421","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sijie Zhuo, Robert Biddle, Jared Daniel Recomendable, Giovanni Russello, Danielle Lottridge
Phishing emails typically masquerade themselves as reputable identities to trick people into providing sensitive information and credentials. Despite advancements in cybersecurity, attackers continuously adapt, posing ongoing threats to individuals and organisations. While email users are the last line of defence, they are not always well-prepared to detect phishing emails. This study examines how workload affects susceptibility to phishing, using eye-tracking technology to observe participants' reading patterns and interactions with tailored phishing emails. Incorporating both quantitative and qualitative analysis, we investigate users' attention to two phishing indicators, email sender and hyperlink URLs, and their reasons for assessing the trustworthiness of emails and falling for phishing emails. Our results provide concrete evidence that attention to the email sender can reduce phishing susceptibility. While we found no evidence that attention to the actual URL in the browser influences phishing detection, attention to the text masking links can increase phishing susceptibility. We also highlight how email relevance, familiarity, and visual presentation impact first impressions of email trustworthiness and phishing susceptibility.
{"title":"Eyes on the Phish(er): Towards Understanding Users' Email Processing Pattern and Mental Models in Phishing Detection","authors":"Sijie Zhuo, Robert Biddle, Jared Daniel Recomendable, Giovanni Russello, Danielle Lottridge","doi":"arxiv-2409.07717","DOIUrl":"https://doi.org/arxiv-2409.07717","url":null,"abstract":"Phishing emails typically masquerade themselves as reputable identities to\u0000trick people into providing sensitive information and credentials. Despite\u0000advancements in cybersecurity, attackers continuously adapt, posing ongoing\u0000threats to individuals and organisations. While email users are the last line\u0000of defence, they are not always well-prepared to detect phishing emails. This\u0000study examines how workload affects susceptibility to phishing, using\u0000eye-tracking technology to observe participants' reading patterns and\u0000interactions with tailored phishing emails. Incorporating both quantitative and\u0000qualitative analysis, we investigate users' attention to two phishing\u0000indicators, email sender and hyperlink URLs, and their reasons for assessing\u0000the trustworthiness of emails and falling for phishing emails. Our results\u0000provide concrete evidence that attention to the email sender can reduce\u0000phishing susceptibility. While we found no evidence that attention to the\u0000actual URL in the browser influences phishing detection, attention to the text\u0000masking links can increase phishing susceptibility. We also highlight how email\u0000relevance, familiarity, and visual presentation impact first impressions of\u0000email trustworthiness and phishing susceptibility.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"49 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183423","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Rhoslyn Roebuck Williams, Jonathan Barnoud, Luis Toledo, Till Holzapfel, David R. Glowacki
Molecular dynamics (MD) simulations provide crucial insight into molecular interactions and biomolecular function. With interactive MD simulations in VR (iMD-VR), chemists can now interact with these molecular simulations in real-time. Our sense of touch is essential for exploring the properties of physical objects, but recreating this sensory experience for virtual objects poses challenges. Furthermore, employing haptics in the context of molecular simulation is especially difficult since textit{we do not know what molecules actually feel like}. In this paper, we build upon previous work that demonstrated how VR-users can distinguish properties of molecules without haptic feedback. We present the results of a gamified two-alternative forced choice (2AFC) psychophysics user study in which we quantify the threshold at which iMD-VR users can differentiate the stiffness of molecular bonds. Our preliminary analysis suggests that participants can sense differences between buckminsterfullerene molecules with different bond stiffness parameters and that this limit may fall within the chemically relevant range. Our results highlight how iMD-VR may facilitate a more embodied way of exploring complex and dynamic molecular systems, enabling chemists to sense the properties of molecules purely by interacting with them in VR.
{"title":"Measuring the limit of perception of bond stiffness of interactive molecules in VR via a gamified psychophysics experiment","authors":"Rhoslyn Roebuck Williams, Jonathan Barnoud, Luis Toledo, Till Holzapfel, David R. Glowacki","doi":"arxiv-2409.07836","DOIUrl":"https://doi.org/arxiv-2409.07836","url":null,"abstract":"Molecular dynamics (MD) simulations provide crucial insight into molecular\u0000interactions and biomolecular function. With interactive MD simulations in VR\u0000(iMD-VR), chemists can now interact with these molecular simulations in\u0000real-time. Our sense of touch is essential for exploring the properties of\u0000physical objects, but recreating this sensory experience for virtual objects\u0000poses challenges. Furthermore, employing haptics in the context of molecular\u0000simulation is especially difficult since textit{we do not know what molecules\u0000actually feel like}. In this paper, we build upon previous work that\u0000demonstrated how VR-users can distinguish properties of molecules without\u0000haptic feedback. We present the results of a gamified two-alternative forced\u0000choice (2AFC) psychophysics user study in which we quantify the threshold at\u0000which iMD-VR users can differentiate the stiffness of molecular bonds. Our\u0000preliminary analysis suggests that participants can sense differences between\u0000buckminsterfullerene molecules with different bond stiffness parameters and\u0000that this limit may fall within the chemically relevant range. Our results\u0000highlight how iMD-VR may facilitate a more embodied way of exploring complex\u0000and dynamic molecular systems, enabling chemists to sense the properties of\u0000molecules purely by interacting with them in VR.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"33 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183419","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
This paper explores the efficacy of online versus offline evaluation methods in assessing conversational chatbots, specifically comparing first-party direct interactions with third-party observational assessments. By extending a benchmarking dataset of user dialogs with empathetic chatbots with offline third-party evaluations, we present a systematic comparison between the feedback from online interactions and the more detached offline third-party evaluations. Our results reveal that offline human evaluations fail to capture the subtleties of human-chatbot interactions as effectively as online assessments. In comparison, automated third-party evaluations using a GPT-4 model offer a better approximation of first-party human judgments given detailed instructions. This study highlights the limitations of third-party evaluations in grasping the complexities of user experiences and advocates for the integration of direct interaction feedback in conversational AI evaluation to enhance system development and user satisfaction.
{"title":"Online vs Offline: A Comparative Study of First-Party and Third-Party Evaluations of Social Chatbots","authors":"Ekaterina Svikhnushina, Pearl Pu","doi":"arxiv-2409.07823","DOIUrl":"https://doi.org/arxiv-2409.07823","url":null,"abstract":"This paper explores the efficacy of online versus offline evaluation methods\u0000in assessing conversational chatbots, specifically comparing first-party direct\u0000interactions with third-party observational assessments. By extending a\u0000benchmarking dataset of user dialogs with empathetic chatbots with offline\u0000third-party evaluations, we present a systematic comparison between the\u0000feedback from online interactions and the more detached offline third-party\u0000evaluations. Our results reveal that offline human evaluations fail to capture\u0000the subtleties of human-chatbot interactions as effectively as online\u0000assessments. In comparison, automated third-party evaluations using a GPT-4\u0000model offer a better approximation of first-party human judgments given\u0000detailed instructions. This study highlights the limitations of third-party\u0000evaluations in grasping the complexities of user experiences and advocates for\u0000the integration of direct interaction feedback in conversational AI evaluation\u0000to enhance system development and user satisfaction.","PeriodicalId":501541,"journal":{"name":"arXiv - CS - Human-Computer Interaction","volume":"23 1","pages":""},"PeriodicalIF":0.0,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142183420","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
京公网安备 11010802042870号
京ICP备2023020795号-1
扫码关注我们
求助内容：
应助结果提醒方式：