Exploring artificial intelligence-powered virtual assistants to understand their potential to support older adults’ search needs

IF 2.2 Human factors in healthcare Pub Date : 2025-06-01 Epub Date: 2025-01-18 DOI:10.1016/j.hfh.2025.100092

Emily M. Langston , Varitnan Hattakitjamroen , Mario Hernandez , Hye Soo Lee , Hannah Ç. Mason , Willencia Louis-Charles , Neil Charness , Sara J. Czaja , Wendy A. Rogers , Joseph Sharit , Walter R. Boot

{"title":"Exploring artificial intelligence-powered virtual assistants to understand their potential to support older adults’ search needs","authors":"Emily M. Langston , Varitnan Hattakitjamroen , Mario Hernandez , Hye Soo Lee , Hannah Ç. Mason , Willencia Louis-Charles , Neil Charness , Sara J. Czaja , Wendy A. Rogers , Joseph Sharit , Walter R. Boot","doi":"10.1016/j.hfh.2025.100092","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>We investigated the accuracy and amount of information provided by artificial intelligence (AI)-powered virtual assistants in response to queries relevant to aging adults in the domains of Medicare, long-term care insurance, and resource access.</div></div><div><h3>Background</h3><div>Older adults are faced with complex decisions and must gather and integrate information from diverse sources to help support these decisions (e.g., across various websites and online resources). Information-seeking, integration, and decision-making are cognitively demanding and can be impacted by age-related cognitive changes. Virtual assistants powered by AI have the potential to provide older adults with easy access to information and answers to their queries. However, it is unclear how accurate this information and these answers might be.</div></div><div><h3>Method</h3><div>Alexa, Google Assistant, Bard, and ChatGPT-4 were queried. Coders assessed the accuracy of these responses, and the amount of supplemental information provided as a measure of response complexity.</div></div><div><h3>Results</h3><div>Overall, Large Language Model (LLM)-based virtual assistants (Bard, ChatGPT-4) responded more accurately than non-LLM assistants (e.g., 6 % inaccurate responses for Bard vs. 60 % for Alexa) and provided substantially more supplemental information (79 % of responses with high supplemental information for Bard and 37 % for Chat-GPT, vs. 20 % or less for others). We note, however, that responses can vary over time.</div></div><div><h3>Conclusion</h3><div>Based on their ability to provide largely accurate responses, LLMs may be helpful tools for older adults seeking information related to health, insurance, and available resources. However, the potential for error, high response complexity, and response variability should be considered.</div></div><div><h3>Application</h3><div>LLM-based virtual assistants may be a helpful tool for older adults seeking information to support health and financial decisions.</div></div>","PeriodicalId":93564,"journal":{"name":"Human factors in healthcare","volume":"7 ","pages":"Article 100092"},"PeriodicalIF":2.2000,"publicationDate":"2025-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Human factors in healthcare","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S277250142500003X","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/1/18 0:00:00","PubModel":"Epub","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

We investigated the accuracy and amount of information provided by artificial intelligence (AI)-powered virtual assistants in response to queries relevant to aging adults in the domains of Medicare, long-term care insurance, and resource access.

Background

Older adults are faced with complex decisions and must gather and integrate information from diverse sources to help support these decisions (e.g., across various websites and online resources). Information-seeking, integration, and decision-making are cognitively demanding and can be impacted by age-related cognitive changes. Virtual assistants powered by AI have the potential to provide older adults with easy access to information and answers to their queries. However, it is unclear how accurate this information and these answers might be.

Method

Alexa, Google Assistant, Bard, and ChatGPT-4 were queried. Coders assessed the accuracy of these responses, and the amount of supplemental information provided as a measure of response complexity.

Results

Overall, Large Language Model (LLM)-based virtual assistants (Bard, ChatGPT-4) responded more accurately than non-LLM assistants (e.g., 6 % inaccurate responses for Bard vs. 60 % for Alexa) and provided substantially more supplemental information (79 % of responses with high supplemental information for Bard and 37 % for Chat-GPT, vs. 20 % or less for others). We note, however, that responses can vary over time.

Conclusion

Based on their ability to provide largely accurate responses, LLMs may be helpful tools for older adults seeking information related to health, insurance, and available resources. However, the potential for error, high response complexity, and response variability should be considered.

Application

LLM-based virtual assistants may be a helpful tool for older adults seeking information to support health and financial decisions.

查看原文

微信好友朋友圈 QQ好友复制链接

本刊更多论文

探索人工智能驱动的虚拟助手，了解它们支持老年人搜索需求的潜力

目的：我们调查了人工智能（AI）驱动的虚拟助手在回答医疗保险、长期护理保险和资源获取领域与老年人相关的查询时提供的信息的准确性和数量。年龄较大的成年人面临着复杂的决策，必须收集和整合来自不同来源的信息来帮助支持这些决策（例如，通过各种网站和在线资源）。信息寻求、整合和决策是认知要求，并可能受到年龄相关认知变化的影响。由人工智能驱动的虚拟助手有可能为老年人提供方便的信息获取和查询答案。然而，目前还不清楚这些信息和答案有多准确。方法对alexa、谷歌Assistant、Bard、ChatGPT-4进行查询。编码员评估这些响应的准确性，以及作为响应复杂性度量提供的补充信息的数量。结果总体而言，基于大型语言模型（LLM）的虚拟助手（Bard, ChatGPT-4）的反应比非LLM助手更准确（例如，Bard的反应不准确率为6%，而Alexa的反应不准确率为60%），并提供了更多的补充信息（Bard的高补充信息应答率为79%，chatgpt为37%，而其他人为20%或更少）。然而，我们注意到，反应会随着时间的推移而变化。基于llm能够提供非常准确的回答，llm可能是老年人寻求健康、保险和可用资源相关信息的有用工具。然而，应该考虑潜在的误差、高响应复杂性和响应可变性。应用法学硕士为基础的虚拟助手可能是一个有用的工具，为老年人寻求信息，以支持健康和财务决策。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文去求助

来源期刊