Pub Date : 2024-09-30eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000612
Scott H Lee, Shannon Fox, Raheem Smith, Kimberly A Skrobarcek, Harold Keyserling, Christina R Phares, Deborah Lee, Drew L Posey
Immigrants and refugees seeking admission to the United States must first undergo an overseas medical exam, overseen by the US Centers for Disease Control and Prevention (CDC), during which all persons ≥15 years old receive a chest x-ray to look for signs of tuberculosis. Although individual screening sites often implement quality control (QC) programs to ensure radiographs are interpreted correctly, the CDC does not currently have a method for conducting similar QC reviews at scale. We obtained digitized chest radiographs collected as part of the overseas immigration medical exam. Using radiographs from applicants 15 years old and older, we trained deep learning models to perform three tasks: identifying abnormal radiographs; identifying abnormal radiographs suggestive of tuberculosis; and identifying the specific findings (e.g., cavities or infiltrates) in abnormal radiographs. We then evaluated the models on both internal and external testing datasets, focusing on two classes of performance metrics: individual-level metrics, like sensitivity and specificity, and sample-level metrics, like accuracy in predicting the prevalence of abnormal radiographs. A total of 152,012 images (one image per applicant; mean applicant age 39 years) were used for model training. On our internal test dataset, our models performed well both in identifying abnormalities suggestive of TB (area under the curve [AUC] of 0.97; 95% confidence interval [CI]: 0.95, 0.98) and in estimating sample-level counts of the same (-2% absolute percentage error; 95% CIC: -8%, 6%). On the external test datasets, our models performed similarly well in identifying both generic abnormalities (AUCs ranging from 0.89 to 0.92) and those suggestive of TB (AUCs from 0.94 to 0.99). This performance was consistent across metrics, including those based on thresholded class predictions, like sensitivity, specificity, and F1 score. Strong performance relative to high-quality radiological reference standards across a variety of datasets suggests our models may make reliable tools for supporting chest radiography QC activities at CDC.
寻求进入美国的移民和难民必须首先接受由美国疾病控制和预防中心(CDC)监督的海外体检,在体检过程中,所有年龄≥15 岁的人都要接受胸部 X 光检查,以寻找结核病的迹象。虽然个别筛查点通常会实施质量控制(QC)计划,以确保正确解读X光片,但美国疾病预防控制中心目前还没有类似的大规模质量控制审查方法。我们获得了作为海外移民体检一部分的数字化胸部 X 光片。使用 15 岁及以上申请人的 X 光片,我们训练了深度学习模型来完成三项任务:识别异常 X 光片;识别提示肺结核的异常 X 光片;识别异常 X 光片中的特定发现(如龋齿或浸润)。然后,我们在内部和外部测试数据集上对模型进行了评估,重点关注两类性能指标:个体级指标(如灵敏度和特异性)和样本级指标(如预测异常射线照片患病率的准确性)。模型训练共使用了 152 012 张图像(每位申请人一张图像;申请人平均年龄 39 岁)。在内部测试数据集上,我们的模型在识别提示肺结核的异常方面表现良好(曲线下面积 [AUC] 为 0.97;95% 置信区间 [CI]:0.95, 0.98):0.95,0.98)和估计样本水平的相同计数(绝对百分比误差-2%;95% 置信区间 [CIC]:-8%,6%)。在外部测试数据集上,我们的模型在识别一般异常(AUC 在 0.89 到 0.92 之间)和提示肺结核的异常(AUC 在 0.94 到 0.99 之间)方面表现相似。这种性能在各种指标上都是一致的,包括那些基于阈值分类预测的指标,如灵敏度、特异性和 F1 分数。与各种数据集的高质量放射参考标准相比,我们的模型具有很强的性能,这表明我们的模型是支持疾病预防控制中心胸部放射质量控制活动的可靠工具。
{"title":"Development and validation of a deep learning model for detecting signs of tuberculosis on chest radiographs among US-bound immigrants and refugees.","authors":"Scott H Lee, Shannon Fox, Raheem Smith, Kimberly A Skrobarcek, Harold Keyserling, Christina R Phares, Deborah Lee, Drew L Posey","doi":"10.1371/journal.pdig.0000612","DOIUrl":"10.1371/journal.pdig.0000612","url":null,"abstract":"<p><p>Immigrants and refugees seeking admission to the United States must first undergo an overseas medical exam, overseen by the US Centers for Disease Control and Prevention (CDC), during which all persons ≥15 years old receive a chest x-ray to look for signs of tuberculosis. Although individual screening sites often implement quality control (QC) programs to ensure radiographs are interpreted correctly, the CDC does not currently have a method for conducting similar QC reviews at scale. We obtained digitized chest radiographs collected as part of the overseas immigration medical exam. Using radiographs from applicants 15 years old and older, we trained deep learning models to perform three tasks: identifying abnormal radiographs; identifying abnormal radiographs suggestive of tuberculosis; and identifying the specific findings (e.g., cavities or infiltrates) in abnormal radiographs. We then evaluated the models on both internal and external testing datasets, focusing on two classes of performance metrics: individual-level metrics, like sensitivity and specificity, and sample-level metrics, like accuracy in predicting the prevalence of abnormal radiographs. A total of 152,012 images (one image per applicant; mean applicant age 39 years) were used for model training. On our internal test dataset, our models performed well both in identifying abnormalities suggestive of TB (area under the curve [AUC] of 0.97; 95% confidence interval [CI]: 0.95, 0.98) and in estimating sample-level counts of the same (-2% absolute percentage error; 95% CIC: -8%, 6%). On the external test datasets, our models performed similarly well in identifying both generic abnormalities (AUCs ranging from 0.89 to 0.92) and those suggestive of TB (AUCs from 0.94 to 0.99). This performance was consistent across metrics, including those based on thresholded class predictions, like sensitivity, specificity, and F1 score. Strong performance relative to high-quality radiological reference standards across a variety of datasets suggests our models may make reliable tools for supporting chest radiography QC activities at CDC.</p>","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000612"},"PeriodicalIF":0.0,"publicationDate":"2024-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11441656/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142333978","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-30eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000372
Katelyn Dempsey, Joao Matos, Timothy McMahon, Mary Lindsay, James E Tcheng, An-Kwok Ian Wong
Disparities in pulse oximetry accuracy, disproportionately affecting patients of color, have been associated with serious clinical outcomes. Although many have called for pulse oximetry hardware replacement, the cost associated with this replacement is not known. The objective of this study was to estimate the cost of replacing all current pulse oximetry hardware throughout a hospital system via a single-center survey in 2023 at an academic medical center (Duke University) with three hospitals. The main outcome was the cost of total hardware replacement as identified by current day prices for hardware. New and used prices for 3,542/4,136 (85.6%) across three hospitals for pulse oximetry devices were found. The average cost to replace current pulse oximetry hardware is $6,834.61 per bed. Replacement and integration costs are estimated at $14.2-17.4 million for the entire medical system. Extrapolating these costs to 5,564 hospitals in the United States results in an estimated cost of $8.72 billion. "Simply replacing" current pulse oximetry hardware to address disparities may not be simple, cheap, or timely. Solutions for addressing pulse oximetry accuracy disparities leveraging current technology may be necessary, and might also be better. Trial Registration: Pro00113724, exempt.
{"title":"The high price of equity in pulse oximetry: A cost evaluation and need for interim solutions.","authors":"Katelyn Dempsey, Joao Matos, Timothy McMahon, Mary Lindsay, James E Tcheng, An-Kwok Ian Wong","doi":"10.1371/journal.pdig.0000372","DOIUrl":"10.1371/journal.pdig.0000372","url":null,"abstract":"<p><p>Disparities in pulse oximetry accuracy, disproportionately affecting patients of color, have been associated with serious clinical outcomes. Although many have called for pulse oximetry hardware replacement, the cost associated with this replacement is not known. The objective of this study was to estimate the cost of replacing all current pulse oximetry hardware throughout a hospital system via a single-center survey in 2023 at an academic medical center (Duke University) with three hospitals. The main outcome was the cost of total hardware replacement as identified by current day prices for hardware. New and used prices for 3,542/4,136 (85.6%) across three hospitals for pulse oximetry devices were found. The average cost to replace current pulse oximetry hardware is $6,834.61 per bed. Replacement and integration costs are estimated at $14.2-17.4 million for the entire medical system. Extrapolating these costs to 5,564 hospitals in the United States results in an estimated cost of $8.72 billion. \"Simply replacing\" current pulse oximetry hardware to address disparities may not be simple, cheap, or timely. Solutions for addressing pulse oximetry accuracy disparities leveraging current technology may be necessary, and might also be better. Trial Registration: Pro00113724, exempt.</p>","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000372"},"PeriodicalIF":0.0,"publicationDate":"2024-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11441667/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142333980","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Groundbreaking data-sharing techniques and quick access to stored research data from the African continent are highly beneficial to create diverse unbiased datasets to inform digital health technologies and artificial intelligence in healthcare. Yet health researchers in sub-Saharan Africa (SSA) experience individual and collective challenges that render them cautious and even hesitant to share data despite acknowledging the public health benefits of sharing. This qualitative study reports on the perspectives of health researchers regarding strategies to mitigate these challenges. In-depth interviews were conducted via Microsoft Teams with 16 researchers from 16 different countries across SSA between July 2022 and April 2023. Purposive and snowball sampling techniques were used to invite participants via email. Recorded interviews were transcribed, cleaned, coded and managed through Atlas.ti.22. Thematic Analysis was used to analyse the data. Three recurrent themes and several subthemes emerged around strategies to improve governance of data sharing. The main themes identified were (1) Strategies for change at a policy level: guideline development, (2) Strengthening data governance to improve data quality and (3) Reciprocity: towards equitable data sharing. Building trust is central to the promotion of data sharing amongst researchers on the African continent and with global partners. This can be achieved by enhancing research integrity and strengthening micro and macro level governance. Substantial resources are required from funders and governments to enhance data governance practices, to improve data literacy and to enhance data quality. High quality data from Africa will afford diversity to global data sets, reducing bias in algorithms built for artificial intelligence technologies in healthcare. Engagement with multiple stakeholders including researchers and research communities is necessary to establish an equitable data sharing approach based on reciprocity and mutual benefit.
{"title":"Trust as moral currency: Perspectives of health researchers in sub-Saharan Africa on strategies to promote equitable data sharing.","authors":"Qunita Brown, Jyothi Chabilall, Nezerith Cengiz, Keymanthri Moodley","doi":"10.1371/journal.pdig.0000551","DOIUrl":"10.1371/journal.pdig.0000551","url":null,"abstract":"<p><p>Groundbreaking data-sharing techniques and quick access to stored research data from the African continent are highly beneficial to create diverse unbiased datasets to inform digital health technologies and artificial intelligence in healthcare. Yet health researchers in sub-Saharan Africa (SSA) experience individual and collective challenges that render them cautious and even hesitant to share data despite acknowledging the public health benefits of sharing. This qualitative study reports on the perspectives of health researchers regarding strategies to mitigate these challenges. In-depth interviews were conducted via Microsoft Teams with 16 researchers from 16 different countries across SSA between July 2022 and April 2023. Purposive and snowball sampling techniques were used to invite participants via email. Recorded interviews were transcribed, cleaned, coded and managed through Atlas.ti.22. Thematic Analysis was used to analyse the data. Three recurrent themes and several subthemes emerged around strategies to improve governance of data sharing. The main themes identified were (1) Strategies for change at a policy level: guideline development, (2) Strengthening data governance to improve data quality and (3) Reciprocity: towards equitable data sharing. Building trust is central to the promotion of data sharing amongst researchers on the African continent and with global partners. This can be achieved by enhancing research integrity and strengthening micro and macro level governance. Substantial resources are required from funders and governments to enhance data governance practices, to improve data literacy and to enhance data quality. High quality data from Africa will afford diversity to global data sets, reducing bias in algorithms built for artificial intelligence technologies in healthcare. Engagement with multiple stakeholders including researchers and research communities is necessary to establish an equitable data sharing approach based on reciprocity and mutual benefit.</p>","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000551"},"PeriodicalIF":0.0,"publicationDate":"2024-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11432837/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142333981","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-27eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000606
Yiye Zhang, Yufang Huang, Anthony Rosen, Lynn G Jiang, Matthew McCarty, Arindam RoyChoudhury, Jin Ho Han, Adam Wright, Jessica S Ancker, Peter Ad Steel
Return visit admissions (RVA), which are instances where patients discharged from the emergency department (ED) rapidly return and require hospital admission, have been associated with quality issues and adverse outcomes. We developed and validated a machine learning model to predict 72-hour RVA using electronic health records (EHR) data. Study data were extracted from EHR data in 2019 from three urban EDs. The development and independent validation datasets included 62,154 patients from two EDs and 73,453 patients from one ED, respectively. Multiple machine learning algorithms were evaluated, including deep significance clustering (DICE), regularized logistic regression (LR), Gradient Boosting Decision Tree, and XGBoost. These machine learning models were also compared against an existing clinical risk score. To support clinical actionability, clinician investigators conducted manual chart reviews of the cases identified by the model. Chart reviews categorized predicted cases across index ED discharge diagnosis and RVA root cause classifications. The best-performing model achieved an AUC of 0.87 in the development site (test set) and 0.75 in the independent validation set. The model, which combined DICE and LR, boosted predictive performance while providing well-defined features. The model was relatively robust to sensitivity analyses regarding performance across age, race, and by varying predictor availability but less robust across diagnostic groups. Clinician examination demonstrated discrete model performance characteristics within clinical subtypes of RVA. This machine learning model demonstrated a strong predictive performance for 72- RVA. Despite the limited clinical actionability potentially due to model complexity, the rarity of the outcome, and variable relevance, the clinical examination offered guidance on further variable inclusion for enhanced predictive accuracy and actionability.
{"title":"Aspiring to clinical significance: Insights from developing and evaluating a machine learning model to predict emergency department return visit admissions.","authors":"Yiye Zhang, Yufang Huang, Anthony Rosen, Lynn G Jiang, Matthew McCarty, Arindam RoyChoudhury, Jin Ho Han, Adam Wright, Jessica S Ancker, Peter Ad Steel","doi":"10.1371/journal.pdig.0000606","DOIUrl":"https://doi.org/10.1371/journal.pdig.0000606","url":null,"abstract":"<p><p>Return visit admissions (RVA), which are instances where patients discharged from the emergency department (ED) rapidly return and require hospital admission, have been associated with quality issues and adverse outcomes. We developed and validated a machine learning model to predict 72-hour RVA using electronic health records (EHR) data. Study data were extracted from EHR data in 2019 from three urban EDs. The development and independent validation datasets included 62,154 patients from two EDs and 73,453 patients from one ED, respectively. Multiple machine learning algorithms were evaluated, including deep significance clustering (DICE), regularized logistic regression (LR), Gradient Boosting Decision Tree, and XGBoost. These machine learning models were also compared against an existing clinical risk score. To support clinical actionability, clinician investigators conducted manual chart reviews of the cases identified by the model. Chart reviews categorized predicted cases across index ED discharge diagnosis and RVA root cause classifications. The best-performing model achieved an AUC of 0.87 in the development site (test set) and 0.75 in the independent validation set. The model, which combined DICE and LR, boosted predictive performance while providing well-defined features. The model was relatively robust to sensitivity analyses regarding performance across age, race, and by varying predictor availability but less robust across diagnostic groups. Clinician examination demonstrated discrete model performance characteristics within clinical subtypes of RVA. This machine learning model demonstrated a strong predictive performance for 72- RVA. Despite the limited clinical actionability potentially due to model complexity, the rarity of the outcome, and variable relevance, the clinical examination offered guidance on further variable inclusion for enhanced predictive accuracy and actionability.</p>","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000606"},"PeriodicalIF":0.0,"publicationDate":"2024-09-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11432862/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142333976","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-25eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000573
James Shaw, Ibukun-Oluwa Omolade Abejirinde, Payal Agarwal, Simone Shahid, Danielle Martin
Research on digital health equity has developed in important ways especially since the onset of the COVID-19 pandemic, with a series of clear recommendations now established for policy and practice. However, research and policy addressing the health system dimensions of digital health equity is needed to examine the appropriate roles of digital technologies in enabling access to care. We use a highly cited framework by Levesque et al on patient-centered access to care and the World Health Organization's framework on digitally enabled health systems to generate insights into the ways that digital solutions can support access to needed health care for structurally marginalized communities. Specifically, we mapped the frameworks to identify where applications of digital health do and do not support access to care, documenting which dimensions of access are under-addressed by digital health. Our analysis suggests that digital health has disproportionately focused on downstream enablers of access to care, which are low-yield when equity is the goal. We identify important opportunities for policy makers, funders and other stakeholders to attend more to digital solutions that support upstream enablement of peoples' abilities to understand, perceive, and seek out care. These areas are an important focal point for digital interventions and have the potential to be more equity-enhancing than downstream interventions at the time that care is accessed. Overall, we highlight the importance of taking a health system perspective when considering the roles of digital technologies in enhancing or inhibiting equitable access to needed health care.
{"title":"Digital health and equitable access to care.","authors":"James Shaw, Ibukun-Oluwa Omolade Abejirinde, Payal Agarwal, Simone Shahid, Danielle Martin","doi":"10.1371/journal.pdig.0000573","DOIUrl":"https://doi.org/10.1371/journal.pdig.0000573","url":null,"abstract":"<p><p>Research on digital health equity has developed in important ways especially since the onset of the COVID-19 pandemic, with a series of clear recommendations now established for policy and practice. However, research and policy addressing the health system dimensions of digital health equity is needed to examine the appropriate roles of digital technologies in enabling access to care. We use a highly cited framework by Levesque et al on patient-centered access to care and the World Health Organization's framework on digitally enabled health systems to generate insights into the ways that digital solutions can support access to needed health care for structurally marginalized communities. Specifically, we mapped the frameworks to identify where applications of digital health do and do not support access to care, documenting which dimensions of access are under-addressed by digital health. Our analysis suggests that digital health has disproportionately focused on downstream enablers of access to care, which are low-yield when equity is the goal. We identify important opportunities for policy makers, funders and other stakeholders to attend more to digital solutions that support upstream enablement of peoples' abilities to understand, perceive, and seek out care. These areas are an important focal point for digital interventions and have the potential to be more equity-enhancing than downstream interventions at the time that care is accessed. Overall, we highlight the importance of taking a health system perspective when considering the roles of digital technologies in enhancing or inhibiting equitable access to needed health care.</p>","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000573"},"PeriodicalIF":0.0,"publicationDate":"2024-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11423990/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142333979","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-25eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000605
Elke Smith, Jan Peters, Nils Reiter
Problem gambling is a major public health concern and is associated with profound psychological distress and economic problems. There are numerous gambling communities on the internet where users exchange information about games, gambling tactics, as well as gambling-related problems. Individuals exhibiting higher levels of problem gambling engage more in such communities. Online gambling communities may provide insights into problem-gambling behaviour. Using data scraped from a major German gambling discussion board, we fine-tuned a large language model, specifically a Bidirectional Encoder Representations from Transformers (BERT) model, to predict signs of problem-gambling from forum posts. Training data were generated by manual annotation and by taking into account diagnostic criteria and gambling-related cognitive distortions. Using cross-validation, our models achieved a precision of 0.95 and F1 score of 0.71, demonstrating that satisfactory classification performance can be achieved by generating high-quality training material through manual annotation based on diagnostic criteria. The current study confirms that a BERT-based model can be reliably used on small data sets and to detect signatures of problem gambling in online communication data. Such computational approaches may have potential for the detection of changes in problem-gambling prevalence among online users.
问题赌博是一个重大的公共健康问题,与深重的心理压力和经济问题有关。互联网上有许多赌博社区,用户在那里交流有关游戏、赌博策略以及赌博相关问题的信息。问题赌博程度较高的人参与此类社区的程度较高。网络赌博社区可以帮助人们了解问题赌博行为。我们利用从德国一个主要赌博讨论区收集的数据,微调了一个大型语言模型,特别是一个来自变换器的双向编码器表征(BERT)模型,以预测论坛帖子中的问题赌博迹象。训练数据由人工注释生成,并考虑了诊断标准和与赌博相关的认知扭曲。通过交叉验证,我们的模型达到了 0.95 的精确度和 0.71 的 F1 分数,证明了通过基于诊断标准的人工标注生成高质量的训练材料可以获得令人满意的分类性能。目前的研究证实,基于 BERT 的模型可以可靠地用于小型数据集,并检测在线交流数据中的问题赌博特征。这种计算方法可能具有检测在线用户中问题赌博流行率变化的潜力。
{"title":"Automatic detection of problem-gambling signs from online texts using large language models.","authors":"Elke Smith, Jan Peters, Nils Reiter","doi":"10.1371/journal.pdig.0000605","DOIUrl":"https://doi.org/10.1371/journal.pdig.0000605","url":null,"abstract":"<p><p>Problem gambling is a major public health concern and is associated with profound psychological distress and economic problems. There are numerous gambling communities on the internet where users exchange information about games, gambling tactics, as well as gambling-related problems. Individuals exhibiting higher levels of problem gambling engage more in such communities. Online gambling communities may provide insights into problem-gambling behaviour. Using data scraped from a major German gambling discussion board, we fine-tuned a large language model, specifically a Bidirectional Encoder Representations from Transformers (BERT) model, to predict signs of problem-gambling from forum posts. Training data were generated by manual annotation and by taking into account diagnostic criteria and gambling-related cognitive distortions. Using cross-validation, our models achieved a precision of 0.95 and F1 score of 0.71, demonstrating that satisfactory classification performance can be achieved by generating high-quality training material through manual annotation based on diagnostic criteria. The current study confirms that a BERT-based model can be reliably used on small data sets and to detect signatures of problem gambling in online communication data. Such computational approaches may have potential for the detection of changes in problem-gambling prevalence among online users.</p>","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000605"},"PeriodicalIF":0.0,"publicationDate":"2024-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11423982/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142333977","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-23eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000299
Cynthia Lokker, Wael Abdelkader, Elham Bagheri, Rick Parrish, Chris Cotoi, Tamara Navarro, Federico Germini, Lori-Ann Linkins, R Brian Haynes, Lingyang Chu, Muhammad Afzal, Alfonso Iorio
Given the suboptimal performance of Boolean searching to identify methodologically sound and clinically relevant studies in large bibliographic databases, exploring machine learning (ML) to efficiently classify studies is warranted. To boost the efficiency of a literature surveillance program, we used a large internationally recognized dataset of articles tagged for methodological rigor and applied an automated ML approach to train and test binary classification models to predict the probability of clinical research articles being of high methodologic quality. We trained over 12,000 models on a dataset of titles and abstracts of 97,805 articles indexed in PubMed from 2012-2018 which were manually appraised for rigor by highly trained research associates and rated for clinical relevancy by practicing clinicians. As the dataset is unbalanced, with more articles that do not meet the criteria for rigor, we used the unbalanced dataset and over- and under-sampled datasets. Models that maintained sensitivity for high rigor at 99% and maximized specificity were selected and tested in a retrospective set of 30,424 articles from 2020 and validated prospectively in a blinded study of 5253 articles. The final selected algorithm, combining a LightGBM (gradient boosting machine) model trained in each dataset, maintained high sensitivity and achieved 57% specificity in the retrospective validation test and 53% in the prospective study. The number of articles needed to read to find one that met appraisal criteria was 3.68 (95% CI 3.52 to 3.85) in the prospective study, compared with 4.63 (95% CI 4.50 to 4.77) when relying only on Boolean searching. Gradient-boosting ML models reduced the work required to classify high quality clinical research studies by 45%, improving the efficiency of literature surveillance and subsequent dissemination to clinicians and other evidence users.
鉴于布尔搜索在大型文献数据库中识别方法可靠且与临床相关的研究方面表现不佳,因此有必要探索机器学习(ML)来对研究进行有效分类。为了提高文献监测计划的效率,我们使用了一个国际公认的大型数据集,其中包含了方法学严谨性标记的文章,并应用自动化的 ML 方法来训练和测试二元分类模型,以预测临床研究文章具有高方法学质量的概率。我们在 2012-2018 年期间被 PubM 索引的 97,805 篇文章的标题和摘要数据集上训练了 12,000 多个模型,这些数据集由训练有素的研究人员对其严谨性进行人工评估,并由执业临床医生对其临床相关性进行评级。由于数据集不平衡,不符合严谨性标准的文章较多,因此我们使用了不平衡的数据集以及过度采样和采样不足的数据集。我们从 2020 年的 30424 篇文章中选择并测试了对高严谨性的灵敏度保持在 99%、特异性最大化的模型,并在对 5253 篇文章的盲法研究中进行了前瞻性验证。最终选定的算法结合了在每个数据集中训练的LightGBM(梯度提升机)模型,在回顾性验证测试中保持了较高的灵敏度,特异性达到57%,在前瞻性研究中达到53%。在前瞻性研究中,找到一篇符合鉴定标准的文章所需的阅读篇数为 3.68(95% CI 3.52 至 3.85)篇,而仅依靠布尔搜索时为 4.63(95% CI 4.50 至 4.77)篇。梯度提升 ML 模型将高质量临床研究分类所需的工作量减少了 45%,提高了文献监测以及随后向临床医生和其他证据使用者传播的效率。
{"title":"Boosting efficiency in a clinical literature surveillance system with LightGBM.","authors":"Cynthia Lokker, Wael Abdelkader, Elham Bagheri, Rick Parrish, Chris Cotoi, Tamara Navarro, Federico Germini, Lori-Ann Linkins, R Brian Haynes, Lingyang Chu, Muhammad Afzal, Alfonso Iorio","doi":"10.1371/journal.pdig.0000299","DOIUrl":"10.1371/journal.pdig.0000299","url":null,"abstract":"<p><p>Given the suboptimal performance of Boolean searching to identify methodologically sound and clinically relevant studies in large bibliographic databases, exploring machine learning (ML) to efficiently classify studies is warranted. To boost the efficiency of a literature surveillance program, we used a large internationally recognized dataset of articles tagged for methodological rigor and applied an automated ML approach to train and test binary classification models to predict the probability of clinical research articles being of high methodologic quality. We trained over 12,000 models on a dataset of titles and abstracts of 97,805 articles indexed in PubMed from 2012-2018 which were manually appraised for rigor by highly trained research associates and rated for clinical relevancy by practicing clinicians. As the dataset is unbalanced, with more articles that do not meet the criteria for rigor, we used the unbalanced dataset and over- and under-sampled datasets. Models that maintained sensitivity for high rigor at 99% and maximized specificity were selected and tested in a retrospective set of 30,424 articles from 2020 and validated prospectively in a blinded study of 5253 articles. The final selected algorithm, combining a LightGBM (gradient boosting machine) model trained in each dataset, maintained high sensitivity and achieved 57% specificity in the retrospective validation test and 53% in the prospective study. The number of articles needed to read to find one that met appraisal criteria was 3.68 (95% CI 3.52 to 3.85) in the prospective study, compared with 4.63 (95% CI 4.50 to 4.77) when relying only on Boolean searching. Gradient-boosting ML models reduced the work required to classify high quality clinical research studies by 45%, improving the efficiency of literature surveillance and subsequent dissemination to clinicians and other evidence users.</p>","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000299"},"PeriodicalIF":0.0,"publicationDate":"2024-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11419392/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142309311","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-23eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000619
Gary M Franklin
{"title":"Google's new AI Chatbot produces fake health-related evidence-then self-corrects.","authors":"Gary M Franklin","doi":"10.1371/journal.pdig.0000619","DOIUrl":"10.1371/journal.pdig.0000619","url":null,"abstract":"","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000619"},"PeriodicalIF":0.0,"publicationDate":"2024-09-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11419366/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142309312","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-19eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000574
Jay Chandra, Raymond Lin, Devin Kancherla, Sophia Scott, Daniel Sul, Daniela Andrade, Sammer Marzouk, Jay M Iyer, William Wasswa, Cleva Villanueva, Leo Anthony Celi
In recent years, there has been substantial work in low-cost medical diagnostics based on the physical manifestations of disease. This is due to advancements in data analysis techniques and classification algorithms and the increased availability of computing power through smart devices. Smartphones and their ability to interface with simple sensors such as inertial measurement units (IMUs), microphones, piezoelectric sensors, etc., or with convenient attachments such as lenses have revolutionized the ability collect medically relevant data easily. Even if the data has relatively low resolution or signal to noise ratio, newer algorithms have made it possible to identify disease with this data. Many low-cost diagnostic tools have been created in medical fields spanning from neurology to dermatology to obstetrics. These tools are particularly useful in low-resource areas where access to expensive diagnostic equipment may not be possible. The ultimate goal would be the creation of a "diagnostic toolkit" consisting of a smartphone and a set of sensors and attachments that can be used to screen for a wide set of diseases in a community healthcare setting. However, there are a few concerns that still need to be overcome in low-cost diagnostics: lack of incentives to bring these devices to market, algorithmic bias, "black box" nature of the algorithms, and data storage/transfer concerns.
{"title":"Low-cost and convenient screening of disease using analysis of physical measurements and recordings.","authors":"Jay Chandra, Raymond Lin, Devin Kancherla, Sophia Scott, Daniel Sul, Daniela Andrade, Sammer Marzouk, Jay M Iyer, William Wasswa, Cleva Villanueva, Leo Anthony Celi","doi":"10.1371/journal.pdig.0000574","DOIUrl":"https://doi.org/10.1371/journal.pdig.0000574","url":null,"abstract":"<p><p>In recent years, there has been substantial work in low-cost medical diagnostics based on the physical manifestations of disease. This is due to advancements in data analysis techniques and classification algorithms and the increased availability of computing power through smart devices. Smartphones and their ability to interface with simple sensors such as inertial measurement units (IMUs), microphones, piezoelectric sensors, etc., or with convenient attachments such as lenses have revolutionized the ability collect medically relevant data easily. Even if the data has relatively low resolution or signal to noise ratio, newer algorithms have made it possible to identify disease with this data. Many low-cost diagnostic tools have been created in medical fields spanning from neurology to dermatology to obstetrics. These tools are particularly useful in low-resource areas where access to expensive diagnostic equipment may not be possible. The ultimate goal would be the creation of a \"diagnostic toolkit\" consisting of a smartphone and a set of sensors and attachments that can be used to screen for a wide set of diseases in a community healthcare setting. However, there are a few concerns that still need to be overcome in low-cost diagnostics: lack of incentives to bring these devices to market, algorithmic bias, \"black box\" nature of the algorithms, and data storage/transfer concerns.</p>","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000574"},"PeriodicalIF":0.0,"publicationDate":"2024-09-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11412657/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142302859","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Pub Date : 2024-09-19eCollection Date: 2024-09-01DOI: 10.1371/journal.pdig.0000607
Junko Kameyama, Satoshi Kodera, Yusuke Inoue
{"title":"Ethical, legal, and social issues (ELSI) and reporting guidelines of AI research in healthcare.","authors":"Junko Kameyama, Satoshi Kodera, Yusuke Inoue","doi":"10.1371/journal.pdig.0000607","DOIUrl":"https://doi.org/10.1371/journal.pdig.0000607","url":null,"abstract":"","PeriodicalId":74465,"journal":{"name":"PLOS digital health","volume":"3 9","pages":"e0000607"},"PeriodicalIF":0.0,"publicationDate":"2024-09-19","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11412540/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142302856","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}