Mayo Clinic Proceedings. Digital health最新文献

Objective

To determine the appropriateness of ophthalmology recommendations from an online chat-based artificial intelligence model to ophthalmology questions.

Patients and Methods

Cross-sectional qualitative study from April 1, 2023, to April 30, 2023. A total of 192 questions were generated spanning all ophthalmic subspecialties. Each question was posed to a large language model (LLM) 3 times. The responses were graded by appropriate subspecialists as appropriate, inappropriate, or unreliable in 2 grading contexts. The first grading context was if the information was presented on a patient information site. The second was an LLM-generated draft response to patient queries sent by the electronic medical record (EMR). Appropriate was defined as accurate and specific enough to serve as a surrogate for physician-approved information. Main outcome measure was percentage of appropriate responses per subspecialty.

Results

For patient information site-related questions, the LLM provided an overall average of 79% appropriate responses. Variable rates of average appropriateness were observed across ophthalmic subspecialties for patient information site information ranging from 56% to 100%: cataract or refractive (92%), cornea (56%), glaucoma (72%), neuro-ophthalmology (67%), oculoplastic or orbital surgery (80%), ocular oncology (100%), pediatrics (89%), vitreoretinal diseases (86%), and uveitis (65%). For draft responses to patient questions via EMR, the LLM provided an overall average of 74% appropriate responses and varied by subspecialty: cataract or refractive (85%), cornea (54%), glaucoma (77%), neuro-ophthalmology (63%), oculoplastic or orbital surgery (62%), ocular oncology (90%), pediatrics (94%), vitreoretinal diseases (88%), and uveitis (55%). Stratifying grades across health information categories (disease and condition, risk and prevention, surgery-related, and treatment and management) showed notable but insignificant variations, with disease and condition often rated highest (72% and 69%) for appropriateness and surgery-related (55% and 51%) lowest, in both contexts.

Conclusion

This LLM reported mostly appropriate responses across multiple ophthalmology subspecialties in the context of both patient information sites and EMR-related responses to patient questions. Current LLM offerings require optimization and improvement before widespread clinical use.

Objective

To thoroughly analyze factors affecting the generalization ability of deep learning algorithms on brain tumor detection and segmentation models.

Patients and Methods

We searched PubMed, Embase, Web of Science, Cochrane Library, and IEEE from inception to July 25, 2023, and 19 studies with 12,000 patients were identified. The criteria required studies to use magnetic resonance imaging (MRI) for brain tumor detection and segmentation, offer clear performance metrics, and use external validation data sets. The study focused on outcomes such as sensitivity and Dice score. Study quality was assessed using QUADAS-2 and CLAIM tools. The meta-analysis evaluated varying algorithms and their performance across different validation data sets.

Results

MRI hardware as the manufacturer may contribute to data set diversity, impacting AI model generalizability. The study found that the best algorithms had a pooled lesion-wise Dice score of 84%, with pooled sensitivities of 87% (patient-wise) and 86% (lesion-wise). Post-2022 methodologies highlighted evolving artificial intelligence techniques. Performance differences were evident among tumor types, likely due to size disparities. 3D models outperformed their 2D and ensemble counterparts in detection. Although specific preprocessing techniques improved segmentation outcomes, some hindered detection.

Conclusion

The study underscores the potential of deep learning in improving brain tumor diagnostics and treatment planning. We also identify the need for further research, including developing a comprehensive diversity index, expanded meta-analyses, and using generative adversarial networks for data diversification, paving the way for AI-driven advancements in oncological patient care.

Trial Registration

PROPERO (CRD42023459108).

Objective

To address thyroid cancer overdiagnosis, we aim to develop a natural language processing (NLP) algorithm to determine the appropriateness of thyroid ultrasounds (TUS).

Patients and Methods

Between 2017 and 2021, we identified 18,000 TUS patients at Mayo Clinic and selected 628 for chart review to create a ground truth dataset based on consensus. We developed a rule-based NLP pipeline to identify TUS as appropriate TUS (aTUS) or inappropriate TUS (iTUS) using patients’ clinical notes and additional meta information. In addition, we designed an abbreviated NLP pipeline (aNLP) solely focusing on labels from TUS order requisitions to facilitate deployment at other health care systems. Our dataset was split into a training set of 468 (75%) and a test set of 160 (25%), using the former for rule development and the latter for performance evaluation.

Results

There were 449 (95.9%) patients identified as aTUS and 19 (4.06%) as iTUS in the training set; there are 155 (96.88%) patients identified as aTUS and 5 (3.12%) were iTUS in the test set. In the training set, the pipeline achieved a sensitivity of 0.99, specificity of 0.95, and positive predictive value of 1.0 for detecting aTUS. The testing cohort revealed a sensitivity of 0.96, specificity of 0.80, and positive predictive value of 0.99. Similar performance metrics were observed in the aNLP pipeline.

Conclusion

The NLP models can accurately identify the appropriateness of a thyroid ultrasound from clinical documentation and order requisition information, a critical initial step toward evaluating the drivers and outcomes of TUS use and subsequent thyroid cancer overdiagnosis.

We review the evolution of virtual care (VC) in Ontario. Pre–COVID-19, the primary focus was on patients in remote and underserved areas who went to host sites for care. Ontario’s vision pre-pandemic was for a gradual increase in VC by physicians registered with the Ontario Telemedicine Network (OTN), using OTN-approved video technologies; some accommodated patients and doctors wherever they were. Less than 1% of care was virtual pre-pandemic. We discuss how policies that altered access to in-person care (pandemic lockdowns and guidelines to seek and provide care virtually), compensation policy changes (allowing any Ontario physician to be compensated for VC), and policies allowing common technologies not previously allowed (including, importantly, the telephone), drove and enabled a rapid shift to >50% of care being virtual at the start of the pandemic, leveling off to ∼30% over time. We review policy changes in late 2022 and predict these will result in a drop in VC compared with the policies during the pandemic, particularly for walk-in clinic patients, in a province where 2.2-4.6 million people do not have a primary care doctor and presumably use walk-in clinics. This is because, going forward, physicians will be compensated less for telephone care than for in-person or video care for rostered patients, and because compensation will be less still for telephone or video care provided to walk-in patients. Through this case study we develop a visual model of how these key policy and technology factors influence the provision of VC.

Objective

To investigate whether an artificial intelligence (AI)–based model can predict tumor invasiveness in patients with multifocal lung adenocarcinoma (MFLA).

Patients and Methods

Patients with MFLA who underwent surgical resection were enrolled to a prospective registry trial (NCT01946100). Each identified nodule underwent retrospective computer-aided nodule assessment and risk yield (CANARY)–based AI to determine a quantitative degree of invasiveness. Data regarding age, sex, medical and surgical management, and survival were collected and analyzed. Pathologic review was performed by a pulmonary pathologist with comprehensive histologic subtyping.

Results

From January 1, 2013, through December 31, 2018, 68 patients with MFLA underwent at least 1 surgical resection. Five-year survival for the cohort was 91%, and 10-year survival was 73.6%. No significant differences in survival were observed when separated by sex, number, or size of the nodules. A 10-year survival trend was seen when comparing patients with unilateral (100% survival) vs bilateral disease (66%). Retrospective CANARY-based AI analysis demonstrated that the majority of the nodules present at the time of diagnosis (229/302; 75.8%) were classified good, with an average score of 0.19, suggesting indolent clinical behavior and noninvasive pathology. However, AI-CANARY scores of the surgically removed nodules were significantly higher compared with those of the nonresected nodules (P=.001).

Conclusion

The long-term survival for patients with N0, M0 MFLA who have undergone surgical resection may approach those of stage I non–small cell lung cancer. CANARY-based AI has the potential to stratify individual nodules to help guide surgical intervention versus observation of nodules.

Trial Registration

clinicaltrials.gov Identifier: NCT01946100