JMIR Formative Research最新文献

Background: Large language models (LLMs) have rapidly advanced across numerous fields, including mental health care. A shortage of trained therapists and mental health care providers has driven informal use of LLMs for therapeutic support. However, their clinical utility remains poorly defined.

Objective: This study aimed to systematically evaluate and compare the therapeutic knowledge and single-turn response capabilities of LLMs versus psychotherapists in training in the context of behavioral activation (BA) therapy for depression, and to assess how both groups' performance changed when provided with structured therapeutic training materials.

Methods: Six LLMs and 8 human participants completed a questionnaire on depression and BA with 20 multiple-choice items and 10 therapy scenarios, each with 3 open-ended items, that postulated empathic response, use of validation strategies, and theory of mind capabilities. Human participants completed the questionnaire before and after a 5-hour workshop and 5-week period with learning materials. The LLMs received identical training content as context during the second test. All open-ended questions were rated on 5-point scales by 2 experts.

Results: At baseline, the LLMs demonstrated higher knowledge scores than human participants (61.0 vs 52.0 out of 100 points) and were rated higher in empathy (U=2.0; P=.005; r=0.917), validation quality (U=2.5; P=.006; r=0.896), anticipation of cognition (U=0.0; P=.002; r=1.000), and anticipation of emotion (U=0.0; P=.002; r=1.000). Following BA training, the LLMs maintained their performance advantage across multiple-choice and open-ended items.

Conclusions: The results suggest that LLMs may generate high-quality therapeutic single-turn responses that integrate clinical knowledge with empathetic communication. The findings hint at LLMs' potential as valuable tools in mental health care, although further clinical trials are needed to evaluate their performance in ongoing therapeutic relationships and clinical outcomes.

Background: Video games are becoming increasingly accessible and occupy a prominent position among students' leisure activities. Recent studies have demonstrated that engagement with video games can facilitate the development of specific abilities, such as visuospatial skills and hand-eye coordination. Thus, it seemed relevant to investigate whether the practice of playing video games could enhance the technical capabilities of novice dental students.

Objective: The objective of this study was to ascertain whether dental students who identified themselves as video game players exhibited differences in fundamental technical skills in comparison to students who identified as nonplayers. This preliminary study aimed to validate the methodology and measurement tools for a subsequent prospective study.

Methods: Second-year dental students who were novices in the field of preclinical dental practice were divided into 2 groups: one designated nonplayers and the other players. The visual, motor, and cognitive coordination of the students was assessed using 3 different tests. The initial assessment focused on evaluating spatial ability, while the subsequent assessments addressed arm-hand coordination and the velocity of execution. The study data were collected in September 2021.

Results: This study included 92 second-year dental students (mean age 19.49, SD 0.8 years). Thirty-seven students were placed in the player group (40%), while 55 were placed in the nonplayer group (60%). The findings showed no statistically significant differences between the two groups when the 3 distinct tests were administered. The nonplayer group made fewer mistakes on the test evaluating spatial ability (P=.051) and achieved a higher score on the modified Precision Manual Dexterity Test, which evaluates arm-hand coordination, compared to the player group (P=.17), but without statistical significance. The nonplayer group took more time to perform the pulpotomy during the third test, which evaluated execution speed, compared to the player group, again without significance (P=.87).

Conclusions: This study did not demonstrate significant differences between player and nonplayer dental students who participated in the study with regard to their fundamental technical abilities in a preclinical training environment. Nevertheless, it validated the feasibility of a methodology for a future longitudinal study to concentrate on the evolution of acquiring technical skills during preclinical training in these two populations. Consequently, further investigation is warranted to determine the potential impact of video games on the acquisition of surgical skills, including in dentistry.

Background: Large language models (LLMs) are increasingly used in medical education for feedback and grading; yet their role in postgraduate examination preparation remains uncertain due to inconsistent grading, hallucinations, and user acceptance.

Objective: This study evaluates the Personalized Anesthesia Study Support (PASS), a specialized GPT-4 model developed to assist candidates preparing for Singapore's postgraduate specialist anesthesiology examination. We assessed user acceptance, grading interrater reliability, and hallucination detection rates to determine the feasibility of integrating specialized LLMs into high-stakes examination preparation.

Methods: PASS was built on OpenAI's GPT-4 and adapted with domain-specific prompts and references. Twenty-one senior anesthesiology residents completed a mock short answer question examination, which was independently graded by 3 human examiners and 3 PASS iterations. Participants reviewed feedback from both PASS and standard GPT-4 and completed a technology acceptance model (TAM) survey. Grading reliability was evaluated using Cohen and Fleiss κ. Hallucination rates were assessed by participants and examiners.

Results: Of the 21 participants, 17 (81%) completed the TAM survey, generating 136 responses. PASS scored significantly higher than standard GPT-4 in usefulness (mean 4.25, SD 0.50 vs mean 3.44, SD 0.82; P<.001), efficiency (mean 4.12, SD 0.61 vs mean 3.41, SD 0.74; P<.001), and likelihood of future use (mean 4.13, SD 0.75 vs mean 3.59, SD 0.90; P<.001), with no significant difference in ease of use (mean 4.56, SD 0.63 vs mean 4.50, SD 0.61; P=.35). Internal grading reliability was moderate for PASS (κ=0.522) and fair for human examiners (κ=0.275). Across 316 PASS-generated responses, 67 hallucinations and 189 deviations were labeled. Hallucination labeling rates were comparable between candidates (10/67, 15%) and examiners (57/249, 22.9%; P=.21), while examiners labeled significantly more deviations (168/249, 67.5% vs 21/67, 31%; P<.001).

Conclusions: PASS demonstrated strong user acceptance and grading reliability, suggesting feasibility in high-stakes examination preparation. Experienced learners could identify major hallucinations at comparable rates to examiners, suggesting potential in self-directed learning but with continued need for caution. Further research should refine grading accuracy and explore multicenter evaluation of specialized LLMs for postgraduate medical education.

[This corrects the article DOI: 10.2196/69943.].

Background: Internet health care plays a crucial role in addressing the challenge of distributing high-quality medical resources and promoting the optimal allocation of these resources and health equity in China. Online medical consultation (OMC) plays a more significant role than online health information seeking (OHIS). Currently, the proportion of Chinese patients using OMC is low. Therefore, it is essential to enhance patient engagement with OMC and fully leverage the role of internet health care in optimizing the allocation of medical resources.

Objective: This study aims to explore the correlation mechanisms of online medical community users' switching behaviors from OHIS to OMC.

Methods: This study is based on the knowledge-attitude-practice theory, which combines the social support theory and the health belief model to construct a research model of users' willingness to transition from OHIS to OMC. The study adopts a questionnaire survey and structural equation modeling method to conduct an empirical study.

Results: Gaining knowledge about information support has a significant positive impact on perceived susceptibility (β=.339, P<.001), perceived severity (β=.348, P<.001), and perceived benefits (β=.361, P<.001), while having a significant negative impact on perceived barriers (β=-.285, P<.001). Gaining knowledge about emotional support positively affects perceived susceptibility (β=.220, P<.001) and perceived benefits (β=.149, P<.01) but does not significantly influence perceived severity (β=-.006, P>.05) or perceived barriers (β=.099, P>.05). Perceived susceptibility (β=.123, P<.05), perceived severity (β=.174, P<.001), and perceived benefits (β=.273, P<.001) positively influence patients' transition to online consultation behavior, whereas perceived barriers (β=-.112, P<.05) negatively impact this switch. In addition, we found that gaining knowledge about information support not only directly affects patients' behavior in switching to online consultations but also impacts patients' OMCs through perceived susceptibility (14.23%), perceived severity (13.17%), and perceived benefits (25.28%). In contrast, gaining knowledge about emotional support does not directly influence patient behavior transfer; it operates only through perceived susceptibility (46.95%) and perceived benefit (52.90%).

Conclusions: This study integrated the knowledge-attitude-practice framework, social support theory, and health belief model to uncover the internal logic of patients' behavioral transfers within online health communities. It confirmed the mediating role of the cognitive-emotional dual-drive pathway and health beliefs. The findings provide a scientific basis for the functional design of online health care platforms and for precise health knowledge dissemination strategies.

Background: Cisgender women account for 23% of new HIV diagnoses in the United States, but there are significant socio-structural barriers to engagement and retention in the preexposure prophylaxis (PrEP) cascade, particularly for women of color.

Objective: In response to the lack of evidence-based interventions to improve PrEP initiation, adherence, and persistence among women in the United States, we developed and piloted a clinic-based, socio-structural intervention to measure (1) the feasibility of delivering the adapted intervention and (2) clinic team and patient perspectives on the intervention, in preparation for a future trial on engagement and retention in the PrEP cascade among women.

Methods: We previously applied the ADAPT-ITT (Assessment, Decision, Adaptation, Production, Topical experts, Integration, Training, Testing) model to develop a culturally appropriate, evidence-based intervention, responsive to Black women's HIV prevention needs. In the present study, we set out to complete the Training and Testing phases: namely, to hire and train a PrEP navigator and to train clinic staff to deliver the adapted intervention. We completed a 4-month pilot to assess the feasibility of delivering the intervention and collecting outcomes of interest and initial outcome trends (compared to baseline). We further assessed the clinic team and patient perspectives on the intervention to understand the potential for future scale and delivery.

Results: The clinic team participants found the adapted Women's PrEP project (W-PrEP) intervention both highly feasible and relevant to patients, with minimal impact on clinic flow. Patient participants reported that the W-PrEP intervention was highly relevant and appreciated the education and counseling from the PrEP navigator-many learning about PrEP for the first time from the navigator and health care provider. The outcome measures were both feasible to collect and appropriate to capture the primary outcomes of interest. Finally, the W-PrEP intervention increased the proportion of patients counseled about PrEP from 65% to 76% of patients seen (P<.001).

Conclusions: The intervention and outcome data collection was feasible, and open-ended clinic team and patient perspectives showed positive feedback about the intervention and relevance. The associated increase in PrEP counseling is promising but necessitates further evaluation of the effects of the W-PrEP intervention on the PrEP cascade (eg, initiation, persistence, adherence) in a larger randomized trial.

Background: Cardiovascular diseases (CVDs) remain the leading global cause of mortality, yet traditional electrocardiogram (ECG) interpretation suffers from subjective variability and limited sensitivity to complex pathologies.

Objective: To address these challenges, we propose the Cardiovascular Multimodal Prediction Network (CaMPNet), a Transformer-based multimodal architecture that integrates raw 12-lead ECG waveforms, nine structured machine-measured the electrocardiogram (ECG) features, and demographic data (age and sex) through cross-attention fusion.

Methods: The model was trained on 384,877 records from the MIMIC-IV-ECG database and evaluated across 12 cardiovascular disease labels. To further assess temporal robustness, a temporal external validation was performed using the most recent 10% of the data, withheld chronologically from model development.

Results: On the internal test set, the model achieved a mean Area Under the Curve (AUC) of 0.845 and Area Under the Precision-Recall Curve (AUPRC) of 0.489, outperforming the ResNet-ECG baseline (AUC 0.848 but F1 0.152) and all single-modality variants. Subgroup analyses demonstrated consistent performance across demographics (male AUC 0.846 vs female 0.843; youngest quartile 0.884 vs oldest 0.811). CaMPNet retained moderate discriminative ability in temporal external validation with a mean AUC of 0.715 and AUPRC of 0.298, though performance declined due to temporal distribution shifts. Despite this, major disease categories such as atrial fibrillation, heart failure, and normal rhythm maintained high AUCs (> 0.84). Attention-based visualization revealed clinically interpretable patterns (e.g., ST-segment elevations in ST-Segment Elevation Myocardial Infarction), and ablation experiments verified the model's tolerance to missing structured inputs.

Conclusions: CaMPNet demonstrates robust and interpretable multimodal ECG-based diagnosis, offering a scalable framework for comorbidity screening and continual learning under real-world temporal dynamics.

Clinicaltrial:

Background: Commercial markerless motion capture (MMC) systems show promise for use in rehabilitation and have been validated for the assessment of various parameters. However, no prior studies have evaluated MMC systems to detect stance asymmetry.

Objective: The objective of this study was to assess the accuracy and variability of the Jintronix Weight Shift Tool MMC system to estimate the percentage of weight borne on each foot.

Methods: Twelve healthy younger adults, 12 healthy older adults, and 12 people living with stroke were recruited for this cross-sectional study. The percentage of weight borne on each foot was simultaneously recorded by the Weight Shift Tool and a validated pressure mat during 2 series (raising the arm to capture the recording and without arm raise) with left lean, right lean, and equal stance. Agreement between the Weight Shift Tool and the pressure mat was assessed using Bland-Altman analyses.

Results: Bias was greatest for older adults for all stances except for right lean without arm raise. Variability was greatest for people living with stroke for all stances except for left lean with arm raise. On average, the limits of agreement were narrower during equal stance. Although bias between the Weight Shift Tool and the pressure mat was small to moderate (0.0%-11.7%), the limits of agreement were wide (12.8%-33.6% above and below the bias).

Conclusions: The Weight Shift Tool is not clinically acceptable for the estimation of the percentage of weight on each foot due to high variability. Investigation of other MMC systems is required to confirm the validity of MMC for clinical assessment.