Journal of Clinical Epidemiology最新文献

Objectives: Non-randomized studies of interventions provide valuable evidence in health sciences but are prone to biases that affect validity. Multiple instruments have been developed for critical appraisal, although their measurement properties remain insufficiently established. These instruments typically aim to evaluate two key theoretical constructs: methodological quality and risk of bias, which reflect different but complementary dimensions of study rigor and internal validity. The COSMIN framework offers internationally recognized standards for evaluating instrument robustness, thus enhancing transparency and comparability in their selection. This systematic review specifically focused on identifying and critically evaluating studies that have empirically tested the measurement properties of instruments developed to assess methodological quality and/or risk of bias in non-randomized studies of interventions.

Study design and setting: This systematic review and meta-analysis was conducted in accordance with the COSMIN initiative which is structured in three parts: (1) a literature search reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses and the PRISMA search extension; (2) an assessment of methodological quality and measurement properties using the "COSMIN Risk of Bias" tool, together with an evaluation of the certainty of evidence following the "Grading of Recommendations Assessment, Development, and Evaluation" and, (3) a meta-analysis of measurement properties when sufficient quantitative data were available across validation studies for a given instrument.

Results: Eleven instruments for critical appraisal of non-randomized studies of interventions were identified. None were evaluated for all COSMIN measurement properties; instrument development, content validity, and reliability were most consistently reported. MINORS, MMERSQI, and ASSESS demonstrated the highest quality evidence for methodological quality, while ROBINS-I provided the strongest evidence for risk of bias assessment. For instruments with sufficient comparable data, such as ROBINS-I, a meta-analysis of inter-rater reliability coefficients was conducted, showing moderate agreement for selection and exposure domains, with substantial heterogeneity across studies.

Conclusion: MINORS emerged as the most robust instrument for critical appraisal of methodological quality, whereas ROBINS-I stood out for risk of bias. ASSESS and MMERSQI provided adequate evidence of content validity, further assessment of different psychometric properties is needed, highlighting that only a small subset of available tools for NRSI have undergone formal psychometric validation. Further research is needed to strengthen the measurement evidence base for these instruments.

Objectives: Core outcome sets (COS) are an agreed upon set of outcomes that should be evaluated in a clinical trial, allowing data from similar areas of healthcare to be compared and pooled in meta-analyses, informing whether treatments work and/or harm. Pediatric COS have been developed by identifying outcomes that are important to relevant stakeholders, and recently the importance of including children and young people (CYP) as stakeholders has been recognized. While research on the overall development of pediatric COS exists, no comprehensive review has examined methods implemented by COS developers to include CYP in the development process. This systematic review aims to establish current practice and identify issues in including CYP in developing a pediatric COS, particularly in the context of rare conditions.

Study design: We conducted a systematic review of studies in the Core Outcome Measures in Effectiveness Trials (COMET) Initiative database that developed pediatric core outcome sets (COS) for children and young people (CYP; 0-18 years). Data were extracted on methods used to include CYP, barriers to their inclusion and stakeholder participation. Authors were contacted when methodological details were unclear or insufficiently reported.

Results: 70 articles corresponding to 53 pediatric COS were analyzed; 18 (34%) included CYP, 16 (30%) were developed for a rare condition/condition with a rare form. CYP were included in advisory groups (6/18 COS), pilot surveys/questionnaires (3/18 COS), focus groups (2/18 COS), workshop (1/18 COS), qualitative interviews (8/18 COS), Delphi (and other) surveys (15/18 COS) and consensus meetings (5/18 COS). COS developers adapted methods for patient and public involvement and engagement work to improve participation in COS, to include CYP in generating outcomes and reach consensus on the most important outcomes to include in the COS. Barriers to including CYP were age and the condition/area of COS development.

Conclusion: Our findings show that with appropriate adaptations, CYP can be included in COS development, regardless of age, communication needs or whether conditions are rare or common. This review proposes adaptations to refine current pediatric COS methodology to increase CYP inclusion.

Objectives: With the exponential growth of biomedical literature, the challenge of conducting systematic reviews is becoming increasingly burdensome. We aimed to evaluate the performance of LLMs in the automation of some or all steps of systematic reviews and meta-analyses.

Study design and setting: In this systematic review, we searched PubMed, Embase, the Cochrane Library and preprint platforms up to 14/01/2025. We included any studies assessing the performance of LLMs (e.g., GPT, Claude, Mistral) in any step of the systematic review process. Pairs of reviewers independently extracted data and assessed risk of bias. We conducted analyses using median(IQR) for positive (PPA) and negative percent agreement (NPA), respectively analogous to sensitivity and specificity, between LLMs and human reviewers.

Results: From 3,889 unique references, we included 63 studies of which 52 reporting performance metrics for a total of 148 LLM performance assessments. Most assessments concerned GPT models (n=114, 77%). The most frequently evaluated tasks were Title and Abstract Screening (n=78, 53%), Data Extraction (n=23, 16%), and Full-Text screening (n=20, 14%). For Title and Abstract screening, overall median PPA was 0.92 (IQR 0.69-0.98) and median NPA was 0.89 (0.72-0.95). For full text screening, the overall median PPA was 0.93 (0.87-1.00) and median NPA was 0.92 (0.78-0.97). Late-generation LLMs released after GPT-4 seemed to provide higher performance than earlier models. For other tasks, authors reported overall good performances, but variability of performance metrics precluded complete quantitative synthesis. Global accuracy for data extraction tasks ranged from 0.36 to 1.00, with a median accuracy of 0.95 (IQR 0.91-0.97, n=11). For the 'Risk of Bias assessment' task, accuracy ranged from 0.44 to 0.90 (median = 0.62, IQR 0.53-0.76, n=6).

Conclusion: The performance of LLMs, particularly newer generations, shows promise in automating some repetitive steps of systematic reviews such as screening. However, their successful integration will require appropriate safeguards and careful implementation.

Background: Genetic mutations often result in antimicrobial resistance. Early identification of emerging resistance due to genetic mutations often relies on multiple co-existing pathways of indirect evidence. The GRADE approach, widely used to rate certainty in treatment comparisons, offers little guidance to address such situations. Motivated by the possibility of emerging resistance to malaria, we provide an example of how GRADE guidance might be adapted to address multiple pathways of indirect evidence, an approach, we term SPICE-GRADE (Simultaneous Processing of Indirect Sources of Causal Evidence using GRADE).

Purpose: We developed and here illustrate a structured approach for integrating direct and indirect causal evidence within GRADE, using antimalarial drug resistance mediated by Plasmodium falciparum Kelch13 mutations as a case study.

Methods: Confronted by the problem of applying GRADE to the question of emerging antimicrobial resistance to malaria, we simultaneous considered low certainty direct evidence and two pathways of indirect evidence addressing a possible causal relation between Kelch13 mutations and malaria recrudescence. Links between Kelch13 mutations and ring-stage survival and between Kelch13 mutations and delayed parasite clearance constitute the two indirect pathways. We addressed each link was independently, and for the two indirect links the extent of indirectness informed the final judgment.

Results: All three links relied on observational studies and therefore started as low certainty evidence. For both the direct link between Kelch13 mutations and recrudescence, and the indirect link between Kelch13 mutations and ring-stage survival, we rated the certainty up two levels for a very strong association, and then down two levels due to imprecision from small sample size in the direct link, and due to very serious indirectness in the indirect link. The third link, between Kelch13 mutations and delayed parasite clearance, was rated up one level for a strong association and down one for serious indirectness. Although each link remained at low certainty, the consistent direction of effect and coherence across all links strengthened the overall causal inference. Situating the entire body of evidence on a continuum allowed us to rate the overall certainty toward the higher end of low certainty, providing a more robust conclusion than relying on direct evidence alone.

Conclusion: In this case study, we illustrate SPICE-GRADE as a structured way of mapping and assessing multiple causal links within the GRADE framework. Establishing SPICE-GRADE as a robust methodology for GRADE assessment will require formal methodological development with application across multiple contexts.

This commentary provides contextual information to support interpretation of new guidance for the use of the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Evidence-to-Decision (EtD) framework for environmental and occupational health (EOH). Based on a systematic review and narrative synthesis of EOH decision frameworks, and input from subject matter experts in environmental health and GRADE, we developed and pilot-tested an EOH framework through a series of virtual workshops. The resulting guidance was approved by the GRADE Working Group in May 2023. The new EtD framework for EOH follows the same structure as existing EtD frameworks, including a scoping and contextualization process and twelve assessment criteria. Modifications to the framework to improve applicability for the EOH context include changes to terminology, tailoring of the framework detailed judgments (e.g., broadening the scope of the equity decision criterion to include non-health equity issues), and guidance for users (e.g., emphasis on the scoping phase of the guideline development process).

Randomized trials are used to evaluate healthcare interventions because they minimize confounding and selection bias through randomization. Harms reporting in trials remains suboptimal, despite established guidelines. To improve transparency and clinical relevance, trialists should share information about harms - whether assessed systematically or non-systematically. Non-systematically assessed adverse events warrant greater attention, as they are often underreported or inconsistently documented. Trialists should specify what was measured, when, and by whom. For each study arm, tables or data should be available that include all harms observed. For dichotomous outcomes, tables or data should include the number of people who experienced each harm in each group: the number of participants at risk of harms (i.e., randomized individuals); the number of deaths; participants with one or more adverse events; withdrawals (discontinuations) due to harms; and the total number of events, if appropriate. Thresholds should not be used to limit the sharing of information about harms. Zero events should be included for harms systematically assessed. For combined adverse events, such as the proportion of participants with one or more serious adverse events (SAEs), researchers should report or share data for all component events (e.g., deaths, major cardiovascular events, cancers, infections, psychiatric events). Better harms reporting could improve evidence synthesis, enhance interpretability, and support informed clinical decision-making as well as patient safety.

Background and objectives: Multimorbidity, defined as the presence of multiple long-term health conditions within an individual, remains a growing challenge in healthcare. Identifying frequently occurring multimorbidity clusters may help to develop targeted interventions and optimise care pathways. However, the validation of multimorbidity clusters derived from real-world data is complicated by the lack of a known "ground truth". We conducted a statistical simulation study that aimed to evaluate the performance of three common validation approaches (cluster separation, clustering stability, and strength of association with health outcomes) in assessing the quality of multimorbidity clusters, where performance was measured by agreement with known ground truth clusters.

Methods: Simulated datasets with predefined clusters were generated across 25 scenarios, varying parameters such as disease prevalence, sample size, and noise levels. Latent class analysis was applied to derive clusters from the simulated data, which were compared to the predefined clusters using the Adjusted Rand Index (ARI). The ARI served as our gold standard quality assessment of derived clusters.

Results: Cluster separation, measured by the Calinski-Harabasz Index, showed the strongest agreement with our gold standard in most scenarios (median correlation: 0.641, IQR: 0.505-0.728). Clustering stability, assessed using resampling, had mixed performance, with a median correlation of 0.421 (IQR: 0.127-0.526). The strength of association with health outcomes, assessed using Nagelkerke's R², consistently showed poor agreement (median correlation: -0.424, IQR: -0.543 to -0.173) with the ARI.

Conclusion: Cluster separation seems to be the most reliable approach to validate multimorbidity clusters. Clustering stability can sometimes be used for validation but has limitations. Assessing the strength of association of multimorbidity clusters with health outcomes, though valuable for understanding clinical relevance, appears to not validate cluster quality despite being commonly used in published literature.

Background: There is increased interest in adaptive platform trials (APTs), i.e., complex randomised clinical trials (RCTs) focusing on a population or a setting, conducted according to a core protocol, with continuous addition of interventions during trial conduct. APTs may run perpetually, assessing many different interventions. APTs come with benefits regarding increased flexibility, efficiency, and cost-effectiveness compared to conventional, stand-alone RCTs. However, planning APTs is complex, and limited guidance is available.

Methods: In this two-part series, we provide a narrative overview of key considerations for the planning of APTs in a clinical setting based on our own experiences planning and initiation APTs.

Results: The first part covered the following 5 overall considerations: stakeholder involvement; population and setting; interventions and comparisons; clinical outcomes; and adaptations. This second part covers 7 overall considerations: statistics and performance; other design features; data infrastructure; operations and organisation; funding; dissemination; and implementation.

Conclusion: The provided guidance on key considerations when planning APTs will aid researchers considering or planning APTs.