BMC Medical Research Methodology最新文献

Background: Ring-based studies are a novel research design commonly used for research involving infectious diseases: contacts of newly infected individuals form a ring that is targeted for interventions (e.g., vaccine, post-exposure prophylaxis). Given the novelty of the research design, it is critical to obtain feedback from participants on their experiences with ring-based studies to help with the development of future trials.

Methods: In 2021, we conducted 26 semi-structured interviews with adult participants of a COVID-19 ring-based post-exposure prophylaxis trial based in Canada. We applied a purposive sampling approach and electronically recruited participants who tested positive for COVID-19 (Index Cases) and either agreed or declined for the study team to contact their potentially exposed contacts. We also included individuals who participated in the trial after being potentially exposed to an Index Case (known as Ring Members), and those who declined to participate after potential exposure. The methodological design of semi-structured interviews allowed participants to share their opinions and experiences in the trial (e.g., elements they enjoyed and disliked regarding their participation in the study).

Results: The majority of participants in our study were women (62%) and the average age was 37.3 years (SD = 13.2). Overall, participants reported being highly satisfied with partaking in the ring-based trial. Notably, no substantial complaints were voiced about the trial's design involving contact after exposure. The most common reason of satisfaction was the knowledge of potentially helping others by advancing knowledge for a greater cause (e.g., development of potential treatment to prevent SARS-CoV-2 infection). Other reasons were curiosity about participating in a trial, and an activity to fill free time during the pandemic. A central element of dislike was confusion about instructions with the trial (e.g., independent at home SARS-CoV-2 testing). Additionally, maintaining confidentiality was a crucial concern for participants, who sought assurance that their data would not be shared beyond the scope of the study.

Conclusions: Our results have the potential to inform future research, including clinical trials such as ring-based studies, by incorporating insights from participants' experiences into the development of study protocols. Despite some protocol-related challenges, participants expressed high satisfaction, driven by the desire to advance science and potentially aid others.

Background: Assessing the methodological quality of case reports and case series is challenging due to human judgment variability and time constraints. We evaluated the agreement in judgments between human reviewers and GPT-4 when applying a standard methodological quality assessment tool designed for case reports and series.

Methods: We searched Scopus for systematic reviews published in 2023-2024 that cited the appraisal tool by Murad et al. A GPT-4 based agent was developed to assess the methodological quality using the 8 signaling questions of the tool. Observed agreement and agreement coefficient were estimated comparing published judgments of human reviewers to GPT-4 assessment.

Results: We included 797 case reports and series. The observed agreement ranged between 41.91% and 80.93% across the eight questions (agreement coefficient ranged from 25.39 to 79.72%). The lowest agreement was noted in the first signaling question about selection of cases. The agreement was similar in articles published in journals with impact factor < 5 vs. ≥ 5, and when excluding systematic reviews that did not use 3 causality questions. Repeating the analysis using the same prompts demonstrated high agreement between the two GPT-4 attempts except for the first question about selection of cases.

Conclusions: The study demonstrates a moderate agreement between GPT-4 and human reviewers in assessing the methodological quality of case series and reports using the Murad tool. The current performance of GPT-4 seems promising but unlikely to be sufficient for the rigor of a systematic review and pairing the model with a human reviewer is required.

Background: The use of historical external control data in clinical trials has grown in interest and needs when considering the design of future trials. Hybrid control designs can be more efficient to achieve the same power with fewer patients and limited resources. The literature is sparse on appropriate statistical methods which can account for the differences between historical external controls and the control patients in a study. In this article, we illustrate the analysis framework of a clinical trial if a hybrid control design was used after determining an RCT may not be feasible.

Methods: We utilize two previously completed RCTs in nonsquamous NSCLC and a nationwide electronic health record derived de-identified database as examples and compare 5 analysis methods on each trial, as well as a set of simulations to determine operating characteristics of such designs.

Results: In single trial estimation, the Case Weighted Adaptive Power Prior provided estimated treatment hazard ratios consistent with the original trial's conclusions with narrower confidence intervals. The simulation studies showed that the Case Weighted Adaptive Power Prior achieved the highest power (and well controlled type-1 error) across all 5 methods with consistent study sample size.

Conclusions: By following the proposed hybrid control framework, one can design a hybrid control trial transparently and accounting for differences between control groups while controlling type-1 error and still achieving efficiency gains from the additional contribution from external controls.

Background: Rapid timescales for the design and delivery of research were common during the COVID-19 pandemic. The recruitment and retention of healthcare workers (HCWs) as participants in research studies are notoriously challenging, but this was exacerbated during the pandemic by the unprecedented demand placed on the workforce. The SARS-CoV-2 Immunity and Reinfection Evaluation (SIREN study) is a prospective multicentre cohort study following HCWs in the UK. This paper discusses the strategies and challenges associated with recruitment and retention of HCW participants in Scotland.

Methods: There were 44,546 HCWs recruited to the SIREN study, of whom 6,285 were recruited by research teams at ten different research sites in Scotland between October 2020 and March 2021. Information on target and actual sample size, availability of resource, recruitment rate, and recruitment and engagement strategies by site was collated from SIREN study documentation and discussions with local key SIREN site staff. Individual-level data from 6,153 HCW participants with ongoing consent for all data usage were also collated, including socio-demographic data and information on withdrawal (in first year) and opt-in to a study extension after one year. Factors associated with these outcomes were explored in logistic regression analyses.

Results: Different recruitment strategies were used in each site according to local agreements, protocol and staff capacity, with the recruitment period ranging from 13 to 160 days. The locally-agreed recruitment target was met in four sites. The proportion of participants who withdrew in the first year ranged from 3.1 to 24.8% by site, while subsequent opt-in to a 12-month study extension ranged from 28.6 to 74.8%. The sites with the highest proportions of withdrawals were the same four sites with lowest proportions of opt-in. On an individual level, there was a lower level of retention among younger participants, and those from lower socio-economic backgrounds and minority ethnic groups.

Conclusions: Site-specific factors including research-readiness likely had a significant influence on recruitment and retention, more so than the specific recruitment or retention strategies employed. Independent of site factors, individual-level variables influenced recruitment and retention, suggesting targeted strategies may be needed to promote research engagement among particular socio-demographic groups.

Background: Asthma is a heterogeneous disease that affects millions of children and adults. There is a lack of objective gold standard diagnosis that spans the ages; instead, diagnoses are made by clinician assessment based on a cluster of signs, symptoms and objective tests dependent on age. Yet, there is a clear morbidity associated with chronic asthma symptoms. Machine learning has become a popular tool to improve asthma diagnosis and classification. There is a paucity of literature on the use of Bayesian machine learning algorithms to predict asthma diagnosis in children. This paper develops a prediction model using the Bayesian additive regression trees (BART) and compares its performance to various machine learning algorithms in predicting the diagnosis of childhood asthma.

Methods: Clinically relevant variables collected at or before 3 years of age from 2794 participants in the CHILD Cohort Study were used to predict physician-diagnosed asthma at age 5. BART and six other commonly used machine learning algorithms, namely adaptive boosting, logistic regression, decision tree, neural network, random forest, and support vector machine were trained. Measures of performance including sensitivity, specificity, and area under the receiver operating characteristic (ROC) curve were calculated. The confidence intervals were calculated using Bootstrapping samples. Important predictors and interaction effects associated with asthma were also identified using BART.

Results: BART, logistic regression and random forest showed the highest area under the ROC curve compared to other machine learning algorithms. Based on BART, recurrent wheeze, respiratory infection and food sensitization at 3 years of age were the most important predictors. The three most important interaction effects were found to be interaction terms of respiratory infection at 3 years and recurrent wheezing at 3 years, maternal asthma and paternal asthma, and maternal wheezing and inhalant sensitization of child at 3 years.

Conclusions: BART demonstrated promising prediction performance when compared to other machine learning algorithms. Future research could validate the BART in an external cohort to evaluate its reliability and generalizability.

Background: The current practice in guideline development is to use the control group event rate (CR) as a surrogate of baseline risk and to assume portability of the relative treatment effect across populations with low, moderate and high baseline risk. We sought to emulate this practice in a very large sample of meta-analyses.

Methods: We retrieved data from all meta-analyses published in the Cochrane Database of Systematic Reviews (2003-2020) that evaluated a binary outcome, reported 2 × 2 data for each individual study and included at least 4 studies. We excluded studies with no events. We conducted meta-analyses with odds ratios and relative risks and performed subgroup analyses based on tertiles of CR. In sensitivity analyses, we evaluated the use of total event rate (TR) instead of CR and using quartiles instead of tertiles.

Results: The analysis included 2,531 systematic reviews (27,692 meta-analyses, 226,975 studies, 25,669,783 patients).The percentages of meta-analyses with statistically significant interaction (P < 0.05) based on CR tertile or quartile ranged 12-18% across various sensitivity analyses. This percentage increased as the number of studies or range of CR per meta-analysis increased, reflecting increased power of the subgroup test. The percentages of meta-analyses with statistically significant interaction (P < 0.05) with TR quantiles were lower than those with CR but remained higher than expected by chance.

Conclusion: This analysis suggests that when CR or TR are used as surrogates for baseline risk, relative treatment effects may not be portable across populations with varying baseline risks in many meta-analyses. Categroization of the continuous CR variable and not addressing measurement error limit inferences from such analyses and imply that CR is an undesirable source for baseline risk. Guideline developers and decision-makers should be provided with relative and absolute treatment effects that are conditioned on the baseline risk or derived from studies with similar baseline risk to their target populations.

Background: Medical outcomes of interest to clinicians may have multiple categories. Researchers face several options for risk prediction of such outcomes, including dichotomized logistic regression and multinomial logit regression modeling. We aimed to compare these methods and provide guidance needed for practice.

Methods: We described dichotomized logistic regression, multinomial continuation-ratio logit regression, which is an alternative to standard multinomial logit regression for ordinal outcomes, and logistic competing risks regression. We then applied these methods to develop prediction models of survival and neurodevelopmental outcomes based on the NICHD Extremely Preterm Birth Outcome Tool model. The statistical and practical advantages and flaws of these methods were examined. Both discrimination and calibration of the estimated logistic models of dichotomized outcomes and continuation-ratio logit model were assessed.

Results: The dichotomized logistic models and multinomial continuation-ratio logit model had similar discrimination and calibration in predicting death and survival without neurodevelopmental impairment. But the continuation-ratio logit model had better discrimination and calibration in predicting neurodevelopmental impairment. The sum of predicted probabilities of outcome categories from the dichotomized logistic models could deviate from 100% substantially, ranging from 87.7 to 124.0%, and the dichotomized logistic model of neurodevelopmental impairment greatly overpredicted low risks and underpredicted high risks.

Conclusions: Estimating multiple logistic regression models of dichotomized outcomes may result in poorly calibrated predictions for an outcome with multiple ordinal categories. Multinomial continuation-ratio logit regression produces better calibrated predictions, constrains the sum of predicted probabilities to 100%, and has the advantages of simplicity in model interpretation, flexibility to include outcome category-specific predictors and random-effect terms for patient heterogeneity by hospital. It also accounts for mutual dependence among multiple categories and accommodates competing risks.

Background: Randomised controlled trials (RCTs) are considered the gold standard for generating clinical evidence. The focus on high internal validity in RCTs challenges the external validity and generalisability of findings, potentially hindering their application in routine care. In neurorehabilitation, limited literature addresses conducting RCTs feasibly and efficiently. We investigated barriers and facilitators to conducting RCTs within routine care of neurorehabilitation centres from the perspective of stakeholders in neurorehabilitation in Germany and Austria.

Methods: We conducted semi-structured interviews with stakeholders in neurorehabilitation from four centres in Germany and Austria, informed by the Theoretical Domains Framework (TDF) and the Capability, Opportunity, Motivation and Behaviour model (COM-B). Employing a hybrid approach, the interview analysis integrated both deductive, theory-driven analysis based on the TDF domains and COM-B model and inductive, reflexive thematic analysis.

Results: Twelve stakeholders (4 physicians, 4 therapy managers, 4 therapists; 5 females, 7 males; with research experience spanning 0-40 years) were interviewed. Key barriers to conducting RCTs in neurological rehabilitation centres include limited financial, human, and time resources, high clinical workloads, and a lack of interest of some therapists. Ineffective leadership, perceived lack of research expertise, and communication issues were also significant barriers. Social influence factors such as lack of employer support and inadequate training access further contributed to the challenges. Additionally, barriers included insufficient research infrastructure, limited space, internal power struggles, and rigid cost bearer specifications. Key facilitators included physicians' and therapists' motivation to advance the field, contribute to knowledge, and to prioritise patient health. Support from supervisors, joint decision-making, and efficient organisation were crucial facilitators. Flexible therapy planning, mutual support, and interdisciplinary collaboration also played important roles.

Conclusion: Our results suggest that increasing professional development and understanding, along with providing adequate financial, human, time, and spatial resources to support research endeavours, implementing effective communication strategies to enhance interdisciplinary collaboration and coordination among team members may contribute to increased motivation and facilitate RCTs within the setting of neurorehabilitation centres.

Trial registration: This study was prospectively registered with the German Clinical Trials Register (08.04.2021 DRKSID DRKS00024982).

Background: The Dutch Committee for the Evaluation of Oncological Drugs evaluates the effectiveness of new oncological treatments. The committee compares survival endpoints to the so-called PASKWIL-2023 criteria for palliative treatments, which define if treatment effects are considered clinically relevant. A positive recommendation depends on whether the median overall survival (OS) is below or above 12 months in the comparator arm. If the former applies, an OS benefit of at least 12 weeks, and a hazard ratio (HR) smaller than 0.7 are required. If the latter applies, an OS or progression free survival (PFS) benefit of at least 16 weeks, and an HR smaller than 0.7 are required. Nonetheless, the median survival time may not be reached and the proportional hazards (PH) assumption, quantified by the HR, is likely violated for immuno-oncology (IO) therapies, deeming these criteria inappropriate.

Methods: We conducted a systematic literature review to identify statistical methods used to represent the clinical effectiveness of IO therapies based on trial data. We searched MEDLINE and EMBASE databases from inception to August 31, 2022, limited to English papers. Methodological studies, randomized controlled trials, and discussion papers recognising key issues of survival data analysis of IO therapies were eligible for inclusion.

Results: A total of 1,035 unique references were identified. After full paper screening, 17 publications were included in the review. Additionally, 43 papers were identified through 'snowballing'. We conclude that the current PASKWIL-2023 criteria are methodologically incorrect under non-PH. In that case, single summary statistics fail to capture the treatment effect and any measure should be interpreted in combination with the Kaplan-Meier curves. We recommend 'parameter-free' measures, such as the difference in restricted mean survival time, avoiding assumptions on the underlying survival.

Conclusions: The HR is commonly used to assess treatment effectiveness, without investigating the validity of the PH assumption. This happens with the application of the PASKWIL-2023 criteria for palliative oncology treatments, which can only be valid under a PH setting. Under non-PH, alternative treatment effect measures are suggested. We propose a step-by-step approach supporting the choice of the most appropriate methods to quantify treatment effectiveness that can be used to redefine the PASKWIL-2023 criteria, or similar criteria in other clinical areas.

Background: Compared to the general population, people who inject drugs have poor health and wellbeing. Longitudinal studies can provide insight into factors driving these worse health outcomes but are subject to methodological challenges, such as cohort attrition. The aim of this study was to assess and characterise attrition in a prospective cohort of people who inject drugs in Victoria, Australia.

Methods: Using annually collected self-reported data from The Melbourne Injecting Drug User Cohort Study (SuperMIX) from September 2008 to January 2021, we estimated the incidence of participants being lost-to-follow-up (LTFU), with an episode of being LTFU defined as participants not undertaking a follow-up interview within two years of their last interview. We utilised a multiple event discrete-time survival analysis on participant period-observation data to estimate the associations between key factors and LTFU. Key areas of exposure measurement in analyses were sociodemographic, drug use and mental health.

Results: A total of n = 1328 SuperMIX participants completed a baseline interview, with n = 489 (36.8%) LTFU, i.e. not completing a follow-up interview in the following two years. Increased attrition was observed among SuperMIX participants who were: born outside Australia, younger than 30 years, reporting having completed fewer years of education, not residing in stable accommodation, not in stable employment and not on opioid agonist therapy (OAT).

Conclusions: The attrition rate of the SuperMIX cohort has largely been stable throughout the duration of the study. Higher attrition rates among individuals at greater sociodemographic disadvantage and not on OAT suggest that additional efforts are required to retain these participants. Findings also suggest that SuperMIX might not be capturing data on adverse health and wellbeing outcomes among subpopulations at high risk of harm.