Simple graphical rules for assessing selection bias in general-population and selected-sample treatment effects

When analyzing a selected sample from a general population, selection bias can arise relative to the causal average treatment effect (ATE) for the general population, and also relative to the ATE for the selected sample itself. In this paper, we provide simple graphical rules that indicate (1) whether a selected-sample analysis will be unbiased for each ATE and (2) whether adjusting for certain covariates could eliminate selection bias. The rules can easily be checked in a standard single-world intervention graph. When the treatment could affect selection, a third estimand of potential scientific interest is the "net treatment difference"-namely the net change in outcomes that would occur for the selected sample if all members of the general population were treated versus not treated, including any effects of the treatment on which individuals are in the selected sample. We provide graphical rules for this estimand as well. We decompose bias in a selected-sample analysis relative to the general-population ATE into (1) "internal bias" relative to the net treatment difference and (2) "net-external bias," a discrepancy between the net treatment difference and the general-population ATE. Each bias can be assessed unambiguously via a distinct graphical rule, providing new conceptual insight into the mechanisms by which certain causal structures produce selection bias.

Association Does Not Mean Causation, When Observational Data Were Misinterpreted as Causal: The Observational Interpretation Fallacy

BACKGROUND: The differentiation between association and causation is a significant challenge in medical research, often further complicated by cognitive biases that erroneously interpret coincidental observational data as indicative of causality. Such misinterpretations can lead to misguided clinical guidelines and healthcare practice, potentially endangering patient safety and leading to inefficient use of resources. METHODS: We conducted an extensive search of PubMed, Cochrane, and Embase databases up to March 2024, identifying circumstances where associations from observational studies were incorrectly deemed causal. These instances led to changes in clinical practice, embodying what we have termed the 'observational interpretation fallacy'. RESULTS: Our search identified 16 notable cases where observational study-derived associations, initially thought to influence clinical practices and guidelines positively, were later contradicted by findings from randomised controlled trials or further studies, necessitating significant revisions in clinical practice. CONCLUSION: In many cases, misinterpretation of observational finding negatively affecting patient care and public health policies. Addressing and rectifying the observational interpretation fallacy is crucial for the progression of medical research and the maintenance of safe and effective clinical practice. It is imperative for health policymakers, clinicians, and the lay public to critically assess research outcomes and make health-related decisions based on a foundation of evidence-based medicine. This approach ensures the alignment of medical practices with the most current and robust scientific evidence, safeguarding patient welfare and optimising resource allocation.

Caveats of Covariate Adjustment in Disproportionality Analysis for Best Practices

Spontaneous reporting systems (SRS) provide valuable data for detecting unidentified adverse events not observed in clinical trials and for conducting safety assessments that accurately reflect real-world clinical practice. With the increasing number of publications using the SRS for disproportionality analysis (DA), there is an increasing demand for a comprehensive understanding of the research limitations associated with the SRS. However, there is a lack of understanding of the caveats associated with adjusting covariates in DA of the SRS. Herein, we summarized the use of covariate adjustment and its caveats in DA. The Council for International Organizations of Medical Sciences VIII suggests considering adjustments such as stratification when they can enhance the sensitivity and/or specificity of statistical analysis. However, several database-specific and statistical caveats have been identified when adjusting for covariates derived from the SRS. Disproportionality analysis may be affected not only by reporting bias at the time of enrollment but also by sparse-data bias due to variations in the number of enrollment reports. Statistical evidence is needed to determine in which cases and to what extent sensitivity and/or specificity are affected. Nevertheless, it is important for researchers to acknowledge that certain limitations discussed in this context may be inherent and cannot be rectified. Based on this understanding, they can then make an informed decision on whether to perform a covariate adjustment.

The Clinical Interpretation of Noninferiority Trials

Noninferiority trials are designed to demonstrate that a new treatment is not unacceptably worse than a standard treatment, considering an allowable difference termed the noninferiority margin. We highlight that selection of noninferiority margins at the time of study design can be biased toward wider margins that favor noninferiority claims. We discuss a clinically oriented approach to interpretation of results with a focus on confidence intervals and recommend that readers base their judgments regarding noninferiority on margins reflecting patient values and preferences rather than those set by investigators. We provide examples from trials in inflammatory bowel diseases.

Interpretation of statistical findings in randomised trials: a survey of statisticians using thematic analysis of open-ended questions

BACKGROUND: Dichotomisation of statistical significance, rather than interpretation of effect sizes supported by confidence intervals, is a long-standing problem. METHODS: We distributed an online survey to clinical trial statisticians across the UK, Australia and Canada asking about their experiences, perspectives and practices with respect to interpretation of statistical findings from randomised trials. We report a descriptive analysis of the closed-ended questions and a thematic analysis of the open-ended questions. RESULTS: We obtained 101 responses across a broad range of career stages (24% professors; 51% senior lecturers; 22% junior statisticians) and areas of work (28% early phase trials; 44% drug trials; 38% health service trials). The majority (93%) believed that statistical findings should be interpreted by considering (minimal) clinical importance of treatment effects, but many (61%) said quantifying clinically important effect sizes was difficult, and fewer (54%) followed this approach in practice. Thematic analysis identified several barriers to forming a consensus on the statistical interpretation of the study findings, including: the dynamics within teams, lack of knowledge or difficulties in communicating that knowledge, as well as external pressures. External pressures included the pressure to publish definitive findings and statistical review which can sometimes be unhelpful but can at times be a saving grace. However, the concept of the minimally important difference was identified as a particularly poorly defined, even nebulous, construct which lies at the heart of much disagreement and confusion in the field. CONCLUSION: The majority of participating statisticians believed that it is important to interpret statistical findings based on the clinically important effect size, but report this is difficult to operationalise. Reaching a consensus on the interpretation of a study is a social process involving disparate members of the research team along with editors and reviewers, as well as patients who likely have a role in the elicitation of minimally important differences.

Statistical methods leveraging the hierarchical structure of adverse events for signal detection in clinical trials: a scoping review of the methodological literature

BACKGROUND: In randomised controlled trials with efficacy-related primary outcomes, adverse events are collected to monitor potential intervention harms. The analysis of adverse event data is challenging, due to the complex nature of the data and the large number of unprespecified outcomes. This is compounded by a lack of guidance on best analysis approaches, resulting in widespread inadequate practices and the use of overly simplistic methods; leading to sub-optimal exploitation of these rich datasets. To address the complexities of adverse events analysis, statistical methods are proposed that leverage existing structures within the data, for instance by considering groupings of adverse events based on biological or clinical relationships. METHODS: We conducted a methodological scoping review of the literature to identify all existing methods using structures within the data to detect signals for adverse reactions in a trial. Embase, MEDLINE, Scopus and Web of Science databases were systematically searched. We reviewed the analysis approaches of each method, extracted methodological characteristics and constructed a narrative summary of the findings. RESULTS: We identified 18 different methods from 14 sources. These were categorised as either Bayesian approaches (n=11), which flagged events based on posterior estimates of treatment effects, or error controlling procedures (n=7), which flagged events based on adjusted p-values while controlling for some type of error rate. We identified 5 defining methodological characteristics: the type of outcomes considered (e.g. binary outcomes), the nature of the data (e.g. summary data), the timing of the analysis (e.g. final analysis), the restrictions on the events considered (e.g. rare events) and the grouping systems used. CONCLUSIONS: We found a large number of analysis methods that use the group structures of adverse events. Continuous methodological developments in this area highlight the growing awareness that better practices are needed. The use of more adequate analysis methods could help trialists obtain a better picture of the safety-risk profile of an intervention. The results of this review can be used by statisticians to better understand the current methodological landscape and identify suitable methods for data analysis - although further research is needed to determine which methods are best suited and create adequate recommendations.

FDA experiences with a centralized statistical monitoring tool

The U.S. Food and Drug Administration (FDA) has broadly supported quality by design initiatives for clinical trials - including monitoring and data validation - by releasing two related guidance documents (FDA 2013 and 2019). Centralized statistical monitoring (CSM) can be a component of a quality by design process. In this article, we describe our experience with a CSM platform as part of a Cooperative Research and Development Agreement between CluePoints and FDA. This agreement's approach to CSM is based on many statistical tests performed on all relevant subject-level data submitted to identify outlying sites. An overall data inconsistency score is calculated to assess the inconsistency of data from one site compared to data from all sites. Sites are ranked by the data inconsistency score ( - log10(p), where p is an aggregated p-value). Results from a deidentified trial demonstrate the typical data anomaly findings through Statistical Monitoring Applied to Research Trials analyses. Sensitivity analyses were performed after excluding laboratory data and questionnaire data. Graphics from deidentified subject-level trial data illustrate abnormal data patterns. The analyses were performed by site, country/region, and patient separately. Key risk indicator analyses were conducted for the selected endpoints. Potential data anomalies and their possible causes are discussed. This data-driven approach can be effective and efficient in selecting sites that exhibit data anomalies and provides insights to statistical reviewers for conducting sensitivity analyses, subgroup analyses, and site by treatment effect explorations. Messy data, data failing to conform to standards, and other disruptions (e.g. the COVID-19 pandemic) can pose challenges.

The Sense and Sensibility of Sensitivity Analyses

No abstract available

Violating independence assumption in medical statistics

Reply: Interpretation of subgroup analyses.

Understanding the Consequences of Collinearity for Multilevel Models: The Importance of Disaggregation Across Levels

In multilevel models, disaggregating predictors into level-specific parts (typically accomplished via centering) benefits parameter estimates and their interpretations. However, the importance of level-specificity has been sparsely addressed in multilevel literature concerning collinearity. In this study, we develop novel insights into the interactivity of centering and collinearity in multilevel models. After integrating the broad literatures on centering and collinearity, we review level-specific and conflated correlations in multilevel data. Next, by deriving formal relationships between predictor collinearity and multilevel model estimates, we demonstrate how the consequences of collinearity change across different centering specifications and identify data characteristics that may exacerbate or mitigate those consequences. We show that when all or some level-1 predictors are uncentered, slope estimates can be greatly biased by collinearity. Disaggregation of all predictors eliminates the possibility that fixed effect estimates will be biased due to collinearity alone; however, under some data conditions, collinearity is associated with biased standard errors and random effect (co)variance estimates. Finally, we illustrate the importance of disaggregation for diagnosing collinearity in multilevel data and provide recommendations for the use of level-specific collinearity diagnostics. Overall, the necessity of disaggregation for identifying and managing collinearity's consequences in multilevel models is clarified in novel ways.

Interpreting diagnostic accuracy studies based on retrospective routinely collected data

Heterogeneity in meta-analysis: a comprehensive overview

In recent years, meta-analysis has evolved to a critically important field of Statistics, and has significant applications in Medicine and Health Sciences. In this work we briefly present existing methodologies to conduct meta-analysis along with any discussion and recent developments accompanying them. Undoubtedly, studies brought together in a systematic review will differ in one way or another. This yields a considerable amount of variability, any kind of which may be termed heterogeneity. To this end, reports of meta-analyses commonly present a statistical test of heterogeneity when attempting to establish whether the included studies are indeed similar in terms of the reported output or not. We intend to provide an overview of the topic, discuss the potential sources of heterogeneity commonly met in the literature and provide useful guidelines on how to address this issue and to detect heterogeneity. Moreover, we review the recent developments in the Bayesian approach along with the various graphical tools and statistical software that are currently available to the analyst. In addition, we discuss sensitivity analysis issues and other approaches of understanding the causes of heterogeneity. Finally, we explore heterogeneity in meta-analysis for time to event data in a nutshell, pointing out its unique characteristics.

Speaking truth about power: Are underpowered trials undercutting evaluation of new care models?