Measuring Research Capacity: Development of the PACER Tool

Abstract Evaluating research activity in research departments and education programs is conventionally accomplished through measurement of research funding or bibliometrics. This limited perspective of research activity restricts a more comprehensive evaluation of a program's actual research capacity, ultimately hindering efforts to enhance and expand it. The objective of this study was to conduct a scoping review of the existing literature pertaining to the measurement of research productivity in research institutions. Using these findings, the study aimed to create a standardized research measurement tool, the Productivity And Capacity Evaluation in Research (PACER) Tool. The evidence review identified 726 relevant articles in a literature search of PubMed, Web of Science, Embase, ERIC, CINAHL, and Google Scholar with the keywords "research capacity" and "research productivity." Thirty-nine English-language studies applicable to research measurement were assessed in full and 20 were included in the data extraction. Capacity/productivity metrics were identified, and the relevance of each metric was data-charted according to 3 criteria: the metric was objective, organizational in scale, and applicable to varied research domains. This produced 42 research capacity/productivity metrics that fell into 7 relevant categories: bibliometrics, impact, ongoing research, collaboration activities, funding, personnel, and education/academics. With the expertise of a Delphi panel of researchers, research leaders, and organizational leadership, 31 of these 42 metrics were included in the final PACER Tool. This multifaceted tool enables research departments to benchmark research capacity and research productivity against other programs, monitor capacity development over time, and provide valuable strategic insights for decisions such as resource allocation.

Characterizing Research Teams

Abstract Research in health professions education (HPE) is often executed by teams-groups of individuals working together to solve a problem, reconceptualize an issue, or conduct a study. Teams frequently involve individuals from different disciplinary backgrounds or professions1,2 who work collaboratively to accomplish specific scholarly goals. Matching team composition to research goals is critical for ensuring the team has the skills and perspectives necessary to generate, integrate, and mobilize the desired knowledge. Drawing on literature,3,4 we describe the who, what, how, and when of 4 types of research teams.

What is the research question? Estimands explained

Abstract Although many papers of medical research report on a treatment effect, it is not always clear what is exactly meant by that effect. An estimand is a precise definition of a treatment effect and includes 5 attributes: population, treatment, (outcome) variable, intercurrent events, and summary measure. In this paper, we discuss how the estimand framework helps to align different phases of studies of medical treatments, from research objectives to design, conduct, analysis, and reporting of results.

From statistics to deep learning: Using large language models in psychiatric research

Abstract Background: Large Language Models (LLMs) hold promise in enhancing psychiatric research efficiency. However, concerns related to bias, computational demands, data privacy, and the reliability of LLM-generated content pose challenges. GAP: Existing studies primarily focus on the clinical applications of LLMs, with limited exploration of their potentials in broader psychiatric research. Objective: This study adopts a narrative review format to assess the utility of LLMs in psychiatric research, beyond clinical settings, focusing on their effectiveness in literature review, study design, subject selection, statistical modeling, and academic writing. Implication: This study provides a clearer understanding of how LLMs can be effectively integrated in the psychiatric research process, offering guidance on mitigating the associated risks and maximizing their potential benefits. While LLMs hold promise for advancing psychiatric research, careful oversight, rigorous validation, and adherence to ethical standards are crucial to mitigating risks such as bias, data privacy concerns, and reliability issues, thereby ensuring their effective and responsible use in improving psychiatric research.

Missing data and long-term outcomes from nutrition research in the critically ill

Abstract Purpose of review: The use of functional outcomes in critical care nutrition research is increasingly advocated; however, this inevitably gives rise to missing data. Consequently there is a need to adopt modern approaches to the foreseeable problem of missing functional and survival outcomes in research trials. Recent findings: Analyses that ignore unobserved or missing data will often return biased effect estimates. An improved approach is to routinely anticipate the types and extent of missing data, and consider the likely mechanisms of that missingness. The researcher and their statistical advisor may then choose from a number of modern strategies to assess the sensitivity of the research conclusions to the patterns of missingness contained in these research data. Methods widely employed include multiple imputation of missing observations, mixed regression models, use of composite outcome variables with patients who die being attributed a value reflecting the lack of ability to function, and selected Bayesian methodology. Summary: Conclusions from clinical research in critical care nutrition will become more clinically interpretable and generalizable with the adoption of modern methods for the statistical handling of missing data.

Death of the P Value? Bayesian Statistics for Orthopaedic Surgeons

Abstract Statistical interpretation is foundational to evidence-based medicine. Frequentist ( P value testing) and Bayesian statistics are two major approaches for hypothesis testing. Studies analyzed with Bayesian methods are increasingly common with a 4-fold increase in the past 10 years. The Bayesian approach can align with clinical decision making by interpreting smaller differences that are not limited by P values and misleading claims of "trends toward significance." Both methods follow a workflow that includes sampling, hypothesis testing, interpretation, and iteration. Frequentist methodology is familiar and common. However, the limitations are the misunderstanding, misuse, and deceptively simple utility of interpreting dichotomous P values. Bayesian approaches are relatively less common and provide an alternative approach to trial design and data interpretation. Marginal differences elucidated by Bayesian methods may be perceived as less decisive than a P value that may reject a null hypothesis. The purposes of this review are to introduce Bayesian principles and Bayes theorem, define how pretest probability and known information may inform diagnostic testing using an example from prosthetic joint infection, contrast Bayesian and frequentist approaches using an example from the VANCO orthopaedic prospective trial, and describe the criteria for critically reviewing Bayesian studies.

Statistics as a Tool in the Physician's Black Bag

Abstract The era of evidence-based practice began in the 1990s with the hope patient outcomes would be improved by eliminating clinical bias and clinically unsound practices. Clinical guidelines which had been previously written by panels of experts were slowly replaced by careful analysis of existing clinical studies and more rigorous designs of new clinical studies based on sound scientific and statistical principles. This still leaves the practitioner with the responsibility of understanding what the evidence is showing them. This article reviews the statistical thinking that underlies the evidence-based literature. We will review some evolutionary changes to statistical analysis being advocated by statisticians and discuss some nuances related to the use of statistics describing diagnosis and treatment in clinical settings.

How to find nothing 2.0

Abstract Various problematic statistical approaches can be used in regression analyses to help find no significant relationship between explanatory variables and response variables-"to find nothing." In an earlier paper, I provided examples of finding nothing from firearm studies, focusing on the lack of statistical power. In this Viewpoint, I offer three examples of "finding nothing" from firearms research and focus on a single hypothesis-that household gun ownership levels affect suicide rates, examining one type of evidence-cross-sectional ecological studies. I discuss studies examining variations in suicide rates across US states, US cities, and nations, highlighting the work of the one firearm researcher who continually "finds nothing."