This little paper is an interesting look at trials in Emergency Medicine and their robustness – or lack thereof.
The “fragility index“, as described by its proprietors, is effectively a “number needed to treat” for confidence intervals and p-values. In simplest form, the fragility index is the number of patients needed to change their primary outcome in order to nullify an otherwise statistically significant result. Effectively, if a trial result would change if only two patients had different outcomes, the fragility would be 2 – and this would reflect a trial whose results are not terribly robust.
The authors then go on to use their tool to evaluate 180 trials across emergency medicine and emergency medicine journals. The medial fragility index across all trials: 4. Of course, this is not terribly unexpected, as the median sample size across all trials was only 140 – not usually enough to illuminate a reliable result except in only the most impressive of effect sizes.
The tool itself is not a novel reinvention of statistical significance or analysis, but just another mechanism for knowledge translation to reflect the relative stability or robustness of a result from a trial. Interpretation of frequentist statistics, p-values, and confidence intervals can frequently be rather opaque or misleading, and reporting a fragility index can be considered one approach to conceptualizing the strength of a result. The real solution, also noted by these authors, is simply to interpret individual trial results in a Bayesian context – adding one trial to a Bayesian prior, when available, to see how a trial shifts the current evidence.
“The Results of Randomized Controlled Trials in Emergency Medicine Are Frequently Fragile”