What do you do when another authorship group performs the exact same meta-analysis and systematic review you’ve been working on – and publishes first? Well, there really isn’t much choice – applaud their great work and learn from the experience.
This is primarily an evaluation of the quick Sequential Organ Failure Assessment, with a little of the old Systemic Inflammatory Response Syndrome thrown in for contextual comparison. These studies included those in the Intensive Care Unit, hospital wards, and Emergency Departments. Their primary outcome was mortality, reported in these studies mostly as in-hospital mortality, but also 28-day and 30-day mortality.
The quick synopsis of their results, pooling 38 studies and 383,333 patients, mostly from retrospective studies, and mostly from ICU cohorts:
- qSOFA is not terribly sensitive, particularly in the settings in which it is most relevant. Their reported overall sensitivity of 60.8% is inflated by its performance in ICU patients, and in ED patients sensitivity is only 46.7%.
- Specificity is OK, at 72.0% overall and 81.3% in the ED. However, the incidence of mortality from sepsis is usually low enough in a general ED population the positive predictive value will be fairly weak.
- In their comparative cohort for SIRS, which is frankly probably irrelevant because SIRS is already well-described, the expected results of higher sensitivity and lower specificity were observed.
Their general conclusion, to which I generally agree, is qSOFA is not an appropriate general screening tool. They did not add much from a further editorial standpoint – so, rather than let our own draft manuscript for this same meta-analysis and systematic review languish unseen, here is an abridged version of the Discussion section of our manuscript written by myself, Rory Spiegel, and Jeremy Faust:
This analysis demonstrates qualitatively similar findings as those observed in the original derivation study performed by Seymour et al. We find our pooled AUC, however, to be lower than the 0.81 reported in their derivation and validation cohort, as well as the 0.78 reported in two external validation cohorts. The meaning of this difference is difficult to interpret, as the clinical utility of this instrument is derived from its use as a binary cut-off, rather than an ordinal AUC. Our sensitivity and specificity from our primary analysis, respectively, compare favorably to their reported 55% and 84%. We also found qSOFA’s predictive capabilities remained robust when exposed to our sensitivity analyses. When only studies at low risk for bias were included, qSOFA’s performance improved.
While our evaluation of SIRS is limited by restricting the comparison solely to those studies which contemporaneously reported qSOFA, our results are broadly consistent with results previously reported. The SIRS criteria at the commonly used cut-off benefits from superior sensitivity for mortality in those with suspected infection, while its specificity is clearly lacking due to its impaired capability to distinguish between clinically important immune system dysregulation and normal host responses to physiologic stress. The important discussion, therefore, is whether and how to incorporate each of these tools – and others, such as the Modified Early Warning Score or National Early Warning Score – into clinical practice, guidelines, and quality measures.
The current approach to sepsis revolves around the perceived significant morbidity and mortality associated with under-recognized sepsis, favoring screening tools whose purpose is minimizing missed diagnoses. Current sepsis algorithms typically rely upon SIRS, depending on its maximal catchment at the expense of over-triage. Such maximal catchment almost certainly represents a low-value approach to sepsis, considering the in-hospital mortality of patients in our cohort with ≥2 SIRS criteria is not meaningfully different than the overall mortality of the entire cohort. The subsequent fundamental question, however, is whether qSOFA and its role in the new sepsis definitions provides a structure for improvement.
Using qSOFA as designed with its cut-off of ≥2, it should be clear its sensitivity does not support its use as an early screening tool, despite its simplicity and exclusion of laboratory measures. However, in a cohort with suspected infection and some physiologic manifestations of sepsis, e.g., SIRS, the true value of qSOFA may be in prioritizing a subgroup for early clinical evaluation. In a healthcare system with unlimited resources, it may be feasible to give each patient uncompromising evaluation and care. Absent that, we must hew towards an idealized approach, where our resources are directed towards those highest-yield patients for whom time-sensitive interventions modify downstream outcomes.
Less discussed are the direct, patient-oriented harms resulting from falsely-positive screening tools and over-enrollment into sepsis bundles. Recent data suggests benefits from shorter time-to-antibiotics administration intervals are realized primarily in critically ill patients. As such, utilization of overly sensitive tools, such as the SIRS criteria, would lead to over-triage and over-treatment, leading to potential iatrogenic harms in excess of net benefits. These harms include effects on individual and community patterns of antibiotic resistance, as exposure to broad-spectrum antibiotics leads to induction of extended-spectrum beta-lactamase resistance in gram-negative pathogens or vancomycin- and carbapenem-resistance in enterococci. Unnecessary antibiotic exposures lead to excess cases of C. difficile infections. The aggressive fluid resuscitation mandated by sepsis bundles leads to metabolic derangement and potential respiratory impairment. Further research should assess the extent of these harms, and in what measure they counterbalance those benefiting from time-sensitive interventions.
This meta-analysis has several limitations. First, we were limited by the relative dearth of high quality prospective data; most of the studies included in our analysis were retrospective. Second, we restricted our prognostic analyses to mortality alone, rather than diagnosis of sepsis. We chose to analyze only mortality because of competing sepsis definitions among expert bodies and government-issued guidelines. Among them, however, mortality is a common feature, the most objective metric, and manifestly the most important patient-centered outcome. Our analysis would not capture other important sequelae of sepsis, including amputation, loss of neurologic and/or independent function, chronic pain, and prolonged psychiatric effects of substantial critical illness. Third, we do not know whether patients included in these studies were septic on presentation, or developed sepsis later in their hospitalization. This may degrade the accuracy assessment of both SIRS and qSOFA. Fourth, while we know that qSOFA alone may miss some cases of sepsis that SIRS might detect, we do not know how many would, in reality, have been deprived of antibiotics and other necessary treatments. In other words, the fate of “qSOFA negative” patients who were evaluated and treated by physicians qualified to detect and treat critical illness via clinical acumen is not known; nor it should not be presumed that all such patients would have necessarily been deprived of timely treatment. Our analysis and comparison of SIRS is definitively incomplete, and not the most reliable estimate of its diagnostic characteristics, but provided for incidental comparison.
The prudent clinical role for qSOFA, however, is as yet undefined, and these data do not offer insight regarding its superiority to clinician judgment for determining a cohort at greatest risk for poor outcomes. Compared with SIRS, at least, those patients identified by qSOFA likely better represent the subset of patients for whom aggressive early treatment confers a particular advantage, and may drive high-value care in the sepsis arena. Future research should assist clinicians in further individualizing initial treatment of sepsis for those stratified to differing levels of risk for poor outcome, as well as to account for the iatrogenic harms and system costs.
“Prognostic Accuracy of the Quick Sequential Organ Failure Assessment
for Mortality in Patients With Suspected Infection: A Systematic Review and Meta-analysis”