What Does a Sepsis Alert Gain You?

The Electronic Health Record is no longer simply that – a recording of events and clinical documentation.  Decision-support has, for good or ill, morphed it into a digital nanny vehicle for all manner of burdensome nagging.  Many systems have implemented a “sepsis alert”, typically based off vital signs collected at initial assessment. The very reasonable goal is early detection of sepsis, and early initiation of appropriately directed therapy. The downside, unfortunately, is such alerts are rarely true positives for severe sepsis in broadest sense – alerts far outnumber the instances in a change of clinical practice results in a change in outcome.

So, what to make of this:

This study describes a before-and-after performance of a quality improvement intervention to reduce missed diagnoses of sepsis, part of which was introduction of a triage-based EHR alert. These alerts fired during initial assessment based on abnormal vital signs and the presence of high-risk features. The article describes baseline characteristics for a pre-intervention phase of 86,037 Emergency Department visits, and then a post-intervention phase of 96,472 visits. During the post-intervention phase, there were 1,112 electronic sepsis alerts, 265 of which resulted in initiation of sepsis protocol after attending physician consultation.  The authors, generally, report fewer missed or delayed diagnoses during the post-intervention period.

But, the evidence underpinning conclusions from these data – as relating to improvements in clinical care or outcomes, or even the magnitude of process improvement highlighted in the tweet above – is fraught. The alert here is reported as having a sensitivity of 86.2%, and routine clinical practice picked up nearly all of the remaining cases that were alert negative.  The combined sensitivity is reported to be 99.4%.  Then, the specificity appears to be excellent, at 99.1% – but, for such an infrequent diagnosis, even using their most generous classification for true positives, the false alerts outnumbered the true alerts nearly 3 to 1.

And, that classification scheme is the crux of determining the value of this approach. The primary outcome was defined as either treatment on the ED sepsis protocol or pediatric ICU care for sepsis. Clearly, part of the primary outcome is directly contaminated by the intervention – an alert encouraging use of a protocol will increase initiation, regardless of appropriateness. This will not impact sensitivity, but will effectively increase specificity and directly inflate PPV.

This led, importantly, for the authors to include a sensitivity analysis looking at their primary outcome. This analysis looks at the differences in overall performance if stricter rules for a primary outcome might be entertained. These analyses evaluate the predictive value of the protocol if true positives are restricted to those eventually requiring vasoactive agents or pediatric ICU care – and, unsurprisingly, even this small decline in specificity results in dramatic drops in PPV – down to 2.4% for the alert alone.

This number better matches the face validity we’re most familiar with for these simplistic alerts – the vast majority triggered have no chance of impacting clinical care and improving outcomes. It should further be recognized the effect size of early recognition and intervention for sepsis is real, but quite small – and becomes even smaller when the definition broadens to cases of lower severity. With nearly 100,000 ED visits in both the pre-intervention and post-intervention periods, there is no detectable effect on ICU admission or mortality. Finally, the authors focus on their “hit rate” of 1:4 in their discussion – but, I think it is more likely the number of alerts fired for each each case of reduced morbidity or mortality is on the order of hundreds, or possibly thousands.

Ultimately, the reported and publicized magnitude of the improvement in clinical practice likely represents more smoke and mirrors than objective improvements in patient outcomes, and in the zero-sum game of ED time and resources, these sorts of alerts and protocols may represent important subtractions from the care of other patients.

“Improving Recognition of Pediatric Severe Sepsis in the Emergency Department: Contributions of a Vital Sign–Based Electronic Alert and Bedside Clinician Identification”


Angiotensin II for Refractory Shock

If you blockade the angiotensin receptor system, you have a treatment for hypertension. If you agonize that same system, it logically follows you may have a corresponding treatment for hypotension. So, this is ATHOS-3, a phase 3 trial of synthetic human angiotensin II infusion in patients with catecholamine-resistant shock.

Roughly speaking, this is a trial evaluating the effectiveness of angiotensin for improving hemodynamic parameters in adult patients in vasodilatory shock – defined by the trialists as based on sufficient cardiac index, intravascular volume measurements, and persistent hypotension. Enrolled patients also needed to display ongoing hemodynamic derangement despite “high-dose vasopressors”. Exclusion criteria abound. The primary outcome was achievement of mean arterial pressure targets at 3 hours after initiation of angiotensin or placebo infusion.

Over the ~1.5 year study period, 404 patients were screened to ultimately initiate study protocol in 321. There’s little ambiguity with respect to the primary outcome – 69.9% of patients met MAP targets in the angiotensin cohort compared with 23.4% with placebo. Improvement in MAP led to corresponding downtitration of catchecholamine vasopressors in the intervention cohort. The intervention cohort displayed improvements in the cardiovascular SOFA, but no difference in overall SOFA at 48 hours. Mortality was quite high, regardless of group assignment, and no reliable difference was noted. Adverse events were common in each group with, again, no reliable differences detected.

This trial is mostly just interesting from a scientific awareness standpoint. The beneficial or harmful effects of angiotensin infusion are not established by these data. The enrolled population – approximately one patient every four months per site, on average – cannot be reliably generalized. As with any sponsored trial replete with conflict of interest among the authors – and particularly those with slow enrollment due to extensive exclusions – skepticism is particularly warranted. That said, this novel vasopressor clearly warrants additional study and comparative effectiveness evaluation.

“Angiotensin II for the Treatment of Vasodilatory Shock”

Blood Cultures Save Lives and Other Pearls of Wisdom

It’s been sixteen years since the introduction of Early Goal-Directed Therapy in the Emergency Department. For the past decade and a half, our lives have been turned upside-down by quality measures tied to the elements of this bundle. Remember when every patient with sepsis was mandated to receive a central line? How great were the costs – in real, in time, and in actual harms from these well-intentioned yet erroneous directives based off a single trial?

Regardless, thanks to the various follow-ups testing strict protocolization against the spectrum of timely recognition and aggressive intervention, we’ve come a long way. However, there are still mandates incorporating the vestiges of such elements of care –such as those introduced by the New York State Department of Health. Patients diagnosed with severe sepsis or septic shock are required to complete protocols consisting of 3-hour and 6-hour bundles including blood cultures, antibiotics, and intravenous fluids, among others.

This article, from the New England Journal, looks retrospectively at the mortality rates associated with completion of these various elements. Stratified by time-to-completion following initiation of the 3-hour bundle within 6 hours of arrival to the Emergency Department, these authors looked at the mortality associations of the bundle elements.

Winners: obtaining blood cultures, administering antibiotics, and measuring serum lactate
Losers: time to completion of a bolus of intravenous fluids

Of course, since blood cultures are obtained prior to antibiotic administration, these outcomes are co-linear – and they don’t actually save lives, as facetiously suggested in the post heading. But, antibiotic administration was associated with a fraction of a percent of increased mortality per hour delay over the first 12 hours after initiation of the bundle. Intravenous fluid administration, however, showed no apparent association with mortality.

These data are fraught with issues, of course, relating to their retrospective nature and the limitations of the underlying data collection. Their adjusted model accounts for a handful of features, but there are still potential confounders influencing mortality of those who received their bundle completion within 3 hours as compared to those who did not.  The differences in mortality, while a hard and important endpoint, are quite small.  Earlier is probably better, but the individual magnitude of benefit will be unevenly distributed around the average benefit, and while a delay of several hours might matter, minutes probably do not.  The authors are appropriately reserved with their conclusions, however, only stating these observational data support associations between mortality and antibiotic administration, and do not extend to any causal inferences.

The lack of an association between intravenous fluids and mortality, however, raises significant questions requiring further prospective investigation. Could it be, after these years wandering in the wilderness with such aggressive protocols, the only universally key feature is the initiation of appropriate antibiotics? Do our intravenous fluids, given without regard to individual patient factors, simply harm as many as they help, resulting in no net benefit?

These questions will need to be addressed in randomized controlled trials before the next level of evolution in our approach to sepsis, but the equipoise for such trials may now exist – to complete our journey from Early Goal-Directed to Source Control and Patient-Centered.  The difficulty will be, again, in pushing back against well-meaning but ill-conceived quality measures whose net effect on Emergency Department resource utilization may be harm, with only small benefits to a subset of critically ill patients with sepsis.

“Time to Treatment and Mortality during Mandated Emergency Care for Sepsis”


Vitamin C for Sepsis

This is just a quick post in response to a tweet – and hype-machine press-release – making the rounds today.

This covers a before-and-after study regarding a single-center practice change in an intensive care unit where their approach to severe sepsis was altered to a protocol including intravenous high-dose vitamin C (1.5g q6), intravenous thiamine (200mg q12), and hydrocortisone (50mg q6). Essentially, this institution hypothesized this combination might have beneficial physiologic effects and, after witnessing initial anecdotal improvement, switched to this aforementioned protocol. This report describes their outcomes in the context of comparing the treatment group to similar patients treated in the seven months prior.

In-hospital mortality for patients treated on the new protocol was 8.5%, whereas previously treated patients were subject to 40.4% mortality. Vasopressor use and acute kidney injury was similarly curtailed in the treatment group. That said, these miraculous findings – as they are exhorted in the EVMS press release – can only be considered as worthy of further study at this point. With a mere 47 patients in both treatment groups, a non-randomized, before-and-after design, and other susceptibilities to bias, these findings must be prospectively confirmed before adoption. When considered in the context of Ioannidis’ “Why Most Published Research Findings Are False”, caution is certainly advised.

I sincerely hope prospective, external validation will yield similar findings – but will likewise not be surprised if they do not.

“Hydrocortisone, Vitamin C and Thiamine for the Treatment of Severe Sepsis and Septic Shock: A Retrospective Before-After Study”

A qSOFA Trifecta

There’s a new sepsis in town – although, by “new” it’s not very anymore. We’re supposedly all-in on Sepsis-3, which in theory is superior to the old sepsis.

One of the most prominent and controversial aspects of the sepsis reimagining is the discarding of the flawed Systemic Inflammatory Response Syndrome criteria and its replacement with the Quick Sequential Organ Failure Assessment. In theory, qSOFA replaces the non-specific items from SIRS with physiologic variables more closely related to organ failure. However, qSOFA was never prospectively validated or compared prior to its introduction.

These three articles give us a little more insight – and, as many have voiced concern already, it appears we’ve just replaced one flawed agent with another.

The first article, from JAMA, describes the performance of qSOFA against SIRS and a 2-point increase in the full SOFA score in an ICU population. This retrospective analysis of 184,875 patients across 15 years of registry data from 182 ICUs in Australia and New Zealand showed very little difference between SIRS and qSOFA with regard to predicting in-hospital mortality. Both screening tools were also far inferior to the full SOFA score – although, in practical terms, the differences in adjusted AUC were only between ~0.69 for SIRS and qSOFA and 0.76 for SOFA. As prognostic tools, then, none of these are fantastic – and, unfortunately, qSOFA did not seem to offer any value over SIRS.

The second article, also from JAMA, is some of the first prospective data regarding qSOFA in the Emergency Department. This sample is 879 patients with suspected infection, followed for in-hospital mortality or ICU admission. The big news from this article is the AUC for qSOFA of 0.80 compared with the 0.65 for SIRS or “severe sepsis”, as defined by SIRS plus a lactate greater than 2mmol/L. However, at a cut-off of 2 or more for qSOFA, the advertised cut-off for “high risk”, the sensitivity and specificity were 70% and 79% respectively.

Finally, a third article, from Annals of Emergency Medicine, also evaluates the performance characteristics of qSOFA in an Emergency Department population. This retrospective evaluation describes the performance of qSOFA at predicting admission and mortality, but differs from the JAMA article by applying qSOFA to a cross-section of mostly high-acuity visits, both with and without suspected infection. Based on a sample of 22,350 ED visits, they found similar sensitivity and specificity of a qSOFA score of 2 or greater for predicting mortality, 71% and 74%, respectively. Performance was not meaningfully different between those with and without infection.

It seems pretty clear, then, this score doesn’t hold a lot of value. SIRS, obviously, has its well-documented flaws. qSOFA seems to have better discriminatory value with regards to the AUC, but its performance at the cut-off level of 2 puts it right in a no-man’s land of clinical utility. It is not sensitive enough to rely upon to capture all patients at high-risk for deterioration – but, then, its specificity is also poor enough using it to screen the general ED population will still result in a flood of false positives.

So, unfortunately, these criteria are probably a failed paradigm perpetuating all the same administrative headaches as the previous approach to sepsis – better than SIRS, but still not good enough. We should be pursuing more robust decision-support built-in to the EHR, not attempting to reinvent overly-simplified instruments without usable discriminatory value.

“Prognostic Accuracy of the SOFA Score, SIRS Criteria, and qSOFA Score for In-Hospital Mortality Among Adults With Suspected Infection Admitted to the Intensive Care Unit”


“Prognostic Accuracy of Sepsis-3 Criteria for In-Hospital Mortality Among Patients With Suspected Infection Presenting to the Emergency Department”


“Quick SOFA Scores Predict Mortality in Adult Emergency Department Patients With and Without Suspected Infection”



I will readily admit I am stepping outside the bounds of my expertise with this post – with respect to the “shenfu injection” and its effects on physiology. The authors describe shenfu as “originated from Shenfu decoction, a well-known traditional Chinese formulation restoring ‘Yang’ from collapse, tonifying ‘Qi’ for relieving desertion”. More specifically, from a physiologic standpoint: “Ginsenosides and aconite alkaloids are the main active ingredients in Shenfu. Ginsenosides are the determinant contributor to the vasodilator benefit of Shenfu, whereas the alkaloids play a vital role in the cardiac electrophysiological effect of Shenfu by blocking ion channels”. In China, a pharmacologic shenfu distillate is used routinely to treat sepsis and septic shock as a 100mL daily injection – and this is a placebo-controlled trial endeavoring to demonstrate its efficacy.

At face value, the trial appears reasonable – a targeted enrollment of 160 patients with a goal of detecting a 20% difference in mortality at 28-days, based on an expected overall mortality of 40%. Their primary outcome, however, were the co-primary outcomes of “length of ICU stay, the duration of vasopressor use, illness severity, and the degree of organ dysfunction.” A proper study, of course, has a single primary outcome – and, considering the study was powered for a mortality difference, this patient-oriented outcome probably ought to have been made primary.

Regardless, from the results presented here, it is reasonable to suggest this is promising and worthy of additional evaluation. Several outcomes – ICU LOS, APACHE II score, and duration of vasopressor us – reached statistical significance favoring the intervention. The mortality outcome did not meet statistical significance with the intervention at 20.5% and the placebo at 27.8%. However, an absolute mortality improvement of 7.3% is nothing to sneeze at – and I would be happy to see more work performed to replicate or generalize these results.

“Shenfu injection for improving cellular immunity and clinical outcome in patients with sepsis or septic shock”


The Magic Bacterial Divining Rod

Antibiotic overuse is a real issue.  In modern countries, despite obsessing over antibiotic stewardship, we are still suckers for the excessive use of both narrow-spectrum antibiotics for ambulatory patients and broad-spectrum antibiotics for the critically ill.  In less resource-capable areas, the tests used to stratify patients as potentially bacterial or viral exceed the cost of the antibiotics – also leading down the path to overuse.

This breathless coverage, featured in Time, the AFP, and proudly advertised by Stanford Medicine, profiles a new panel of tests that is destined to bring clarity.  Rather than relying simply on a single biomarker, “our test can detect an infection anywhere in the body by ‘reading the immune system’”.

They used retrospective genetic expression cohorts from children and adults with supposedly confirmed non-infectious or infectious etiologies to derive and validate a scoring system to differentiate the underlying cause of sepsis.  They then further trim their model by eliminating infants and predominately healthy patients from outpatient cohorts.  Ultimately, they then test their model on a previously uncharacterized whole blood sample from 96 pediatric sepsis patients and report an AUC for viral vs. bacterial sepsis of 0.84, with a -LR of 0.15 and +LR of 3.0 for bacterial infections.  At face value, translated to a presumed clinical setting with a generally low prevalence of bacterial infection complicating SIRS, this is an uninspiring result.

However, these authors rather focus their discussion and press releases around the -LR of 0.10 and +LR of 2.34 produced as part of their ideal validation cohort, trumpeting its superiority over the -LR for procalcitonin of 0.29 as “three-fold improvement”.  This is, of course, nonsense, as the AUC from that same procalcitonin meta-analysis was 0.85, and these authors are simply cherry-picking one threshold and performance characteristic for their comparison.

Now, that’s hardly to say this is not novel work, and their confusion matrices showing clustering of non-infected SIRS vs. bacterial sepsis vs. viral sepsis are quite lovely.  Their approach is interesting, and very well could ultimately outperform existing strategies.  However, their current performance clearly does not match the hype, and they are miles away from a meaningful validation.  Furthermore, the sort of nano-array assay required is neither fast enough to be clinically useful nor likely to be produced cheaply enough to be used in some of the resource-poor settings they claim to be addressing.

It makes for a nice headline, but it’s better consigned to the “Fantasy/Science Fiction” shelf of your local bookstore for now.

“Robust classification of bacterial and viral infections via integrated host gene expression diagnostics”

Severe Sepsis … or ß-Agonist

As our sepsis overlords entrenched new “quality measures” and other protocol-driven resuscitation requirements in our Emergency Departments, this article serves as a lovely reminder of the importance of staying cognitively engaged.

Lactate levels can be elevated by metabolic and microcirculatory derangements related to the spectrum of sepsis – but also other, non-infectious causes.  These include hepatic disease, multiple toxodromes, and multiple medications – one of the most commonly used being beta-agonist therapy for obstructive airways.  This very simple study examines the physiologic changes in healthy volunteers receiving 10mg of nebulized albuterol, as compared with nebulized saline.  Placebo volunteers had no change in lactate or placebo.  Albuterol receiving volunteers had an average increase in lactate of 0.77 mmol/L and an average decrease in potassium of 0.5 mEq/L.  Lactate increases, however, were highly variable – ranging from 0.04 to 2.02 mmol/L.

These data aren’t perfectly generalizable to the critically or pseudo-critically ill, but they’re a reasonable starting point for a gross estimate.  They’re also justification for reconsideration of potentially inappropriate therapies for an intermediate-range lactate that obstinately refuses to clear – in the context of receiving multiple rounds of nebulizers.

At the very least, it’s a reminder of the various exceptions to our protocols we need to consider to prevent costly and avoidable harms.

“The Effect of Nebulized Albuterol on Serum Lactate and Potassium in Healthy Subjects”

Tying Procalcitonin to Critical Care

It has been hard, over the years, to truly identify a role for procalcitonin.  Generally speaking, its best niche seems to be as a sort of C-reactive protein on steroids – a non-specific infectious or inflammatory marker with better sensitivity than WBC.  This has led to some usage in zero-miss contexts such as neonatal sepsis, as well as a potential role in antibiotic stewardship.

These authors, many of which are supported by the manufacturers of the procalcitonin assay, evaluate its predictive power in the setting of pneumonia hospitalization, attempting to risk-stratify patients for the combined endpoint of vasopressor support or invasive ventilation.  Their goal, they say, is to use procalcitonin levels to better inform level-of-care decisions – both escalated and de-escalated – at the time of hospital admission.

They analyzed 1,770 patients from a prior pneumonia study for whom banked serum samples were adequate for procalcitonin measurement, 115 of whom met their combined critical illness endpoint.  They report risk of critical illness increased approximately linearly with procalcitonin from 4% when procalcitonin was undetectable, to 22.4% when procalcitonin was 10ng/mL or above.  The AUC for procalcitonin alone was 0.69, as compared to WBC at 0.54.  Then, they further go on to add usage of procalcitonin in conjunction with other risk-stratification scores – ATS minor criteria, PSI, and SMART-COP – provided additional discriminatory information.

This could be a potentially useful and interesting application of procalcitonin – except they don’t really make any comparisons to other available tools, other than a straw man comparison with WBC.  Would the venerable CRP have a similar AUC?  Or, better yet, a lab we already use nearly ubiquitously to detect occult severe sepsis – a lactic acid level?  The authors do not present any specific discussion of alternative approaches – of which their friends at BioMerieux probably appreciate.

“Procalcitonin as an Early Marker of the Need for Invasive Respiratory or Vasopressor Support in Adults with Community-Acquired Pneumonia”

Informatics Trek III: The Search For Sepsis

Big data!  It’s all the rage with tweens these days.  Hoverboards, Yik Yak, and predictive analytics are all kids talk about now.

This “big data” application, more specifically, involves the use of an institutional database to derive predictors for mortality in sepsis.  Many decision instruments for various sepsis syndromes already exist – CART, MEDS, mREMS, CURB-65, to name a few – but all suffer from the same flaw: how reliable can a rule with just a handful of predictors be when applied to the complex heterogeneity of humanity?

Machine-learning applications of predictive analytics attempt to create, essentially, Decision Instruments 2.0.  Rather than using linear statistical methods to simply weight a small handful of different predictors, most of these applications utilize the entire data set and some form of clustering.  Most generally, these models replace typical variable weighted scoring with, essentially, a weighted neighborhood scheme, in which similarity to other points helps predict outcomes.

Long story short, this study out of Yale utilized 5,278 visits for acute sepsis and a random forest model to create a training set and a validation set.  The random forest model included all available data points from the electronic health record, while other models used up to 20 predictors based on expert input and prior literature.  For their primary outcome of predicting in-hospital death, the AUC for the random forest model was 0.86 (CI 0.82-0.90), while none of the rest of the models exceeded an AUC of 0.76.

This still simply at the technology demonstration phase, and requires further development to become actionable clinical information.  However, I believe models and techniques like this are our next best paradigm in guiding diagnostic and treatment decisions for our heterogenous patient population.  Many challenges yet remain, particularly in the realm of data quality, but I am excited to see more teams engaged in development of similar tools.

“Prediction of In-hospital Mortality in Emergency Department Patients with Sepsis: A Local Big Data Driven, Machine Learning Approach”