PECARN, CATCH, CHALICE … or None of the Above?

The decision instrument used to determine the need for neuroimaging in minor head trauma essentially a question of location. If you’re in the U.S., the guidelines feature PECARN. In Canada, CATCH. In the U.K., CHALICE. But, there’s a whole big world out there – what ought they use?

This is a prospective observational study from two countries out in that big remainder of the world – Australia and New Zealand. Over approximately 3.5 years, these authors enrolled patients with non-trivial mild head injuries (GCS 13-15) and tabulated various rule criteria and outcomes. Each rule has slightly different entry criteria and purpose, but over the course of the study, 20,317 patients were gathered for their comparative analysis.

And, the winner … is Australian and New Zealand general practice. Of these 20,000 patients included, only 2,106 (10%) underwent CT. It is hard to read between the lines and determine how many of the injuries included in this analysis were missed on the initial presentation, but if rate of neuroimaging is the simplest criteria for winning, there’s no competition. Applying CHALICE to their analysis cohort would have increased their CT rate to approximately 22%, and CATCH would raise the rate to 30.2%. Application of PECARN would place 46% of the cohort into CT vs. observation – an uncertain range, but certainly higher than 10%.

Regardless, in their stated comparison, the true winner depends on the value-weighting of sensitivity and resource utilization. PECARN approached 100% or 99% sensitivity, missing only 1 patient with clinically important traumatic brain injury out of ~10,000. Contrawise, CATCH and CHALICE missed 13 and 12 out of ~13,000 and ~14,000, respectively. Most of these did not undergo neurosurgical intervention, but a couple missed by CHALICE and CATCH would. However, as noted above, PECARN is probably substantially less specific than both CATCH and CHALICE, which has relatively profound effect on utilization for a low-frequency outcome.

Ultimately, however, any of these decision instruments is usable – as a supplement to your clinical reasoning. Each of these rules simplifies a complex decision into one less so, with all its inherent weaknesses. Fewer than 1% of children with mild head injury need neurosurgical intervention and these are certainly rarely missed by any typical practice. In settings with high CT utilization rates, any one of these instruments will likely prove beneficial. In Australia and New Zealand – as well as many other places around the world – potentially not so much.  This is probably a fine example of the need to compare decision instruments to clinician gestalt.

“Accuracy of PECARN, CATCH, and CHALICE head injury decision rules in children: a prospective cohort study”

D-Dimer, It’s Not Just a Cut-Off

It’s certainly simpler to have a world where everything is black or white, right or wrong, positive or negative. Once upon a time, positive cardiac biomarkers meant acute coronary syndrome – now we have more information and shades of grey in between. The D-dimer, bless its heart, is probably like that, too.

This is a simple study that pooled patients from five pulmonary embolism studies to evaluate the diagnostic performance characteristics of the D-dimer assay. Conventional usage is simply to deploy the test as a dichotomous rule-out – a value below our set sensitivity threshold obviates further testing, while above consigns us to the bitter radiologic conclusion. These authors, perhaps anticipating a more sophisticated diagnostic strategy, go about trying to calculate interval likelihood ratios for the test.

Using over 6,000 patients as their substrate for analysis, these authors determine the various likelihood ratios for D-dimer levels between 250 ng/mL and greater than 5,000 ng/mL, and identify intervals of gradually increasing width, starting at 250 and building up to 2,500. Based on logistic regression modeling, the fitted and approximate iLR range from 0.0625 for those with D-dimer less 250 ng/mL and increasing to 8 for levels greater than 5,000. Interestingly, a D-dimer between 1,000 and 1,499 had an iLR of roughly 1 – meaning those values basically have no effect on the post-test likelihood of PE.

The general implication of these data would be to inform more precise accounting of the risk for PE involving the decision to proceed to CTPA. That said, with our generally inexact tools for otherwise estimating pretest likelihood of disease (Wells, Geneva, gestalt), these data are probably not quite ready for clinical use. I expect further research to develop more sophisticated individual risk prediction models, for which these likelihood ratios may be of value.

“D-Dimer Interval Likelihood Ratios for Pulmonary Embolism”

The Failing Ottawa Heart

Canada! So many rules! The true north strong and free, indeed.

This latest innovation is the Ottawa Heart Failure Risk Scale – which, if you treat it explicitly as titled, is accurate and clinically interesting. However, it also masquerades as a decision rule – upon which it is of lesser standing.

This is a prospective observational derivation of a risk score for “serious adverse events” in an ED population diagnosed with acute heart failure and potential candidates for discharge. Of these 1,100 patients, 170 (15.5%) suffered an SAE – death, myocardial infarction, hospitalization. They used the differences between the groups with and without SAEs to derive a predictive risk score, the elements of which are:

• History of stroke or TIA (1)
• History of intubation for respiratory distress (2)
• Heart rate on ED arrival ≥110 (2)
• Room are SaO2 <90% on EMS or ED arrival (1)
• ECG with acute ischemic changes (2)
• Urea ≥12 mmol/L (1)

This scoring system ultimately provided a prognostic range from 2.8% for a score of zero, up to 89.0% at the top of the scale. This information is – at least within the bounds of generalizability from their study population – interesting from an informational standpoint. However, they then take it to the next level and use this as a potential decision instrument for admission versus discharge – projecting a score ≥2 would decrease admission rates while still maintaining a similar sensitivity for SAEs.

However, the foundational flaw here is the presumption admission is protective against SAEs – both here in this study and in our usual practice. Without a true, prospective validation, we have no evidence this change in and its potential decrease in admissions improves any of many potential outcome measures. Many of their SAEs may not be preventable, nor would the protections from admission be likely durable out to the end of their 14-day follow-up period. Patients were also managed for up to 12 hours in their Emergency Department before disposition, a difficult prospect for many EDs.

Finally, regardless, the complexity of care management and illness trajectory for heart failure is not a terribly ideal candidate for simplification into a dichotomous rule with just a handful of criteria. There were many univariate differences between the two groups – and that’s simply on the variables they chose to collect The decision to admit a patient for heart failure is not appropriately distilled into a “rule” – but this prognostic information may yet be of some value.

“Prospective and Explicit Clinical Validation of the Ottawa Heart Failure Risk Scale, With and Without Use of Quantitative NT-proBNP”

Outsourcing the Brain Unnecessarily

Clinical decision instruments are all the rage, especially when incorporated into the electronic health record – why let the fallible clinician’s electrical Jello make life-or-death decisions when the untiring, unbiased digital concierge can be similarly equipped? Think about your next shift, and how frequently you consciously or unconsciously use or cite a decision instrument in your practice – HEART, NEXUS, PERC, Well’s, PECARN, the list is endless.

We spend a great deal of time deriving, validating, and comparing decision instruments – think HEART vs. TIMI vs. GRACE – but, as this article points out, very little time actually examining their performance compared to clinician judgment.

These authors reviewed all publications in Annals of Emergency Medicine concerned with the performance characteristics of a decision instrument. They identified 171 articles to this effect, 131 of which performed a prospective evaluation. Of these, the authors were able to find only 15 which actually bothered to compare the performance of the objective rule with unstructured physician assessment. With a little extra digging, these authors then identified 6 additional studies evaluating physician assessment in other journals relevant to their original 171.

Then, of these 21 articles, two favored the decision instrument: a 2003 assessment of the Canadian C-Spine Rule, and a 2002 neural network for chest pain. In the remainder, the comparison either favored clinician judgment or was a “toss up” in the sense the performance characteristics were similar and the winner depended on a value-weighting of sensitivity or specificity.

This should not discourage the derivation and evaluation of further decision instruments, as yes, the conscious and unconscious biases of human beings are valid concerns.  Neither should it be construed from these data that many common decision instruments are of lesser value than our current usage places in them, only that they have not yet been tested adequately. However, many of these simple models are simply that – and the complexity of many clinical questions will at least favor the more information-rich approach of practicing clinicians.

“Structured Clinical Decision Aids Are Seldom Compared With Subjective Physician Judgment, and are Seldom Superior”

Ottawa, the Land of Rules

I’ve been to Canada, but I’ve never been to Ottawa. I suppose, as the capital of Canada, it makes sense they’d be enamored with rules and rule-making. Regardless, it still seems they have a disproportionate burden of rules, for better or worse.

This latest publication describes the “Ottawa Chest Pain Cardiac Monitoring Rule”, which aims to diminish resource utilization in the setting of chest pain in the Emergency Department. These authors posit the majority of chest pain patients presenting to the ED are placed on cardiac monitoring in the interests of detecting a life-threatening malignant arrhythmia, despite such being a rare occurrence. Furthermore, the literature regarding alert fatigue demonstrates greater than 99% of monitor alarms are erroneous and typically ignored.

Using a 796 patients sample of chest pain patients receiving cardiac monitoring, these authors validate their previously described rule for avoiding cardiac monitoring: chest pain free and normal or non-specific ECG changes. In this sample, 284 patients met these criteria, and none of them suffered an arrhythmia requiring intervention.

While this represents 100% sensitivity for their rule, as a resource utilization intervention, there is obviously room for improvement. Of patients not meeting their rule, only 2.9% of this remainder suffered an arrhythmia – mostly just atrial fibrillation requiring pharmacologic rate or rhythm control. These criteria probably ought be considered just a minimum standard, and there is plenty of room for additional exclusion.

Anecdotally, not only do most of our chest pain patients in my practice not receive monitoring – many receive their entire work-up in the waiting room!

“Prospective validation of a clinical decision rule to identify patients presenting to the emergency department with chest pain who can safely be removed from cardiac monitoring”

The Chest Pain Decision Instrument Trial

This is a bit of an odd trial. Ostensibly, this is a trial about the evaluation and disposition of low-risk chest pain presenting to the Emergency Department. The authors frame their discussion section by describing their combination of objective risk-stratification and shared decision-making in terms of reducing admission for observation and testing at the index visit.

But, that’s not technically what this trial was about. Technically, this was a trial about patient comprehension – the primary outcome is actually the number of questions correctly answered by patients on an immediate post-visit survey. The dual nature of their trial is evident in their power calculation, which starts with: “We estimated that 884 patients would provide 99% power to detect a 16% difference in patient knowledge between decision aid and usual care arms”, which is an unusual choice of beta and threshold for effect size – basically one additional question correct on their eight-question survey. The rest of their power calculation, however, makes sense “… and 90% power to detect a 10% difference in the proportion of patients admitted to an observation unit for cardiac testing.” It appears the trial was not conducted to test their primary outcome selected by their patient advocates designing the trial, but in actuality to test the secondary outcomes thought important to the clinicians.

So, it is a little hard to interpret their favorable result with respect to the primary outcome – 3.6 vs 4.2 questions answered correctly. After clinicians spent an extra 1.3 minutes (4.4 vs 3.1) with patients showing them a visual aid specific to their condition, I am not surprised patients had better comprehension of their treatment options – and they probably did not require a multi-center trial to prove this.

Then, the crossover between resource utilization and shared decision-making seems potentially troublesome. An idealized version of shared decision-making allows patients to participate in their treatment when there is substantial individual variation between the perceived value of different risks, benefits, and alternatives. However, I am not certain these patients are being invited to share in a decision between choices of equal value – and the authors seem to express this through their presentation of the results.

These are all patients without known coronary disease, normal EKGs, a negative initial cardiac troponin, and considered by treating clinicians to otherwise fall into a “low risk” population. This is a population matching the cohort of interest from Weinstock’s study of patients hospitalized for observation from the Emergency Department, 7,266 patients of whom none independently suffered a cardiac event while hospitalized.  A trial in British Columbia deferred admission for a cohort of patients in favor of outpatient stress tests.  By placing a fair bit of emphasis on their significant secondary finding of a reduction in observation admission from 52% to 37%, the authors seems to indicate their underlying bias is consistent with the evidence demonstrating the safety of outpatient disposition in this cohort.  In short, it seems to me the authors are not using their decision aid to help patients choose between equally valued clinical pathways, but rather to try and convince more patients to choose to be discharged.

In a sense, it represents offering patients a menu of options where overtreatment is one of them.  If a dyspneic patient meets PERC, we don’t offer them a visual aid where a CTPA is an option – and that shouldn’t be our expectation here, either.  These authors have put in tremendous effort over many years to integrate many important tools, but it feels like the end result is a demonstration of a shared decision-making instrument intended to nudge patients into choosing the disposition we think they ought, but are somehow afraid to outright tell them.

“Shared decision making in patients with low risk chest pain: prospective randomized pragmatic trial”

The Machine Can Learn

A couple weeks ago I covered computerized diagnosis via symptom checkers, noting their imperfect accuracy – and grossly underperforming crowd-sourced physician knowledge. However, one area that continues to progress is the use of machine learning for outcomes prediction.

This paper describes advances in the use of “big data” for prediction of 30-day and 180-day readmissions for heart failure. The authors used an existing data set from the Telemonitoring to Improve Heart Failure Outcomes trial as substrate, and then applied several unsupervised statistical models to the data with varying inputs.

There were 236 variables available in the data set for use in prediction, weighted and cleaned to account for missing data. Compared with the C statistic from logistic regression as their baseline comparator, the winner was pretty clearly Random Forests. With a baseline 30-day readmission rate of 17.1% and 180-day readmission of 48.9%, the C statistic for the logistic regression model predicting 30-day readmission was 0.533 – basically no predictive skill. The Random Forest model, however, achieved a C statistic of 0.628 by training on the 180-day data set.

So, it’s reasonable to suggest there are complex and heterogenous data for which machine learning methods are superior to traditional models. These are, unfortunately, pretty terrible C statistics, and almost certainly of very limited use for informing clinical care. As with most decision-support algorithms, I would be curious also to see a comparison with a hypothetical C statistic for clinician gestalt. However, for some clinical problems with a wide variety of influential factors, these sorts of models will likely become increasingly prevalent.

“Analysis of Machine Learning Techniques for Heart Failure Readmissions”

Don’t CTPA With Your Gut Alone

Many institutions are starting to see roll-out of some sort of clinical decision-support for imaging utilization. Whether it be NEXUS, Canadian Head CT, or Wells for PE, there is plenty of literature documenting improved yield following implementation.

This retrospective evaluation looks at what happens when you don’t obey your new robot overlords – and perform CTPA for pulmonary embolism outside the guideline-recommended pathway. These authors looked specifically at non-compliance at the low end – patients with a Wells score ≤4 and performed with either no D-dimer ordered or a normal D-dimer.

During their 1.5 year review period, there were 2,993 examinations and 589 fell out as non-compliant. Most – 563 – of these were low-risk by Wells and omitted the D-dimer. Yield for these was 4.4% positivity, compared with 11.2% for exams ordered following the guidelines. This is probably even a high-end estimate for yield, because this includes 8 (1.4%) patients who had subsegmental or indeterminate PEs but were ultimately anticoagulated, some of whom were undoubtedly false positives. Additionally, none of the 26 patients that were low-risk with a normal D-dimer were diagnosed with PE.

Now, the Wells criteria are just one tool to help reinforce gestalt for PE, and it is a simple rule that does not incorporate all the various factors with positive and negative likelihood ratios for PE. That said, this study should reinforce that low-risk patients should mostly be given the chance to avoid imaging, and a D-dimer can be used appropriately to rule-out PE in those where PE is a real, but unlikely, consideration.

“Yield of CT Pulmonary angiography in the emergency Department When Providers Override evidence-based clinical Decision support”

From Way Too Many CTs to Many CTs

I am always keen to hear reports of successful imaging reduction interventions – and, even moreso, in trauma. The typical, modern, approach to trauma involves liberal use of advanced imaging – almost to the point of it being a punch line.

This single-center before-and-after report details their experiences between 2006 and 2013. Before 2010, there was no specific protocol regarding CT in trauma – leading to institutional self-examination in the setting of rampant overuse. After 2010, the following protocol was in effect:

trauma algorithm

There isn’t much besides good news presented here. Their primary imaging use outcome, abdominopelvic CT, decreased from 76.7% to 44.6% of all presentations. This was related to an increase in mean ISS for those undergoing CT. When free fluid from non-traumatic causes was individually accounted for, the rate of positivity of these CT rose from 12.3% to 17.5%. Finally, mortality was unchanged – 3.1% vs. 2.7%.

No doubt, any reduction in imaging will miss some important findings. The net counterbalancing effect, however, is likely a massive reduction in costs and harms from further evaluation of false-positives, renal contrast injury, and radiation. And, after all, they’re still performing CTs on nearly half their patients!

“Effect of an Institutional Triaging Algorithm on the Use of Multidetector CT for Patients with Blunt Abdominopelvic Trauma over an 8-year Period”

Shaking Out Stroke Mimics

In a world of continued aggressive guideline- and pharmaceutical-sponsored expansion of stroke treatment with thrombolytics, this article fills and important need – better codifying the predictors of stroke mimics. While other editorials espouse the need to be fast without being sure, this is frankly irresponsible medicine – and, in resource-constrained environments, unsustainable.

These authors at two academic centers performed a retrospective clinical and imaging review of 784 patients evaluated for potential acute cerebral ischemia. Patients were excluded if they had signs of acute stroke on initial non-contrast imaging, and if they did not subsequently undergo MRI. Based on review of the totality of clinical information for each patient, 41% of this cohort were deemed stroke mimics. The authors scoring system, then derived 6 variables – and 3 or more were present, the chance of stroke mimic being cause of the current presentation was 87.2%. Their criteria:

  • Absence of facial droop
  • Age <50 y/o
  • Absence of atrial fibrillation
  • SBP <150 mm Hg
  • Presence of isolated sensory deficit
  • History of seizure disorder

When the rate of tPA administration to stroke mimics is ~15%, and 30-40% of patients evaluated for stroke are stroke mimics – there is a lot of waste and potential harm occurring here. These authors suggest the use of this score could potentially halve these errant administrations for 94% sensitivity, or cut errant administrations down to 2% with 90% sensitivity. Considering the patients for which stroke/stroke mimic is an ambiguous diagnosis, it is reasonably likely the symptoms are of lesser severity – and in the range for which tPA is of most tenuously “proven” value. While their rule has not been prospectively validated, some of these elements certainly have face validity, and can be incorporated into current practice at least as a reminder.

“FABS: An Intuitive Tool for Screening of Stroke Mimics in the Emergency Department”