How To Evaluate Decision Instruments

This lovely editorial by Steven Green from Loma Linda succinctly summarizes the limitations of clinical decision instruments.  Decision instruments, referred to in this article as decision “rules”, are potentially valuable distillations of data from large research cohorts meant to concisely address vital clinical concerns.  These include such well-known instruments as NEXUS, PERC, Centor, Alvarado, Wells, and Geneva.

He describes a need for rigorous derivation, external validation, and ease of application as important criteria.  However, the most important topics he addresses are the related issues of “1-way” versus “2-way” application and whether the rule improves upon pre-existing clinical practice.  A “1-way” decision instrument informs clinicians only when its criteria are all met – such as the PERC rule.  A patient who fails the PERC rule does not necessarily need any additional testing due to its low specificity.  The NEXUS criteria, on the other hand, is a 2-way decision rule – where its use in appropriately selected patients typically leads to radiography if its criteria are not met.

The danger, however, is the natural propensity to using a “1-way” rule like a “2-way” rule.  His example for this error is the PECARN blunt abdominal trauma article for which I previously expressed concerns.  In the PECARN blunt trauma instrument, the specificity of the derivation was actually lower than the performance of the clinical gestalt of the physicians involved.  This means the authors recommend its use only as a “1-way” rule, based on sensitivity.  However, if the cognitive error is made to apply it as a “2-way” rule, CT scanning will increase by 13%.  Then, unfortunately, if used as a “1-way” rule, the PECARN instrument only has 97% sensitivity compared with the clinician gestalt of 99% sensitivity.  This means that, if implemented as routine practice, the PECARN instrument may have a non-trivial number of misses while potentially increasing scanning.  This illustrates his point as a “poorly-designed” decision rule, despite the statistical power of the cohort evaluated.

Overall, a lovely read regarding how to properly evaluate and apply decision instruments.

“When Do Clinical Decision Rules Improve Patient Care?”
www.ncbi.nlm.nih.gov/pubmed/23548403