ED Hocus POCUS … or Just a Hoax?

A guest post by Rory Spiegel (@EMNerd_) who blogs on nihilism and the art of doing nothing at emnerd.com.

A landmark paper recently published in Lancet Respiratory Medicine is certainly destined to send the ED Ultrasound world into a tizzy. This is the first RCT examining the utility of Emergency Department based Point of Care ultrasound (POCUS) for patients presenting with undifferentiated respiratory complaints. Authors randomized patients presenting to the ED with signs or symptoms concerning for a respiratory etiology to either a standard work up as determined by the treating physician or the addition of POCUS performed by a single experienced operator. The US protocol consisted of sonographic examination of the heart, lungs and lower extremity deep veins to identify possible causes of patients’ symptoms. The authors’ primary outcome was the percentage of patients with a correct presumptive diagnosis 4 hours after presentation to the Emergency Department as determined by two physicians blinded to ED POCUS findings, but with access to the records of the entire hospital stay.

Using this POCUS protocol the authors found stunning success in their primary endpoint. Specifically, the rate of correct diagnoses made at 4-hours in the POCUS group was 88% compared to 63.7% in the standard work up group. Furthermore 78% of the patients in the POCUS group received “appropriate” treatment in the Emergency Department compared to 56.7% in the standard work up group.

Though promising, these benefits did not translate into improvements in true patient oriented benefits. Though not statistically significant, the observed in-hospital and 30-day mortality trended towards harm in the POCUS arm ( 8.2% vs 5.1% and 12% vs 7% respectively). Nor was there any meaningful difference in length of stay or hospital-free days between those in the POCUS group and those in the control group. Even more concerning, was the significant increase in downstream testing that occurred in patients randomized to the POCUS group. Specifically the amount of chest CTs (8.2% vs 1.9%), echocardiograms (10.1% vs 3.8%) and diagnostic thoracocenthesis (5.7% vs 0%). This, of course, may be statistical whimsy, but these findings are concerning for a certain degree of overdiagnosis. Unless detected pathology results in improved patient outcomes secondary to treatment, are we truly helping, or just piling on potential costs of increased vigilance?

I’m sure we all have experienced firsthand the utility of bedside US and this is by no means a call to abandon our probes, but rather an acknowledgement of the possibility of subtle harms. We must keep in mind, all testing comes at a price no matter how non-invasive and radiation-free it appears. The cost in this case is information and how we choose to act on it. This would certainly not be the first time increased access to medical technology has lead to such unintended consequences.

“Point-of-Care Ultrasonography in Patients Admitted With Respiratory Symptoms: a Single-Blind, Randomised Controlled Trial”
http://www.ncbi.nlm.nih.gov/pubmed/24998674

25 thoughts on “ED Hocus POCUS … or Just a Hoax?”

  1. While I don't have documentation to support this, I feel that when CXR and EKG became available, it increased downstream testing as well. That increased testing is not necessarily a bad thing.

    While I do see that trend toward mortality in the US group, that statement implies that coming to the correct diagnosis leads to worse mortality. I think that if either a larger study, one with multiple institutions or more than one ultrasonographer performing the scans, perhaps a mortality benefit would be observed. More studies need to be performed.

    Also, I hate the word POCUS. While I think ultrasound is magical, the word POCUS brings up connotations of a cheap wizard's incantation rather than the patient-centered, life-saving examination that it is.

  2. While I don't have documentation to support this, I feel that when CXR and EKG became available, it increased downstream testing as well. That increased testing is not necessarily a bad thing.

    While I do see that trend toward mortality in the US group, that statement implies that coming to the correct diagnosis leads to worse mortality. I think that if either a larger study, one with multiple institutions or more than one ultrasonographer performing the scans, perhaps a mortality benefit would be observed. More studies need to be performed.

    Also, I hate the word POCUS. While I think ultrasound is magical, the word POCUS brings up connotations of a cheap wizard's incantation rather than the patient-centered, life-saving examination that it is.

  3. "Though not statistically significant, the observed in-hospital and 30-day mortality trended towards harm in the PLACEBO a compared to MAGICBULLET® ( 8.2% vs 5.1% and 12% vs 7% respectively)."

    If I read this I'd say a trend is nothing,since not significant.
    So in this POCUS case I think we should ignore the "trend" an dsee it as a hoax as yu say.

    The word "trend" is actually akin to mere advertisment with no data. When I read it , I get a red flag up warning for some rubbish to come.

    Now depending on the study power, the study doth not show too much benefit on patient oriented outcome.

  4. "Though not statistically significant, the observed in-hospital and 30-day mortality trended towards harm in the PLACEBO a compared to MAGICBULLET® ( 8.2% vs 5.1% and 12% vs 7% respectively)."

    If I read this I'd say a trend is nothing,since not significant.
    So in this POCUS case I think we should ignore the "trend" an dsee it as a hoax as yu say.

    The word "trend" is actually akin to mere advertisment with no data. When I read it , I get a red flag up warning for some rubbish to come.

    Now depending on the study power, the study doth not show too much benefit on patient oriented outcome.

  5. Hi Axel,
    Great to hear from you! Thanks for writing! While I agree that nothing definitive can be said from a mortality difference with a P-Value greater than 0.05%, these results do not have to be completely discarded. Even if statistically significant, given the small sample size and the fact that it is one of the many secondary endpoints measured, these findings would only be hypothesis generated.

    What can be said is despite what appears to be a fairly advantageous intervention (increased correct diagnosis at 4hrs by 24.3%), no patient oriented benefits were gained by this early diagnosis and treatment. Interestingly in the control group the final diagnosis was more often asthma and pulmonary edema while the POCUS group found more patients with a final diagnosis of pulmonary effusion, pulmonary embolism, DVT and malignancy. Was this statistical chance or did the POCUS influence the end diagnosis? Can’t say for sure but certainly we should be aware of the possibility of overdiagnosis and the potential harms that inevitably arise with increased testing.

    Thanks again for your support!

  6. Hi Axel,
    Great to hear from you! Thanks for writing! While I agree that nothing definitive can be said from a mortality difference with a P-Value greater than 0.05%, these results do not have to be completely discarded. Even if statistically significant, given the small sample size and the fact that it is one of the many secondary endpoints measured, these findings would only be hypothesis generated.

    What can be said is despite what appears to be a fairly advantageous intervention (increased correct diagnosis at 4hrs by 24.3%), no patient oriented benefits were gained by this early diagnosis and treatment. Interestingly in the control group the final diagnosis was more often asthma and pulmonary edema while the POCUS group found more patients with a final diagnosis of pulmonary effusion, pulmonary embolism, DVT and malignancy. Was this statistical chance or did the POCUS influence the end diagnosis? Can’t say for sure but certainly we should be aware of the possibility of overdiagnosis and the potential harms that inevitably arise with increased testing.

    Thanks again for your support!

  7. This is a really interesting study with, what I think, is an a little bit of a red flag regarding the "increased diagnostic accuracy" accorded to bedside ultrasound. If you look at the list of "missed diagnoses" in each group, there are some interesting differences that seem to point, probably, additional, less clinically-significant, diagnoses being made in the ultrasound group. These are short-term outcomes re: mortality/morbidity, so, perhaps, some of these extra diagnoses and extra tests lead to long-term benefit, but it may also just be a case of over-detection and over-intervention. It would be interesting to conduct a similar study to this without the treating physicians aware of the ultrasound results, and see exactly how many additional, relevant diagnoses were added by the ultrasound.

  8. This is a really interesting study with, what I think, is an a little bit of a red flag regarding the "increased diagnostic accuracy" accorded to bedside ultrasound. If you look at the list of "missed diagnoses" in each group, there are some interesting differences that seem to point, probably, additional, less clinically-significant, diagnoses being made in the ultrasound group. These are short-term outcomes re: mortality/morbidity, so, perhaps, some of these extra diagnoses and extra tests lead to long-term benefit, but it may also just be a case of over-detection and over-intervention. It would be interesting to conduct a similar study to this without the treating physicians aware of the ultrasound results, and see exactly how many additional, relevant diagnoses were added by the ultrasound.

  9. Hello Rory ,
    I beg to differ a little.

    With a big but (one t will do) : I couldn't access the full paper.

    Yet you write :

    " Even if statistically significant, given the small sample size and the fact that it is one of the many secondary endpoints measured, these findings would only be hypothesis generating".

    Could we say "it depends" ?

    Small is beautiful.

    Well I thought so until John Ioannidis and accomplices wrote the opposite. http://www.mayoclinicproceedings.org/article/S0025-6196%2813%2900403-5/fulltext
    and with Pereira: http://www.ncbi.nlm.nih.gov/pubmed/23093165

    But theoretically, from a maths point of view, small trials can only prove large effect size differences.

    And the reason is that the statistical test used takes the patient numbers into account and is more stringent for small numbers .

    That same reason explains why theoretically, we should accept small trial significant differences as as valid (and clinically more significant) as those from large trials.

    Hard luck, small trials may test heterogeneous or not typical / representative enough patients , just because of a random effect, but carefully done small studies may (may not) include homogenous representative sets of patients.

    I have to admit that mathematics can hardly be distorted (no pinch of salt warranted) when it comes to using them simple events such as heads or tails, peas with colour differences etc. rather than on subjects as complex as (sick) humans. There's the rub, likely.

    So we're scr. oops , trapped anyway , if we are to be overcautious with small trials that are eventually proven exaggerated. Damned if we do, and damned if we don't, relying on big trials that boast statistically significant clinically ludicrous differences ?

    As an aside , about Ryan's word, I don't see pleurisy as a diagnosis but rather as a symptom in need of a diagnosis. But again I haven't seen the full paper (the abstract reader doth comment too much).

  10. Hello Rory ,
    I beg to differ a little.

    With a big but (one t will do) : I couldn't access the full paper.

    Yet you write :

    " Even if statistically significant, given the small sample size and the fact that it is one of the many secondary endpoints measured, these findings would only be hypothesis generating".

    Could we say "it depends" ?

    Small is beautiful.

    Well I thought so until John Ioannidis and accomplices wrote the opposite. http://www.mayoclinicproceedings.org/article/S0025-6196%2813%2900403-5/fulltext
    and with Pereira: http://www.ncbi.nlm.nih.gov/pubmed/23093165

    But theoretically, from a maths point of view, small trials can only prove large effect size differences.

    And the reason is that the statistical test used takes the patient numbers into account and is more stringent for small numbers .

    That same reason explains why theoretically, we should accept small trial significant differences as as valid (and clinically more significant) as those from large trials.

    Hard luck, small trials may test heterogeneous or not typical / representative enough patients , just because of a random effect, but carefully done small studies may (may not) include homogenous representative sets of patients.

    I have to admit that mathematics can hardly be distorted (no pinch of salt warranted) when it comes to using them simple events such as heads or tails, peas with colour differences etc. rather than on subjects as complex as (sick) humans. There's the rub, likely.

    So we're scr. oops , trapped anyway , if we are to be overcautious with small trials that are eventually proven exaggerated. Damned if we do, and damned if we don't, relying on big trials that boast statistically significant clinically ludicrous differences ?

    As an aside , about Ryan's word, I don't see pleurisy as a diagnosis but rather as a symptom in need of a diagnosis. But again I haven't seen the full paper (the abstract reader doth comment too much).

  11. We are fighting the battle in changing the convention to ultrasound, as well as dealing with consultants not comfortable with US as diagnostic test. As we get better, younger specialists are used to US diagnostics, and we have better image incorporation into the record, we will improve metrics, cost, and decrease radiation

  12. We are fighting the battle in changing the convention to ultrasound, as well as dealing with consultants not comfortable with US as diagnostic test. As we get better, younger specialists are used to US diagnostics, and we have better image incorporation into the record, we will improve metrics, cost, and decrease radiation

  13. Great points Axel. To some extent I agree with you. If a trial suffering from a small sample size has any chance of demonstrating a statistically significant result it can only do so if the effect size is large and thus is more likely to be a clinically significant finding. Unfortunately like both those fantastic papers you cited state, small studies are far more prone to the errors of random chance. Especially in a study such as this one, which is examining a multitude of pathologic states with varying 30-day prognoses, the randomization process is of vital importance. If because of random chance and the small sample size one group found themselves in possession of a greater number of malignancies or heart failure patients, this would surely affect outcomes. Though I contend in this case the difference in final diagnosis between the two groups was more likely due to the overdiagnosis caused by the POCUS findings, we are incapable of knowing for sure with such a small dataset.

    This is an interesting study with important findings but until replicated using a larger sample size, is purely hypothesis generating.

    Thanks again for all your support!

  14. Great points Axel. To some extent I agree with you. If a trial suffering from a small sample size has any chance of demonstrating a statistically significant result it can only do so if the effect size is large and thus is more likely to be a clinically significant finding. Unfortunately like both those fantastic papers you cited state, small studies are far more prone to the errors of random chance. Especially in a study such as this one, which is examining a multitude of pathologic states with varying 30-day prognoses, the randomization process is of vital importance. If because of random chance and the small sample size one group found themselves in possession of a greater number of malignancies or heart failure patients, this would surely affect outcomes. Though I contend in this case the difference in final diagnosis between the two groups was more likely due to the overdiagnosis caused by the POCUS findings, we are incapable of knowing for sure with such a small dataset.

    This is an interesting study with important findings but until replicated using a larger sample size, is purely hypothesis generating.

    Thanks again for all your support!

  15. Hi Eric, thanks for writing! Everything you say is completely true. We are becoming far more proficient with the use of bedside US, and these skills will inevitably improve metrics, decrease cost, and radiation dose. What we don’t know as of yet and are incapable of discerning without studies like this, is what the harms of this increased access to testing might cause. It is easy to demonize the CT as the enemy in the battle against overtesting because of the obvious flaws it possesses. The real danger is the information uncovered from such tests and how we choose to act on it. As we become more facile in the use of bedside US this danger becomes ever more relevant. We at least owe it to our patients to examine the possible harms involved.

    Thanks again for the support!

  16. Hi Eric, thanks for writing! Everything you say is completely true. We are becoming far more proficient with the use of bedside US, and these skills will inevitably improve metrics, decrease cost, and radiation dose. What we don’t know as of yet and are incapable of discerning without studies like this, is what the harms of this increased access to testing might cause. It is easy to demonize the CT as the enemy in the battle against overtesting because of the obvious flaws it possesses. The real danger is the information uncovered from such tests and how we choose to act on it. As we become more facile in the use of bedside US this danger becomes ever more relevant. We at least owe it to our patients to examine the possible harms involved.

    Thanks again for the support!

  17. Nice review. We picked up on this paper as well as my first thought was that this was going to be a complete game changer for EM. In the end it's not, but it will no doubt get people thinking. Over at St.Emlyn's our conclusions are very similar (we've linked back to here if that's OK as you address some different issues).

    For me it's important to be mindful to be just as critical of a study that I 'want to be true' just as much as one that I don't believe in.

    Great review from the emlitofnote team.

    vb

    S

  18. Thank you Simon for your support. As I am a big fan of your work it means a great deal.

    As always you have written a fantastic review. I dare say far more comprehensive than my own. As you said, it is of the upmost importance for us to be equally critical of the diagnostic tools and treatments we favor as we are of the ones we hold in a lesser regard. I will be curious to see, if this study is replicated on a larger scale, what the results will demonstrate.

    Thank you again for the support!

  19. Thank you Simon for your support. As I am a big fan of your work it means a great deal.

    As always you have written a fantastic review. I dare say far more comprehensive than my own. As you said, it is of the upmost importance for us to be equally critical of the diagnostic tools and treatments we favor as we are of the ones we hold in a lesser regard. I will be curious to see, if this study is replicated on a larger scale, what the results will demonstrate.

    Thank you again for the support!

  20. Rory –

    While I can't see the full paper (seriously, not even the Yale library has access to this journal…), it provoked some hypothesis generating of my own.

    Are there any studies that show that the (presumably) increased diagnostic accuracy of CT over x-ray has shown a corresponding improvement true patient oriented benefits? For that matter, are there studies that show that the chest x-ray produced such benefits.

    Heck, let's take this back to 1816 – is there any evidence that the (presumably) increased diagnostic accuracy of the stethoscope over the physical exam produced true patient oriented benefits?

    Actually, there is some weak evidence that the introduction of the stethoscope was associated with better diagnosis, but also with false-positive and even specious (e.g. puerile respiration) diagnoses. Trends at hospitals where this new technology was introduced saw increased rates of cardiac pathology documented in patients prior to death, rather than at autopsy, but we don't have too much evidence if mortality or hospitalization rates improved.

    (Ref: A great medical history book "To See with a Better Eye: A Life of R. T. H. Laennec" by Jacalyn Duffin)
    Millhillavecommand.blogspot.com/

  21. Rory –

    While I can't see the full paper (seriously, not even the Yale library has access to this journal…), it provoked some hypothesis generating of my own.

    Are there any studies that show that the (presumably) increased diagnostic accuracy of CT over x-ray has shown a corresponding improvement true patient oriented benefits? For that matter, are there studies that show that the chest x-ray produced such benefits.

    Heck, let's take this back to 1816 – is there any evidence that the (presumably) increased diagnostic accuracy of the stethoscope over the physical exam produced true patient oriented benefits?

    Actually, there is some weak evidence that the introduction of the stethoscope was associated with better diagnosis, but also with false-positive and even specious (e.g. puerile respiration) diagnoses. Trends at hospitals where this new technology was introduced saw increased rates of cardiac pathology documented in patients prior to death, rather than at autopsy, but we don't have too much evidence if mortality or hospitalization rates improved.

    (Ref: A great medical history book "To See with a Better Eye: A Life of R. T. H. Laennec" by Jacalyn Duffin)
    Millhillavecommand.blogspot.com/

Comments are closed.