This lovely article was passed along to me by David Newman during a discussion of IST-3 – the recently infamous, massive randomized trial of thrombolysis for acute stroke. There are two ways of thinking about IST-3, and how the results are viewed in the literature seems to depend how much funding you receive from Boehringer or Genentech. The first way of thinking seems to be accept the results as published, pick apart the subgroups, do statistical contortions, and then either come out in the “pro” camp (Boehringer) or the “con” camp.
The second way of thinking, supported by this article, is “garbage-in, garbage-out”. The key issue for this approach is that IST-3 is an unblinded, open trial, which introduces bias – treating clinicians and patients who believe TPA is a “promising, yet unproven” treatment (from the uncertainty principle of the study) are perceived as more likely to contribute to favorable reported outcomes when receiving the experimental intervention. This effect is probably even more pronounced given that much of the follow-up scoring for the Oxford Handicap Scale was performed by mail-in questionnaire, rather than standardized expert evaluation – which has rather poor kappa to begin with.
Page three of this article delves into the empiric analysis of the impact of blinding, and the relative likelihood of unblinded trials to report favorable outcomes. Essentially, the relative chance of reporting both favorable and unfavorable outcomes are significantly affected. In clinical terms, this leads to presentation of results in which the benefits are exaggerated and the harms are minimized. In the context of IST-3, this essentially means the likelihood of any hidden positive effects vanishes, while the poor outcomes are underreported – and it’s more “negative” than “neutral”.
The authors also note they are preparing a systematic review of trials with blind and non-blind outcome assessors, which would be particularly apt to IST-3, as well.
“Blinding in Randomized Clinical Trials: Imposed Impartiality”
http://www.ncbi.nlm.nih.gov/pubmed/21993424
2 thoughts on “Empiric Measurement of Bias in Unblinded Trials”
Comments are closed.
"the relative chance of reporting both favorable and unfavorable outcomes are significantly affected – leading to exaggeration of both treatment effects and minimizations of treatment harms"
but why then :
"significantly more likely any favorable treatment effects vanish" ?
I would expect the contrary.
Muddy language, but the intent is as you expected! It's been cleaned up.