Relevant notes and citations provided to TED by Ben Goldacre.
Note: Ben Goldacre supplied these annotated footnotes in August 2012.
C. Glenn Begley and Lee M. Ellis, "Raise standards for preclinical cancer research," Nature, March 29, 2012
Molecular biology flukes are not limited to precognition research! A group of researchers report how they tried to replicate 53 early, laboratory, preclinical studies of potential targets for cancer treatments. 47 of the 53 could not be replicated. They recommend that there should be more opportunity for researchers to publish negative data, and shifts in the incentives for researchers to do so.
AJ Cowley et al., "The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction," International Journal of Cardiology, July 1993
In the 1980s, more than 100,000 people who'd had a heart attack died because they were prescribed anti-arrhythmic drugs. We could have had earlier warning of this danger if we had seen the results of an earlier trial on a similar anti-arrhythmic drug. Because the drug was abandoned and never came to open market, the findings of the trial weren't published until a decade later. Then, the researchers explained that they were concerned their findings could have helped to avert disaster. This story is well told here by the researchers themselves:
"When we carried out our study in 1980, we thought that the increased death rate that occurred in the lorcainide group was an effect of chance," and, "the development of lorcainide was abandoned for commercial reasons, and this study was therefore never published; it is now a good example of ‘publication bias.' The results described here might have provided an early warning of trouble ahead."
This story is also well covered in the excellent book Testing Treatments, available free for download online (full disclosure: I wrote the foreword!).
Dirk Eyding et al., "Reboxetine for acute treatment of major depression," British Medical Journal, October 12, 2010
Seven trials were conducted comparing reboxetine against a placebo. Six trials showed the drug was no better than a dummy sugar pill; one showed it was better than a dummy pill. Only that positive trial was published. Similarly, three trials showed reboxetine was as good as any other antidepressant, but data from three times as many patients showed it was worse. The unflattering data wasn’t published.
Erick Turner et al., "Selective Publication of Antidepressant Trials and Its Influence on Apparent Efficacy," New England Journal of Medicine, January 17, 2008
One good way to document how often trials go missing is to find a list of trials you know have been conducted and completed and then go to see if they have been published. For example, you can often get information on trials conducted before a drug was licensed. This isn’t all the trial data on a drug, but it’s more than enough to explore the phenomenon of unflattering data that goes missing in action before being published in the academic journals that doctors read.
These researchers went and found all the trials on all antidepressants. 37 of the positive trials — all but one — were published in full. The trials with negative results had a different fate. Only three were published; 22 were never published. The remaining 11 trials — which had negative results in the FDA summaries — did appear in the academic literature, but when they did, they were written up as if the drug were a success. In reality, we have 38 positive trials and 37 negative ones; in the academic literature, we have 48 positive trials and 3 negative ones.
F Song et al., "Dissemination and publication of research findings," Health Technology Assessment, February 2010
The problem of trial results that go missing in action is endemic, as shown in this systematic review summarizing all the studies ever conducted. The authors found that overall, positive results are about twice as likely to be published as negative results. This is an excellent overview of the literature on how often trials go missing, and why. It’s not a pop science read but it’s thorough!
Peter Doshi, "Neuraminidase inhibitors--the story behind the Cochrane review," British Medical Journal, December 2009
The company that makes Tamiflu refused to hand over results in a form that the Cochrane reviewers were able to use. They have since got access to some of the full clinical study reports, which exposed possible discrepancies.
Other discussion on this story appears in the April 2012 PloS, in which the authors discuss the company’s explanation of why they won’t share the data and whether they find these reasons credible.
Iain Chalmers, "From optimism to disillusion about commitment to transparency in the medico-industrial complex," Journal of the Royal Society of Medicine, July 2006
People have made various grand claims to have fixed this problem: This has been harmful, since it has left the wider academic and medical community with the false impression that there is no longer a problem. This paper is a good account of the history of how people came to realize that there was a problem to fix, and how the fixes failed to materialize.
Syvain Mathieu et al., "Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials," Journal of the American Medical Association, September 2, 2009
The International Medical Journal Editors Association (IMJEA), said that it would force people to register all trials by saying they would only publish pre-registered trials. Sadly, they didn’t hold the line, and continued to publish unregistered trials.
Andrew Prayle, Matthew Hurley and Alan Smyth, "Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov," British Medical Journal, January 3, 2012
You’re supposed to post results to clinicaltrials.gov within a year of completion of your trial. Agreed upon in 2008, it seems this rule has not been followed.
Even if this fix had been implemented, it would not be enough, because it only requires publication of trials starting from now. We don’t practice medicine using only the results of trials starting from now! We need the results of all trials, for all drugs in current use, without exception, to reduce the risk of us inflicting avoidable harm on patients.