The Deep Dark Hole Where Clinical Trials Go to Die

This is a really fascinating and troubling article about all those clinical trials that never see the light of day after being completed. This is what’s behind the phenomenon where you wonder “what ever happened to [insert investigational drug treatment for diabetes] here?”, you go look through the literature and can find no updates. I wonder, however, what the answer is. This data is illuminating but there are probably compelling financial, PR or other reasons why people aren’t reporting these results – overcoming those incentives probably will take more than just shining a spotlight on the problem.

A key advocate for Alltrials is Ben Goldacre. Sadly in the US (and in the UK) the pharma companies have been allowed to corrupt the transparency of science and hide bad results. This has resulted in our regulators (FDA) not being able to really see the evidence for efficacy and safety and we have had a number of truly epic debacles. Despite that, few pharma companies have bought into the idea of transparency. Some companies who have been totally humiliated and shamed have finally bought into the idea. For instance, after the Vioxx debacle, GSK has become a big supporter of the effort.

Although some believe that drug companies have patients in their hearts as the ultimate motivation, I don’t believe that stuff. It is about money. And unless you have transparency things can become corrupt and the patient becomes a victim.

2 Likes

So, I’m a research scientist, and although I don’t do bench level wet work anymore, I spent more than five years doing so. In order to get funding to run a study, you have to do some PR work. So you talk (honestly) about what the potential of a new treatment (for plants in my case) will be, how it might change the understanding of basic pathology, what it might do in terms of costs and benefits of treatment (for plant diseases, in my case), etc.

Invariably, the University or Center or Agency PR department picks up on these proposals and casts them in the most glowing of terms in how “Study X by Professor McGenius is going to Cure Diabetes in Less than 5 Years!” Or, in my case, “New funding to Dr. McWilliger’s Lab is going to Cure Totally Disgusting Foot-Rot of Cereals in 5 Years!” The truth is, the vast majority of scientific studies, whether it be of a fungal pathogen, a new strain of rice, or a patented biomolecule for treating Type 1 DM, will produce negative results. As in, “Study X by Professor McGenius shows that Molecule XYZ is no more effective in treating Diabetes than the control (no treatment).” These don’t make it into the popular press, and often times institutional PR departments let those results (95% of all results) die in peace behind the lab building.

These kinds of studies do help to push the science forward, and from a scientific point of view they aren’t failures. They are successful because they help deepen an understanding of the underlying mechanisms of disease and what is not effective in treatment. From a patient’s perspective, clinical trial failures are a terrible thing. From a drug company or agency or researcher’s point of view, they are a necessary step towards finding a cure (if that’s even possible). Still frustrating though.

1 Like

David49, I totally get the reasons why an individual scientist might not publish negative or lackluster results after lots of publicity and hype initially surrounded their work. If someone is studying Drosophila or knockout mice or T-cells in a petri dish, I get it: There’s no easy way to publish negative results, putting together a paper is a pain, and obviously the media relations team is not going to put out a press release on it. If you can’t get it published, that tenure clock is ticking and there’s no point spending your valuable time writing up a paper that will be published in a journal with a low impact factor, if at all.

But I think the difference is these trials are using human beings. This AllTrials website is not even tracking Phase I safety trials, but rather all Phase II and above trials that never saw the light of day. So real people have volunteered, for often little remuneration, a lot of inconvenience and the possibility of harm, to participate in these fairly late-stage studies with at least some expectation that the results will help further scientific knowledge and may even help people with their condition down the road. They deserve to have these results published. Beyond that, a third of NIH trials did not post results to ClinicalTrials.gov. This is public money! Others have taken grants from the government to fund these studies. These scientists have an obligation, through the terms of their grant, to make the results of the study publicly available, either on ClinicalTrials.gov or in some other fashion. They don’t even have to publish the results in a peer reviewed journal – just make them available!

Beyond that, I think this problem maybe a matter of lack of time for researchers at public institutions, but I doubt the reason that pharmas aren’t publishing their results is so benign. If they have any studies that support the FDA-approval of an investigational drug, they will want them out there in marquee lights. The ones they’re not publishing (or are outright burying/hiding) probably reflect badly on products they are hoping to get approved/have already gotten approved.

2 Likes

I agree with you that all studies should be published, even if it’s just on Drosophila :wink: I’m sort of a “pure science” fanatic, and believe that all results are worthy of both publication and consideration. I’m not inclined to think that human studies are more important, because I think all studies are important. But here’s the thing. The NIH, like the NSF, USDA, FDA, and other federal agencies, only recently has a requirement that all funded results be published in open-source journals and all data be archived in publicly available digital spaces. It’s really only shown up in the requirements in the last five years. That means studies that were started five or more years ago aren’t subject to any kind of reporting requirements.

Even worse, there aren’t any accepted enforcement mechanisms for publishing results funded by federal agencies in the US. I know this because it’s what I do for a living. There is language in the proposals and funding agreements requiring publication of data and results, but we (scientists) just haven’t figured out a good way to do that yet. There are good reasons: archiving data in perpetuity in a location that is available to the public is expensive, and there are a lot of funded studies.

Anyways, remember this about pharmaceutical studies: the vast majority are not conducted by pharmaceutical companies, but rather at public or private research universities. We (the public) pay attention to what the PR wings of Bayer and other big companies tell us about diabetes treatments, but drug discovery is almost always in a non-commercial lab. Unless the funding agencies not only have a paper requirement to publish results, but also a mechanism for enforcing that requirement, companies in particular are going to avoid publishing their “failures” because they have no incentive to advertise that failure.

I agree: publicize all results. But, I just don’t see it happening in the current US regulatory environment. That environment is far too profit-driven, and regulatory capture is a real concern. That being said, I think the reason many of these trials disappear is not because of some nefarious backroom deals to get ineffective treatments approved. I think the simpler answer is likely correct: they are failed trials that the company doesn’t want to advertise and the FDA/NIH/USDA isn’t making them publish as part of their funding/regulatory oversight duties.

1 Like

All results should be published, and here’s the critical reason why: negative results advance knowledge just as much as positive ones do. (Somewhat parallels the old saw that human beings learn more from failures than successes, but this is not quite the same dynamic.) Knowing what doesn’t work is just as important as knowing what does, not only because it gives clues about which direction to look in next, but because if nobody knows that study x was done—whether it succeeded or not—the way is open for someone to repeat it, which is a waste of resources and most critically, time.

Forgive me but I can’t resist passing on the following anecdote.

A scientist, unjustly accused and convicted of a major crime, found himself incarcerated with a long-term setence in a jail in the midst of the desert. His cellmate turned out to be another scientist. Determined to escape, the first man tried to convince his co-professional to make the attempt with him; but the man refused. After much planning, and with the undetected help of other inmates, our scientist made his escape.

But the heat of the desert, the lack of food and water, and his inability to locate another human being anywhere drove him almost mad, and he was forced to turn around and return to the prison. He reported his terrible experience to the other scientist, who surprised him by saying, “Yes, I know; I tried and failed, too, for the same reasons.”

The first scientist responded bitterly, “For heaven’s sake, man, when you knew I was going to make a break for it, why didn’t you warn me what it was like out there?”

To which his cellmate replied with a shrug, “Who publishes negative results?”

1 Like

Exactly! Most of us would be perfectly happy to publish negative results if the money was available. It’s not a matter of pride! People don’t understand how much it costs to publish in the peer-reviewed scientific literature. I submit between 4 and 8 manuscripts a year, and I expect to pay between $500 and $2,000 per manuscript that is accepted for publication. Hosting datasets is also very expensive, and the going rate currently seems to be about $100 for each set of raw data (often now required by certain journals and federal agencies). So where do I come up with $10,000 to $15,000 a year? And if I were to religiously publish negative results (which I would happily do), another $10,000 on top of the previous? While many granting agencies are happy to offer multimillion dollar, multiyear funding to research scientists like myself, they often balk at asking for a $25,000 per year line-item for publishing costs.

Science is hard right now, especially if you’re in the public sphere. We don’t work to get rich (because nobody likes to pay us anyways), we work because we love the science and we want to contribute to the public good. However, we can’t self-fund projects, and public funding is ever scarcer year by year. Depending on how this current election goes, there is a good chance we could see even less funding than we do now…

Maybe I think human trials matter more because of consent. Most consent forms clearly lay out that the treatment is unlikely to benefit participants personally but usually say the benefit will accrue to society in the form of scientific knowledge. To not live up to those expectations (especially if they’re explicitly stated in the consent form) when you’re subjecting humans to potential harms is, in my opinion, a violation of trust. Now, some animal rights fanatics may argue that Drosophila maybe should have the opportunity to consent to experiments, but that’s probably a fringe point of view. So a person who just tosses out his negative fruit fly research may be letting down the scientific community and society in an indirect way, but a person who tells a sick person they’ll be helping to advance scientific knowledge and then doesn’t bother to publish the results is harming that person more directly.

I’ve heard that places like PLOS ONE charge higher fees for potential authors in exchange for eliminating the subscription fees. Solves one problem but then opens a whole other kettle of worms, I guess!

Exactly. You can’t have open science without someone paying. Open access journals charge sometimes outrageous fees so that anyone can read the articles (I’ve paid those outrageous fees in that journal; it’s worth it sometimes). The more traditional journals don’t charge (except for hosting data sets), but they can be impossible to access by “normal” folks without a University subscription (I’ve even had US Forest Service colleagues in the past who couldn’t access work in Science and Nature, sadly).

Everything is a tradeoff. But, the truth is this: we would publish everything if we could afford to and journals would actually publish it. I’ve had manuscripts rejected because “the results, though interesting, don’t have any positive effects.” What can you do about that? And I understand your consent issue with human trials: the ethics are a different kettle of fish. I do human-related research now (human-environmental interactions), so I do understand the IRB and ethical considerations, and agree to a certain extent. But I think the reason these studies are likely not published is mostly because journals don’t like negative articles and publishing is expensive.

AllTrials isn’t about whether negative results should be published. It is about keeping trials that are submitted to the FDA transparent and honest. The biggest problem has been that companies fund a trial, it is an epic failure (i.e. does show a drug works or that the drug harms people) and the company declines to provide the trial result to the FDA. A secondary problem is that companies change the outcomes and methodologies after a trial starts once it doesn’t go the way they want. AllTrials is supposed to record trials at their start to avoid these improper behaviors.

Brian_BSC, yes, hopefully AllTrials has enough teeth to keep industry accountable!! I agree that’s the major danger here. David49 highlights the more benign reasons that scientists don’t publish results, but for big splashy trials by drug companies, the motives may be a little different.