More Than Half Of High-Impact Cancer Studies Can't Be Replicated

An ambitious project that set out 8 years ago to replicate findings from top cancer labs has drawn to a discouraging close.

Photo Insert: Cell structure flasks and petri dishes for cancer research

The Reproducibility Project: Cancer Biology (RP:CB) reported that when it attempted to repeat experiments drawn from 23 high-impact papers published about 10 years ago, fewer than half yielded similar results, Jocelyn Kaiser wrote for Science.

The findings pose “challenges for the credibility of preclinical cancer biology,” says psychologist Brian Nosek, executive director of the Center for Open Science (COS), a co-organizer of the effort.

The project points to a need for authors to share more details of their experiments so others can try to reproduce them, he and others involved argue. Indeed, vague protocols and uncooperative authors, among other problems, ultimately prevented RP:CB from completing replications for 30 of the 53 papers it had initially flagged, the team reports in two capstone papers in eLife.

“It is useful to have this level of objective data about how challenging it can be to measure reproducibility,” says Charles Sawyers of Memorial Sloan Kettering Cancer Center, who reviewed the designs and results for some of the project’s early replication studies. But he wonders whether the project will have much impact. “It’s hard to know whether anything will change as a consequence.”

Nosek’s center and the company Science Exchange set up RP:CB in 2013, after two drug companies reported they could not reproduce many published preclinical cancer studies.

The goal was to replicate key work from top papers in basic cancer biology published by journals such as Science, Nature, and Cell from 2010 to 2012. With funding from the Arnold Foundation (now Arnold Ventures), the organizers designed replication studies that were peer-reviewed by eLife to ensure they would faithfully mimic the original experiments. Outside contract firms or academic service labs would do the experiments.

WEEKLY FEATURE : MVP Group Keeps Lights On During Pandemic