Scientific reproducibility hit mainstream this month with a feature article and editorial in The Economist; they highlight issues around the failure to replicate peer-reviewed research, academic biases found in science, use of scarce resources in national science budgets, and even share a primer in statistics around false positives and negatives. Like much from The Economist, it’s a good piece and worth reading. Even the LA Times has picked up on the theme (here), which is good evidence the topic has been fully examined by this point.
This move to mainstream is great news, and follows several years of active dialogue on the subject. Back in April 2011, I blogged about the unspoken rule that >50% of academic science was unreproducible and its impact on startups (here); over the course of the next year, Bayer and Amgen teams published work on the topic as well, as well as commentaries from Nature (here) and the Wall Street Journal (here). A checklist that I called “Begley’s Six Rules” (a hat tip to Lipinsky’s Rule of Five, named after Glenn Begley of Amgen) was presented here in the summer of 2012.
So to beat a dead horse, I wanted to share two exciting recent updates on the subject of scientific reproducibility from efforts that I’ve been involved with – an update on the Reproducibility Initiative and some of the studies they are validating, and a great example of testing scientific reproducibility from miRagen Therapeutics related to a high profile microRNA study.
Reproducibility Initiative
This Initiative, set up by Science Exchange and other collaborators (and mentioned in the Economist feature above), is now officially conducting experiments to replicate academic findings. As one of their Advisory Board members, I’ve watched this progress over the past couple years and am thrilled to be a part of it: kudos to Elizabeth Iorns and the team for pulling this together.
About two weeks ago, the Initiative announced it had been given a $1.3M grant from the Laura and John Arnold Foundation to examine 50 of the most widely cited “novel” cancer studies published between 2010-2012. For those interested, here’s a google doc spreadsheet with the list of the 50 papers and their authors, with PubMed links.
Papers from labs linked to biotech/pharma, or related to ongoing drug R&D programs, are well represented on the list.:
- IDH1/IDH2: Three papers (Figueroa et al 2010, Ward et al 2010, Li et al 2012) address these targets and the role of mutations in cancer metabolism. This target is being addressed by many research teams, and is Agios’ lead program in Phase 1.
- BRD4 and BET inhibition (Zuber et al 2011, Delmore et al 2011): Bromodomains have become very hot targets in industry, and several epigenetic-focused firms are attacking this one, including Tensha Therapeutics, GSK’s Epinova DPU, and many others.
- microRNA-34: This miR has been implicated as a tumor suppressor (Liu et al 2011), and a mimic of this is in Phase 1 for Mirna Therapeutics.
- CD47: This target is an exciting one for solid tumors (Willingham et al 2012) and is being pursued by many, including Stem Cell Therapeutics and Inhibrx, among others.
- Cell penetrating peptides: CPPs coadministered with chemo apparently enhance their effectiveness (Sugahara et al 2010); this is the founding IP of a biotech called CendR Inc out of the Burnham. Lots of CPP approaches have come and gone, hopefully for them this one stands the test of time.
- Lastly, there are a bunch of other hot targets/approaches on the list: regulatory RNA (microRNA networks, long non-coding RNAs); B-RAF mutations, alterations, and inhibition; c-MYC; NRF2; cancer stem cells; microbiome; many others…
The Initiative has just completed a set of experiments on its first project, which has kicked off a fascinating, detailed discussion amongst the Advisory Board about the findings. Was it replicated or not? How stringent should the definition of replication be? Exact reproduction of an IC50, or is general confirmation of the hypothesis enough? Were the conditions conducted properly? Did the right controls get used in the replication, above and beyond what was in the paper? There’s a long line of follow-up questions that the Initiative is sorting through, but it’s exciting progress. The list of topics highlighted in Begley’s Six Rules are important elements: blinding, n’s, controls, repeats, reagents, statistics, etc.
Keep your eyes out for updates on the progress. It will be very interesting to see the final metric on how many of these studies’ findings are sufficiently “replicated” to be called “reproducible”. Internal Atlas team polling thinks that around 30% of these will get replicated, with a spread from 15-40%. Since the list of papers is public (here), someone should set up a prediction market to gauge expectations over time. Just reading the methods and materials sections of the papers is probably enough to make a very educated guess as to what’s likely to replicate or not.
microRNAs and Nature Biotechnology
In its November issue out today, Nature Biotechnology and it’s editorial team are showing real leadership on the subject of reproducibility. They have worked with miRagen Therapeutics to publish a “negative” finding related to a very high profile microRNA paper (disclosure: a company I founded and remain as Chairman/Lead Investor), and wrote an accompanying editorial on the subject.
Back in September 2011, a lab from Nanjing University published an extraordinary claim in Cell Research suggesting that the ingestion of rice containing a plant microRNA could impact liver function: in short, they claimed the ingested microRNA (miR-168a) was taken up by hepatocytes where it would regulate cholesterol and lipoprotein by repressing specific mRNAs. The findings were so amazing that it was widely picked up in the popular science press: Discover Magazine, Scientific American, etc as well as many blogs.
At miRagen, where we are focused on microRNA biology, these findings were obviously of interest. If we could harness a natural mechanism to ingest microRNAs and have then engage targets in the body that would open up oral delivery routes. Working with Monsanto given their agbio expertise, the team set out to replicate the miR-168a findings. In short, after an extensive set of experiments, the team found no evidence for uptake of plant microRNA in plasma or liver, and contradicted several other findings. As the axiom goes, “extraordinary claims require extraordinary proof” and this study failed the test.
But failing to replicate the findings was only half the battle. The authors submitted their paper to Cell Research only to be turned away because they were negative findings – that’s where Editor Andy Marshall and Nature Biotech came into the picture, offering to publish the letter with a compelling editorial on the subject of replication. Kudos to Team miRagen and Nature Biotech.
It’s worth noting the although every field is fraught with challenges, studies involving regulatory RNA biology, in particular microRNAs, are notoriously difficult to replicate. Complex interactions of microRNAs with their targets lead to subtle changes across many proteins. Plus, “tool compounds” based on simple gene sequences are relatively easy to make via mimics or anti-miRs, but getting the right drug-like properties or tissue distribution isn’t easy. We’ve found that a very high number of seemingly interesting microRNA observations in the academic literature are just not reproducible. Miragen, and presumably other miR-focused companies like Regulus and Mirna, pay very careful attention to the validation of academic observations before embarking on new drug discovery campaigns. The art of predicting and confirming scientific reproducibility is a skillset these companies, like many successful biotechs, require.
Conclusion
Conducting ‘validation studies’ like the Reproducibility Initiative is undertaking, and publishing negative results like Nature Biotech has just done, are both important parts of strengthening the pursuit of translational bench-to-bedside science, especially in a world of scarce intellectual and financial resources. Its frightening to think about all the wasted NIH grant money, Pharma R&D funding, and biotech venture capital that have chased findings that can’t be replicated in other labs. It’s hugely inefficient, and like all inefficiencies a better market for information – both positive and negative – will help improve the outcomes of the system and allocate resources more effectively.
