Bottom line. Being early with a new method is an advantage only if the method is real. Evidence from several fields shows that a large share of published results do not reproduce. We treat every emerging method as a hypothesis and pass it through six questions, in order, before it goes into a product. Most methods stop at the first or second question, which is when stopping is cheapest.
Most new results do not survive replication
The problem is measurable, and it is not limited to one discipline.
| Study | Field | Finding |
|---|---|---|
| Prinz et al., 2011 [1] | Drug discovery | Published data fully matched in-house results in about 20-25% of 67 projects |
| Begley and Ellis, 2012 [2] | Preclinical cancer research | 6 of 53 landmark studies confirmed (11%) |
| Open Science Collaboration, 2015 [3] | Psychology | 36% of 100 replications were statistically significant, against 97% of originals |
| Baker, 2016 [4] | Survey of 1,576 researchers | More than 70% had tried and failed to reproduce another scientist’s experiment |
| Kapoor and Narayanan, 2023 [5] | Machine-learning-based science | Data leakage found in 294 papers across 17 fields |
Data leakage means information from the test data reaches the model during training. The reported accuracy is then higher than the model will achieve on new data.
None of this means research is unreliable in general. It means that a single published result, a conference demonstration or a benchmark table is a starting hypothesis. It is not yet a fact to build a venture on.
Readiness is a scale, not a yes or no
Technology readiness levels (TRLs) describe maturity on a nine-point scale, from basic principles observed (TRL 1) to a system proven in operation (TRL 9). The scale was developed at NASA and is now used by many funding programs, including in Canada [6], [7]. The hardest transition is usually in the middle: from a component validated in a laboratory (TRL 4) to a prototype demonstrated in a relevant environment (TRL 6). This stretch is often called the valley of death, because research funding tends to end before commercial investment begins.
The six questions below are how we cross that stretch with evidence rather than optimism.
Six questions, in order
- Does it reproduce? Can we reproduce the published result ourselves, on the original data and setup? If not, stop. Nothing downstream matters.
- Does it transfer? Does it hold on data and conditions that resemble the target use, not the paper’s? Many methods are tuned, deliberately or not, to their benchmark.
- Does it beat a strong baseline? Compare it against the simplest credible alternative, tuned with the same effort.
- What does it cost at scale? Count compute, energy, latency, data labelling and the people needed to run it. A method that is 2% more accurate and ten times more expensive is rarely the right choice.
- How does it fail? Identify the failure modes and whether they can be detected in operation. A method that fails silently needs monitoring before it needs scaling.
- Can we back out? Keep the integration replaceable. If the method stops holding up, or something better appears, the product should not need a rebuild.
Question 3 deserves emphasis. In one review of 18 neural recommendation methods published at leading conferences, only 7 could be reproduced with reasonable effort, and 6 of those 7 were outperformed by simple, well-tuned conventional methods [8]. New methods are often compared against weak baselines. A strong baseline is the cheapest test there is.
| Question | Evidence that passes | Typical stage |
|---|---|---|
| Reproduces | Our run matches the published result within its stated error | TRL 3-4 |
| Transfers | Holds on representative data from the target setting | TRL 4-5 |
| Beats a strong baseline | Margin larger than the measurement uncertainty | TRL 4-5 |
| Cost at scale | Cost per useful result fits the business case | TRL 5-6 |
| Failure modes | Failures detected and handled in a realistic trial | TRL 6-7 |
| Reversible | Integration can be replaced without a rebuild | TRL 6-7 |
Why the order matters
The questions are ordered by cost. Reproducing a result takes days. Discovering in production that a method fails silently can take months, and it damages trust that is hard to rebuild. Asking the cheap questions first means most weak methods are set aside before significant money is spent. The strong ones reach the expensive questions with evidence behind them.
The order also protects against a common bias. Once a team has invested in a method, it tends to explain away disappointing results. A fixed sequence of questions, agreed before the work starts, makes the decision to stop a matter of record rather than of morale.
What working at the bleeding edge means
For us, working at the bleeding edge means testing emerging methods early, adopting what holds up under evidence and setting aside what does not. Speed comes from running the gate quickly, not from skipping it. A method that passes all six questions is one we can build a venture on. A method that fails is still useful: the reasons it failed are knowledge that others may not yet have.
Conclusion
Research should not remain in papers, but not every paper should become a product. The path from research to venture runs through reproduction, transfer, baselines, cost, failure modes and reversibility. Asked in that order, those six questions turn early access to new ideas into an advantage rather than a risk.
References
- F. Prinz, T. Schlange, and K. Asadullah, “Believe it or not: How much can we rely on published data on potential drug targets?,” Nature Reviews Drug Discovery, vol. 10, p. 712, 2011.
- C. G. Begley and L. M. Ellis, “Raise standards for preclinical cancer research,” Nature, vol. 483, pp. 531-533, 2012.
- Open Science Collaboration, “Estimating the reproducibility of psychological science,” Science, vol. 349, no. 6251, 2015.
- M. Baker, “1,500 scientists lift the lid on reproducibility,” Nature, vol. 533, pp. 452-454, 2016.
- S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science,” Patterns, vol. 4, no. 9, 2023.
- J. C. Mankins, “Technology readiness levels: A white paper,” NASA Office of Space Access and Technology, 1995.
- Innovation, Science and Economic Development Canada, “Technology readiness levels,” Innovative Solutions Canada, Government of Canada.
- M. Ferrari Dacrema, P. Cremonesi, and D. Jannach, “Are we really making much progress? A worrying analysis of recent neural recommendation approaches,” in Proc. 13th ACM Conf. Recommender Systems (RecSys), 2019.